Deformation detection method, device and system, computer equipment and storage medium

Abstract

The invention relates to a deformation detection method, device and system, computer equipment and a storage medium. The method comprises the following steps: acquiring multiple frames of images of a scanned object; when it is confirmed that the position relation of the multiple frames of images in the global coordinate system meets a preset condition and the images contain coding points, the coding points in the multiple frames of images are matched based on the coding values of the coding points, and coding point matching pairs are obtained; according to the coded point matching pairs, converting the coded point three-dimensional coordinates in the camera coordinate system into a global coordinate system; and under the global coordinate system, according to the three-dimensional distance between the three-dimensional coordinates of the corresponding coding points in the multiple frames of images, determining the error of the coding points so as to detect the corresponding deformed area of the scanned object. According to the method and the device, the coding points can be used for specifying the coding point matching pairs, the three-dimensional coordinates of the coding points are converted into the global coordinate system, and the errors in the neighborhood are represented through the errors of the coding points in the multiple frames of images in the global coordinate system, so that the deformed area is determined according to the errors of the neighborhood.

Description

Technical Field

[0001] The present application relates to the field of three-dimensional scanning technology, and in particular to a deformation detection method, device, system, computer equipment and storage medium. Background Art

[0002] Marker points are an effective means of assisting point cloud stitching in handheld or tracking laser 3D scanning, which can greatly improve the efficiency of point cloud stitching during scanning. During the scanning process, photogrammetry is used to shoot the object frame by frame, and then the image is stitched by matching the marker points in each frame. However, if the object shakes or deforms during the scanning process, the continuous accumulation of errors during the stitching process will cause the same marker points in multiple frames to be layered and unable to be stitched normally. In addition, the position where the marker points are layered does not correspond to the area where the deformation occurs, making it impossible to determine the area where the deformation occurs.

[0003] With regard to the problem in the related art that the deformation region cannot be determined, no effective solution has been proposed so far. Summary of the invention

[0004] Based on this, it is necessary to provide a deformation detection method, device, system, computer equipment and storage medium that can determine the area where deformation occurs in order to solve the above technical problems.

[0005] In a first aspect, a deformation detection method is provided in this embodiment, including:

[0006] Acquire multiple frames of images of a scanned object; wherein encoding points are set on the surface of the scanned object;

[0007] When it is confirmed that the positional relationship of the multiple frames of the image in the global coordinate system meets the preset conditions and the image contains the code point, matching the code points in the multiple frames of the image based on the code values ​​of the code point to obtain a code point matching pair;

[0008] According to the matching pairs of code points, the three-dimensional coordinates of the code points in the camera coordinate system are converted to the global coordinate system;

[0009] In the global coordinate system, the error of the encoding point is determined according to the three-dimensional distance between the three-dimensional coordinates of the corresponding encoding points in the multiple frames of the image, so as to detect the corresponding area of ​​the scanned object where the deformation occurs.

[0010] In some of the embodiments, the surface of the scanned object is also provided with non-coding marking points; the method further comprises:

[0011] Based on the coding point matching pair, non-coding marker points in multiple frames of the image are matched to obtain marker point matching pairs;

[0012] According to the matching pairs of marker points, a corresponding stitching matrix is ​​calculated; the stitching matrix is ​​used to transform the three-dimensional coordinates of the non-encoded marker points in the camera coordinate system into the global coordinate system;

[0013] In the global coordinate system, the error of the encoding point is determined according to the three-dimensional distance between the three-dimensional coordinates of the corresponding non-encoding marking points in the multiple frames of the image, so as to detect the corresponding area where the scanned object is deformed.

[0014] In some embodiments, the step of matching the non-encoded marker points in the multiple frames of the image based on the encoded point matching pair to obtain the marker point matching pair includes:

[0015] Identify and determine the three-dimensional coordinates of the coded points and the non-coded marker points in the image;

[0016] Determine the topological relationship between the coded points and the non-coded marker points in the image according to the three-dimensional coordinates;

[0017] The non-encoded marking points are matched according to the topological relationship and the encoding point matching pairs to obtain the marking point matching pairs.

[0018] In some of the embodiments, it also includes:

[0019] When it is confirmed that the positional relationship of the multiple frames of the image in the global coordinate system meets the preset conditions and the image does not contain the coded point, any non-coded mark point in one of the images is selected as the first principal point; and the topological relationship between the first principal point and the remaining non-coded mark points in the image is determined, and the first principal point is repeatedly selected until all non-coded mark points in the image are traversed;

[0020] Based on the topological relationship, determining the first principal point corresponding to the second principal point in the remaining image;

[0021] Based on the matching pair of the first principal point and the second principal point, the remaining non-encoded marker points in the multiple frames of the image are matched to obtain matching pairs of marker points.

[0022] In some embodiments, converting the three-dimensional coordinates of the code points in the camera coordinate system to the global coordinate system according to the code point matching pairs includes:

[0023] Calculate a stitching matrix according to the matching pairs of code points and the three-dimensional coordinates of the code points in the camera coordinate system and the global coordinate system;

[0024] Based on the stitching matrix, the three-dimensional coordinates of the encoding points in the camera coordinate system are converted to the global coordinate system.

[0025] In some of the embodiments, it also includes:

[0026] Determine the origin of the camera coordinate system where the multiple frames of the image are located and transform it to the global coordinate system;

[0027] If the distance between the origins is within a preset threshold, it is confirmed that the positional relationship between the multiple frames of the image in the global coordinate system meets the preset conditions.

[0028] In a second aspect, a deformation detection device is provided in this embodiment, including:

[0029] An acquisition module, used to acquire multiple frames of images of a scanned object; wherein encoding points are set on the surface of the scanned object;

[0030] A first matching module is used for matching the code points in the multiple frames of the image based on the code values ​​of the code points to obtain matching pairs of code points when it is confirmed that the positional relationship of the multiple frames of the image in the global coordinate system meets the preset conditions and the image contains the code points;

[0031] A deformation detection module, used for converting the three-dimensional coordinates of the code points in the camera coordinate system to the global coordinate system according to the code point matching pairs;

[0032] In the global coordinate system, the error of the encoding point is determined according to the three-dimensional distance of the three-dimensional coordinates of the encoding points corresponding to the multiple frames of the image, so as to detect the corresponding area of ​​the scanned object where the deformation occurs.

[0033] In a third aspect, a deformation detection system is provided in this embodiment, including: a three-dimensional scanning device and a processing device;

[0034] The three-dimensional scanning device is used to scan the scanning object;

[0035] The processing device is connected to the three-dimensional scanning device and is used to execute the deformation detection method described in the first aspect.

[0036] In a fourth aspect, a computer device is provided in this embodiment, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the deformation detection method described in the first aspect when executing the computer program.

[0037] In a fifth aspect, a storage medium is provided in this embodiment, on which a computer program is stored, and when the program is executed by a processor, the deformation detection method described in the first aspect is implemented.

[0038] Compared with the related art, the deformation detection method, device, system, computer equipment and storage medium provided in this embodiment obtain multiple frames of images of the scanned object; wherein, the surface of the scanned object is provided with coding points; when it is confirmed that the positional relationship of the multiple frames of the images in the global coordinate system meets the preset conditions and the images contain the coding points, the coding points in the multiple frames of the images are matched based on the coding values ​​of the coding points to obtain coding point matching pairs; according to the coding point matching pairs, the three-dimensional coordinates of the coding points in the camera coordinate system are converted to the global coordinate system; in the global coordinate system, according to the three-dimensional distance between the corresponding three-dimensional coordinates of the coding points in the multiple frames of the images, the error of the coding points is determined to detect the corresponding area where the scanned object is deformed. Through this embodiment, it is possible to specify coding point matching pairs using coding points, convert the three-dimensional coordinates of the coding points to the global coordinate system, and characterize the errors in the neighborhood by the errors of the coding points in the multiple frames of the images in the global coordinate system, so as to determine the area where the deformation occurs according to the errors in the neighborhood, thereby solving the problem of being unable to determine the area where the deformation occurs.

[0039] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0041] Figure 1 is a hardware structure block diagram of a terminal of a deformation detection method in an embodiment;

[0042] Figure 2 is a flow chart of a deformation detection method in an embodiment;

[0043] Figure 3 is a schematic diagram of coding points and non-coding marking points in one embodiment;

[0044] Figure 4 is a flow chart of a deformation detection method in another embodiment;

[0045] Figure 5 is a structural block diagram of a deformation detection device in an embodiment.

[0046] In the figure: 102, processor; 104, memory; 106, transmission device; 108, input and output device; 10, acquisition module; 20, matching module; 30, deformation detection module. DETAILED DESCRIPTION

[0047] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0048] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the", "these" and the like in this application do not represent quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether directly or indirectly. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. Usually, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0049] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 FIG. 1 is a hardware structure diagram of a terminal of the deformation detection method of this embodiment. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) processor 102 and memory 104 for storing data, wherein processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0050] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the deformation detection method in the present embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0051] The transmission device 106 is used to receive or send data via a network. The above network includes a wireless network provided by the communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0052] Marker points are an effective means of assisting point cloud stitching in handheld or tracking laser 3D scanning, which can greatly improve the efficiency of point cloud stitching during scanning. During the scanning process, photogrammetry is used to shoot the object frame by frame, and then the image is stitched by matching the marker points in each frame. However, if the object shakes or deforms during the scanning process, the continuous accumulation of errors during the stitching process will cause the same marker points in multiple frames to be layered and unable to be stitched normally. In addition, the position where the marker points are layered does not correspond to the area where the deformation occurs, making it impossible to determine the area where the deformation occurs.

[0053] In this embodiment, a deformation detection method is provided. Figure 2 is a flow chart of the deformation detection method in this embodiment. Figure 2 As shown, the method comprises the following steps:

[0054] Step S201, obtaining multiple frames of images of a scanned object; wherein coding points are set on the surface of the scanned object.

[0055] Specifically, a number of coding points are distributed on the surface of the scanned object, and there are no strict requirements on the distance and distribution between the coding points to reduce the influence of factors such as the experience of the setting personnel. Generally speaking, the more coding points are set, the more accurate the detection. Among them, the coding points are points with unique coding patterns designed for the purpose of identifying, matching and splicing images, including a central positioning part and a peripheral coding part. The central positioning part (such as a center circle or a cross mark) provides coordinates that can be accurately positioned, while the peripheral coding part (such as a ring belt, a line segment or a circle) is encoded according to specific rules to represent different identity information. For example, a common encoding method is to divide the outer ring into several equal parts, each of which can be black or white, representing the number 0 or 1, respectively, thereby forming a binary code.

[0056] The surface of the scanned object refers to the surface that needs to be measured. For example: if you want to measure the outer surface of the scanned object, you only need to set marking points on the outer surface of the scanned object. If you want to measure the entire surface of the scanned object, you need to set marking points on all involved surfaces such as the outer surface and inner surface of the scanned object; that is, if you want to measure the entire scanned object, the acquired multi-frame images need to cover the entire surface of the scanned object. There are many types of devices for acquiring multi-frame images of the scanned object, and this application does not limit this. The devices include but are not limited to three-dimensional scanning devices, etc.

[0057] Step S202, when it is confirmed that the positional relationship of the multiple frames of images in the global coordinate system meets the preset conditions and the images contain coded points, the coded points in the multiple frames of images are matched based on the coded values ​​of the coded points to obtain matching pairs of coded points.

[0058] Specifically, in the process of scanning frame by frame to obtain the image of the scanned object, the trajectory of the 3D scanning device is recorded, and loop detection is performed when the position of the current frame image is close to the position of a certain historical frame image. During the scanning process, each frame image is usually converted from the current camera coordinate system to the global coordinate system for splicing, and the position relationship can be judged based on the distance of multiple frames in the global coordinate system to determine whether it meets the preset conditions.

[0059] During the frame-by-frame scanning process, the coding points and the corresponding coding values ​​in the image are identified. Since the coding points are distributed on the surface of the scanned object and have unique coding values, only when it is confirmed that the positional relationship of the multiple frames of images in the global coordinate system meets the preset conditions and the images contain coding points, the corresponding coding points in the multiple frames of images are matched pairwise based on the coding values ​​of the coding points to obtain coding point matching pairs.

[0060] Step S203: transform the three-dimensional coordinates of the code points in the camera coordinate system into the global coordinate system according to the code point matching pairs.

[0061] Specifically, the three-dimensional coordinates of the coding points in multiple frames are obtained through binocular reconstruction, including the three-dimensional coordinates of the coding points in the current frame image in the camera coordinate system and the three-dimensional coordinates of the coding points in the historical frame images in the global coordinate system. According to the above-mentioned mandatory coding point matching pairs, the splicing matrix between the three-dimensional coordinates of the coding points in the current frame image in the camera coordinate system and the three-dimensional coordinates of the coding points in the historical frame image in the global coordinate system is calculated. The splicing matrix is ​​used to convert the three-dimensional coordinates of the coding points in the current frame image in the camera coordinate system to the global coordinate system, that is, to represent the spatial transformation relationship of rotation and translation between the two frames.

[0062] Step S204 , in the global coordinate system, according to the three-dimensional distance between the three-dimensional coordinates of the corresponding coding points in the multiple frames of images, the error of the coding points is determined to detect the corresponding area where the deformation of the scanned object occurs.

[0063] Specifically, for multiple frames of images that conform to the positional relationship in the global coordinate system, the coding points in the images all have corresponding matching relationships. Since the calculation process of the stitching matrix will optimize and balance each coding point matching pair, as well as the influence of errors in the stitching process, the coding points of the current frame image converted to the global coordinate system by the stitching matrix do not completely overlap with the corresponding coding points in the historical frame image. Based on this, the error of each coding point in the image is determined according to the three-dimensional distance between the three-dimensional coordinates of the corresponding coding points in the multiple frames of images. Since the coding point matching pairs are specified, all errors in the neighborhood of each coding point are expressed in the coding point, so that the error of each coding point is used to characterize all errors in the neighborhood of the coding point.

[0064] If the errors of all code points are average but large, it means that there is a problem with the accuracy of the scanning device itself. If the errors of some code points are abnormally large, it means that the scanned object itself has deformed during the measurement process, and the area where the code points with large errors are located indicates the corresponding area where the scanned object has deformed.

[0065] Through the above steps, when the positions of multiple frames of images are close in the global coordinate system, the code points can be used to specify the code point matching pairs, and the three-dimensional coordinates of the code points can be converted to the global coordinate system. The error of each code point in the multiple frames of images in the global coordinate system can be used to represent the error in the neighborhood. In this way, even if the splicing process continues to accumulate errors, resulting in the inability to splice normally and the delamination of the marker points, the area where the deformation occurs can be determined based on the error of the code points, solving the problem of being unable to determine the area where the deformation occurs. In addition, using the code point matching pairs to calculate the splicing matrix between images can reduce the failure of code point loop detection.

[0066] In some of the embodiments, non-coding marking points are also provided on the surface of the scanned object; the above method further comprises the following steps:

[0067] Based on the coding point matching pairs, the non-coding marker points in the multi-frame images are matched to obtain the marker point matching pairs; based on the marker point matching pairs, the corresponding stitching matrix is ​​calculated; the stitching matrix is ​​used to transform the three-dimensional coordinates of the non-coding marker points in the camera coordinate system to the global coordinate system; in the global coordinate system, the error of the coding point is determined according to the three-dimensional distance between the corresponding three-dimensional coordinates of the non-coding marker points in the multi-frame images to detect the corresponding area where the scanned object is deformed.

[0068] Specifically, since the setting cost of the coding points is relatively high, a number of coding points and non-coding marking points can be distributed on the surface of the scanned object, where the coding points and non-coding marking points can be set in a certain quantity ratio, for example, 1:100, but there are no strict requirements on the quantity ratio, the distance and distribution between the coding points and non-coding marking points, so as to reduce the influence of factors such as the experience of the setting personnel. Generally speaking, the more coding points are set, the more accurate the detection is.

[0069] During the frame-by-frame scanning process, the coding points and the corresponding coding values ​​in the image are identified. Since the coding points are distributed on the surface of the scanned object and have unique coding values, only when it is confirmed that the positional relationship of the multiple frames of images in the global coordinate system meets the preset conditions and the images contain coding points, the corresponding coding points in the multiple frames of images are matched pairwise based on the coding values ​​of the coding points to obtain coding point matching pairs.

[0070] Based on the coding point matching pair, the non-coding marker points in the multi-frame images are matched to obtain the marker point matching pair. Among them, the topological relationship between the coding point and the non-coding marker point can be used to establish more marker point matching pairs of non-coding marker points on the basis of the coding point matching pair, and there is no need to set too many coding points. The three-dimensional coordinates of the non-coding marker points in the multi-frame images are obtained by binocular reconstruction, including the three-dimensional coordinates of the non-coding marker points in the current frame image in the camera coordinate system, and the three-dimensional coordinates of the non-coding marker points in the historical frame image in the global coordinate system. According to the marker point matching pair, the splicing matrix between the three-dimensional coordinates of the non-coding marker points in the current frame image in the camera coordinate system and the three-dimensional coordinates of the non-coding marker points in the historical frame image in the global coordinate system is calculated. The splicing matrix is ​​used to convert the three-dimensional coordinates of the non-coding marker points in the current frame image in the camera coordinate system to the global coordinate system, that is, to represent the spatial transformation relationship of rotation and translation between the two frames.

[0071] For multiple frames of images that conform to the positional relationship in the global coordinate system, the non-coding marker points in the image have corresponding matching relationships. Since the calculation process of the splicing matrix will optimize and balance each coding point matching pair, as well as the influence of errors in the splicing process, the non-coding marker points of the current frame image converted to the global coordinate system by the splicing matrix do not completely overlap with the corresponding non-coding marker points in the historical frame image. Based on this, the error of each coding point in the image is determined according to the three-dimensional distance between the three-dimensional coordinates of the corresponding non-coding marker points in the multiple frames. Among them, for a coding point, the three-dimensional distances of all non-coding marker points within a certain distance range of the coding point can be averaged as the error of the coding point. Since the coding point matching pairs are specified, all errors in the neighborhood of each coding point are expressed in the coding point, so that the error of each coding point can be used to characterize all errors in the neighborhood of the coding point.

[0072] If the errors of all code points are average but large, it means that there is a problem with the accuracy of the scanning device itself. If the errors of some code points are abnormally large, it means that the scanned object itself has deformed during the measurement process, and the area where the code points with large errors are located indicates the corresponding area where the scanned object has deformed.

[0073] Figure 3 is a schematic diagram of the coding points and the non-coding marking points in this embodiment. Figure 3 As shown, taking the left and right frames as an example, Figure 3 The pink dots in the left and right frames are the corresponding coded points, the black dots are the corresponding non-coded marker points, and the red dots are the points that cannot be accurately matched due to deformation or precision loss. Without the constraints of coded points, the red dots in the left and right frames are likely to be layered or shuffled. Therefore, in this embodiment, based on the matching of coded points, more marker point matching pairs of non-coded marker points are established, which can reduce the stratification of marker points.

[0074] In this embodiment, coding points and non-coding marking points are set on the scanned object, and the topological relationship between coding points and non-coding points is utilized to establish more coding point matching pairs based on the coding point matching pairs. Finally, the error in the neighborhood is characterized by the error of each coding point in multiple frame images in the global coordinate system. This enables the determination of the deformed area based on the coding point error at a lower cost, thereby solving the problem of being unable to determine the deformed area.

[0075] In some of the embodiments, the above-mentioned matching of the coded point is used as a reference to match the non-coded marker points in the multiple frames of images to obtain the marker point matching pair, which may include the following steps:

[0076] Identify and determine the three-dimensional coordinates of the coded points and non-coded marker points in the image; determine the topological relationship between the coded points and non-coded marker points in the image based on the three-dimensional coordinates; match the non-coded marker points based on the topological relationship and the coded point matching pairs to obtain the marker point matching pairs.

[0077] Specifically, the three-dimensional coordinates of the coding points and non-coding marker points in the image are obtained through binocular reconstruction and recognition. A topological relationship is formed between the three-dimensional coordinates of the coding points and the three-dimensional coordinates of the non-coding marker points in each image, and the topological relationship usually includes one coding point and multiple non-coding marker points. For example: the distance, angle and other relationships between the coding point and other non-coding marker points are calculated with the coding point as the center, so as to determine the topological relationship based on the three-dimensional coordinates. When the coding point matching pair has been specified, all the topological relationships in the multiple frames of images are compared one by one, and the topological relationship between the coding points and the non-coding marker points is used to match the corresponding non-coding marker points in the multiple frames of images to obtain the marker point matching pair.

[0078] In this embodiment, through the topological relationship between the coded points and the non-coded marker points in the image, it is possible to use limited coded point matching pairs to establish more marker point matching pairs in multiple frame images. The marker point matching pairs can then be used to more accurately calculate the errors between coordinates and better detect deformation areas.

[0079] In some of the embodiments, since the coding points are distributed on the surface of the scanned object and the scanned object is a large piece, some images may not contain coding points during the scanning process. In this case, the following steps can be used to obtain the marker point matching pairs.

[0080] When it is confirmed that the positional relationship of multiple frame images in the global coordinate system meets the preset conditions and the image does not contain the coded point, select any non-coded mark point in an image as the first principal point; and determine the topological relationship between the first principal point and the remaining non-coded mark points in the image, and repeatedly select the first principal point until all non-coded mark points in the image are traversed; based on the topological relationship, determine the first principal point corresponding to the second principal point in the remaining image; and match the remaining non-coded mark points in the multiple frame images based on the matching pair of the first principal point and the second principal point to obtain a matching pair of mark points.

[0081] Specifically, in the process of scanning frame by frame to obtain the image of the scanned object, the trajectory of the 3D scanning device is recorded, and loop detection is performed when the position of the current frame image is close to the position of a certain historical frame image. During the scanning process, each frame image is usually converted from the current camera coordinate system to the global coordinate system for splicing, and the position relationship can be judged based on the distance of multiple frames in the global coordinate system to determine whether it meets the preset conditions.

[0082] During the frame-by-frame scanning process, if no coding point is identified in the image, any non-coding marker point is selected from any image as the first principal point, and the adjacent non-coding marker points in the neighborhood are searched based on a certain distance (as the slave points of the first principal point), and the distances between the first principal point and all its slave points and the corresponding slave point indexes are recorded, that is, the topological relationship between the first principal point and the remaining non-coding marker points in the image. If there are non-coding marker points outside the neighborhood of the first principal point in the image, another non-coding marker point is selected as the first principal point again, and the adjacent non-coding marker points in the neighborhood are searched based on a certain distance until all non-coding marker points in the image are traversed to obtain a topological relationship including the distance. In the remaining images, any non-coding marker point is selected as the second principal point in the same way, and the neighborhood is searched at a certain distance to obtain the three-dimensional distance between the second principal point and the slave points in the remaining images. Based on the topological relationship in the image where the first principal point is located, combined with the three-dimensional distance in the remaining image, the first principal point corresponding to the second principal point in the remaining image is determined, thereby obtaining a matching pair of the first principal point and the second principal point. Based on the matching pairs, the coordinate transfer matrix is ​​calculated, and all non-encoded markers in the multi-frame images are rotated according to the matrix, thereby establishing more matching pairs of markers. According to the matching pairs of markers, the corresponding stitching matrix is ​​calculated; in the global coordinate system, the error of the encoding point is determined according to the 3D distance between the 3D coordinates of the corresponding non-encoded markers in the multi-frame images, so as to detect the corresponding area where the scanned object is deformed.

[0083] Taking frame A and frame B as an example, the positional relationship between frame A and frame B in the global coordinate system meets the preset conditions, and the image does not contain coded points. Select any non-coded marker point in frame A as the first principal point, and based on a certain distance, obtain the non-coded marker point (slave point) adjacent to the first principal point through neighborhood search, and record the distance of all slave points and the corresponding slave point index. Repeatedly select other non-coded marker points as the first principal point until all non-coded marker points in frame A are traversed, thereby obtaining the distance table of all non-coded marker points in frame A.

[0084] In the B frame, any non-coded marker point is selected as the second principal point. According to the search distance in the A frame, the neighborhood search is performed for non-coded marker points (slave points) adjacent to the second principal point, and the distances of all slave points and the corresponding slave point indexes are recorded. Based on the three-dimensional distances of the master and slave points in the B frame, the master and slave point relationships with the same three-dimensional distances are traversed in the A frame, thereby determining the first principal point corresponding to the second principal point, and obtaining a matching pair of the first principal point and the second principal point. Based on the matching pairs, the coordinate transfer matrix is ​​calculated, and all non-coded marker points in the multi-frame images are rotated according to this matrix, thereby establishing more matching pairs of marker points.

[0085] Through this embodiment, even if some images may not contain coding points during the scanning process, matching pairs of marker points in multiple frames of images can be established to calculate the splicing matrix.

[0086] In some of the embodiments, the above steps of converting the three-dimensional coordinates of the code points in the camera coordinate system to the global coordinate system according to the code point matching pairs include the following steps:

[0087] According to the matching pairs of code points and the three-dimensional coordinates of the code points in the camera coordinate system and the global coordinate system, a stitching matrix is ​​calculated; based on the stitching matrix, the three-dimensional coordinates of the code points in the camera coordinate system are converted to the global coordinate system.

[0088] Specifically, the three-dimensional coordinates of the code points in multiple frames are obtained through binocular reconstruction, including the three-dimensional coordinates of the code points in the current frame image in the camera coordinate system and the three-dimensional coordinates of the code points in the historical frame images in the global coordinate system. According to the above-mentioned mandatory code point matching pairs, the splicing matrix between the three-dimensional coordinates of the code points in the current frame image in the camera coordinate system and the three-dimensional coordinates of the code points in the historical frame images in the global coordinate system is calculated to convert the three-dimensional coordinates of the code points in the current frame image in the camera coordinate system to the global coordinate system.

[0089] By establishing more matching pairs of encoding points between multiple frames of images in this embodiment, a more accurate splicing matrix for coordinate system conversion can be calculated.

[0090] In some embodiments, the method further comprises:

[0091] Determine the origin of the camera coordinate system where the multiple frame images are located converted to the global coordinate system; if the distance between the origins is within a preset threshold, it is confirmed that the positional relationship of the multiple frame images in the global coordinate system meets the preset conditions.

[0092] Specifically, during the scanning process, each frame of the image is usually converted from the current camera coordinate system to the global coordinate system for splicing, and the distance between multiple frames of the image in the global coordinate system can be used to determine whether the positional relationship meets the preset conditions. Among them, the camera coordinate system where the image is located is converted to the global coordinate system to obtain the origin in the global coordinate system. According to the origins in the global coordinate system corresponding to the multiple frames of the image, it is determined whether the distance between the origins is within the preset threshold. If so, it is confirmed that the positional relationship between the multiple frames of the image in the global coordinate system meets the preset conditions, and further establishes a coding point matching pair and / or a marking point matching pair. Otherwise, it does not meet the preset conditions.

[0093] By using the preset condition established by the distance between the origins in the global coordinate system of multiple frame images in this embodiment, loop detection can be performed more accurately when the position at which the current frame image is acquired is close to the position at which a certain historical frame image is acquired.

[0094] The present embodiment is described and illustrated below through preferred embodiments.

[0095] Figure 4 is a flow chart of the deformation detection method of this embodiment. Figure 4 As shown, the method comprises the following steps:

[0096] Step S401, obtaining multiple frames of images of a scanned object; wherein, coded points and non-coded marking points are set on the surface of the scanned object.

[0097] Step S402, determining the origin of the camera coordinate system where the multiple frames of images are located is converted to the global coordinate system; if the distance between the origins is within a preset threshold, it is confirmed that the positional relationship of the multiple frames of images in the global coordinate system meets the preset conditions.

[0098] When the image contains coding points, steps S403 to S404 are executed; when the image does not contain coding points, steps S405 to S406 are executed.

[0099] Step S403, when it is confirmed that the positional relationship of the multiple frames of images in the global coordinate system meets the preset conditions and the images contain coded points, the coded points in the multiple frames of images are matched based on the coded values ​​of the coded points to obtain matching pairs of coded points.

[0100] Step S404, identifying and determining the three-dimensional coordinates of the coding points and non-coding marker points in the image; determining the topological relationship between the coding points and non-coding marker points in the image based on the three-dimensional coordinates; matching the non-coding marker points based on the topological relationship and the coding point matching pairs to obtain marker point matching pairs.

[0101] Step S405, when it is confirmed that the positional relationship of the multiple frame images in the global coordinate system meets the preset conditions and the image does not contain the coding point, select any non-coding mark point in an image as the first principal point; and determine the topological relationship between the first principal point and the remaining non-coding mark points in the image, and repeatedly select the first principal point until all non-coding mark points in the image are traversed.

[0102] Step S406, based on the topological relationship, determine the first principal point corresponding to the second principal point in the remaining image; and match the remaining non-encoded marker points in the multi-frame image based on the matching pair of the first principal point and the second principal point to obtain a matching pair of marker points.

[0103] Step S407: Calculate and obtain a corresponding splicing matrix based on the matching pairs of marker points.

[0104] Step S408, in the global coordinate system, according to the three-dimensional distance between the three-dimensional coordinates of the corresponding non-encoded marking points in the multiple frames of images, the error of the encoding point is determined to detect the corresponding area where the scanned object is deformed.

[0105] Through this embodiment, when the positions of multiple frames of images are close in the global coordinate system, the coding points can be used to specify the coding point matching pairs, and the non-coding marker points in the multiple frames of images can be matched based on the coding point matching pairs to obtain the marker point matching pairs. By converting the three-dimensional coordinates of the non-coding marker points to the global coordinate system, the coding point error is determined by the error of each non-coding marker point in the multiple frames of images in the global coordinate system to characterize the error in the neighborhood, so that even if the splicing process continuously accumulates errors, resulting in the inability to splice normally and the delamination of the marker points, the deformed area can be determined at a lower cost based on the coding point errors, solving the problem of being unable to determine the deformed area.

[0106] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0107] In this embodiment, a deformation detection device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. The terms "module", "unit", "subunit", etc. used below can implement a combination of software and / or hardware of predetermined functions. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0108] Figure 5 is a structural block diagram of the deformation detection device in this embodiment, such as Figure 5 As shown, the device comprises:

[0109] The acquisition module 10 is used to acquire multiple frames of images of the scanned object; wherein the surface of the scanned object is provided with coding points;

[0110] The matching module 20 is used to match the code points in the multiple frames of images based on the code values ​​of the code points to obtain matching pairs of code points when it is confirmed that the positional relationship of the multiple frames of images in the global coordinate system meets the preset conditions and the images contain code points;

[0111] The deformation detection module 30 is used to convert the three-dimensional coordinates of the coding points in the camera coordinate system to the global coordinate system based on the coding point matching pairs; in the global coordinate system, the error of the coding point is determined according to the three-dimensional distance of the corresponding three-dimensional coordinates of the coding points between multiple frame images to detect the corresponding area where the scanned object is deformed.

[0112] Through the device provided in this embodiment, when the positions of multiple frames of images are similar in the global coordinate system, the coding points can be used to specify the coding point matching pairs, and the three-dimensional coordinates of the coding points can be converted to the global coordinate system. The errors in the neighborhood are represented by the errors of each coding point in the multiple frames of images in the global coordinate system. In this way, even if the splicing process continuously accumulates errors, resulting in the inability to splice normally and the delamination of the marking points, the area where the deformation occurs can be determined based on the errors of the coding points, solving the problem of being unable to determine the area where the deformation occurs. In addition, using the coding point matching pairs to calculate the splicing matrix between images can reduce the failure of coding point loop detection.

[0113] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0114] In this embodiment, a deformation detection system is provided, including: a three-dimensional scanning device and a processing device;

[0115] The three-dimensional scanning device is used to scan the scanning object; the processing device is connected to the three-dimensional scanning device and is used in the deformation detection method in the above embodiment.

[0116] Specifically, the scanning object can be industrial products, agricultural products, and construction products, etc. The three-dimensional scanning device is a binocular vision scanner, which can be a handheld three-dimensional scanning device or a tracking scanning device. For example, in a deformation detection system, it can include multiple three-dimensional scanning devices, such as the three-dimensional scanning device is a plurality of handheld three-dimensional scanning devices, such as the three-dimensional scanning device is a cascaded tracking scanning device, and another example is a handheld three-dimensional scanning device and another is a tracking scanning device. The three-dimensional scanning device can have multiple scanning modes; for example: deformation detection mode and scanning mode, etc.

[0117] Among them, the processor device may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The processing device may include one or more processors and a memory for storing data; the memory can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the deformation detection method in this embodiment, and the processor executes various functional applications and data processing by running the computer program stored in the memory, that is, implementing the above method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0118] In this embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0119] Optionally, the computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0120] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0121] In addition, in combination with the deformation detection method provided in the above embodiments, a storage medium may be provided in this embodiment to implement the deformation detection method. The storage medium stores a computer program, and when the computer program is executed by a processor, any deformation detection method in the above embodiments is implemented.

[0122] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0123] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0124] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

[0125] The term "embodiment" in this application refers to a specific feature, structure or characteristic described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.

[0126] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A deformation detection method, characterized in that: include: Acquire multiple frames of images of a scanned object; wherein encoding points are set on the surface of the scanned object; When it is confirmed that the positional relationship of the multiple frames of the image in the global coordinate system meets the preset conditions and the image contains the code point, matching the code points in the multiple frames of the image based on the code values ​​of the code point to obtain a code point matching pair; According to the matching pairs of code points, the three-dimensional coordinates of the code points in the camera coordinate system are converted to the global coordinate system; In the global coordinate system, the error of the encoding point is determined according to the three-dimensional distance between the three-dimensional coordinates of the corresponding encoding points in the multiple frames of the image, so as to detect the corresponding area of ​​the scanned object where the deformation occurs.

2. The deformation detection method according to claim 1, characterized in that: The surface of the scanned object is also provided with non-coding marking points; the method further comprises: Based on the coding point matching pair, non-coding marker points in multiple frames of the image are matched to obtain marker point matching pairs; According to the matching pairs of marker points, a corresponding stitching matrix is ​​calculated; the stitching matrix is ​​used to transform the three-dimensional coordinates of the non-encoded marker points in the camera coordinate system into the global coordinate system; In the global coordinate system, the error of the encoding point is determined according to the three-dimensional distance between the three-dimensional coordinates of the corresponding non-encoding marking points in the multiple frames of the image, so as to detect the corresponding area where the scanned object is deformed.

3. The deformation detection method according to claim 2, characterized in that: The step of matching the non-coded mark points in the multiple frames of the image based on the coded point matching pair to obtain the mark point matching pair includes: Identify and determine the three-dimensional coordinates of the coded points and the non-coded marker points in the image; Determine the topological relationship between the coded points and the non-coded marker points in the image according to the three-dimensional coordinates; The non-encoded marking points are matched according to the topological relationship and the encoding point matching pairs to obtain the marking point matching pairs.

4. The deformation detection method according to claim 2, characterized in that: Also includes: When it is confirmed that the positional relationship of the multiple frames of the image in the global coordinate system meets the preset conditions and the image does not contain the encoding point, any non-encoding mark point in one of the images is selected as the first principal point; and determining a topological relationship between the first principal point and the remaining non-coding marker points in the image, and repeatedly selecting the first principal point until all non-coding marker points in the image are traversed; Based on the topological relationship, determining the first principal point corresponding to the second principal point in the remaining image; Based on the matching pair of the first principal point and the second principal point, the remaining non-encoded marker points in the multiple frames of the image are matched to obtain matching pairs of marker points.

5. The deformation detection method according to claim 1, characterized in that: The converting the three-dimensional coordinates of the code point in the camera coordinate system to the global coordinate system according to the code point matching pair includes: Calculate a stitching matrix according to the matching pairs of code points and the three-dimensional coordinates of the code points in the camera coordinate system and the global coordinate system; Based on the stitching matrix, the three-dimensional coordinates of the encoding points in the camera coordinate system are converted to the global coordinate system.

6. The deformation detection method according to claim 1, characterized in that: Also includes: Determine the origin of the camera coordinate system where the multiple frames of the image are located and transform it to the global coordinate system; If the distance between the origins is within a preset threshold, it is confirmed that the positional relationship between the multiple frames of the image in the global coordinate system meets the preset conditions.

7. A deformation detection device, characterized in that: include: An acquisition module, used to acquire multiple frames of images of a scanned object; wherein encoding points are set on the surface of the scanned object; A matching module, for matching the code points in the multiple frames of the image based on the code values ​​of the code points to obtain matching pairs of code points when it is confirmed that the positional relationship of the multiple frames of the image in the global coordinate system meets a preset condition and the image contains the code points; A deformation detection module, used for converting the three-dimensional coordinates of the code points in the camera coordinate system to the global coordinate system according to the code point matching pairs; In the global coordinate system, the error of the encoding point is determined according to the three-dimensional distance of the three-dimensional coordinates of the encoding points corresponding to the multiple frames of the image, so as to detect the corresponding area of ​​the scanned object where the deformation occurs.

8. A deformation detection system, characterized in that: include: 3D scanning equipment and processing equipment; The three-dimensional scanning device is used to scan the scanning object; The processing device is connected to the three-dimensional scanning device and is used to execute the deformation detection method according to any one of claims 1 to 6.

9. A computer device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the deformation detection method according to any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the deformation detection method according to any one of claims 1 to 6 are implemented.

Images