Three-dimensional reconstruction method and device, electronic equipment and machine readable storage medium

Abstract

The application provides a three-dimensional reconstruction method, device, electronic equipment and machine readable storage medium. The method comprises: collecting images of laser lines projected on a measured object by using a binocular camera; extracting center points of each laser line image; for any group of laser line images, determining a candidate matching result between a center point in a first laser line image and a center point in a second laser line image in the group of laser line images according to an internal parameter of the binocular camera; determining a final matching result from the candidate matching result between the center point in the first laser line image and the center point in the second laser line image according to a position relationship between the center points in the same row of the laser line images and a position relationship between the light planes; and performing three-dimensional reconstruction according to the final matching result. The method can reduce the implementation complexity of three-dimensional reconstruction.

Description

Technical Field

[0001] This application relates to the field of high-precision stereo vision technology, and in particular to a three-dimensional reconstruction method, apparatus, electronic device and machine-readable storage medium. Background Technology

[0002] 3D reconstruction refers to establishing a mathematical model of a 3D object that is suitable for computer representation and processing. It is the basis for processing, manipulating and analyzing the properties of the object in a computer environment, and it is also a key technology for establishing virtual reality that expresses the objective world in a computer. Summary of the Invention

[0003] In view of this, this application provides a three-dimensional reconstruction method, apparatus, electronic device, and machine-readable storage medium.

[0004] According to a first aspect of the embodiments of this application, a three-dimensional reconstruction method is provided, comprising:

[0005] During the scanning of the object under test using a multi-line laser, a binocular camera is used to acquire images of the laser lines projected onto the object under test, so as to obtain multiple sets of laser line images; the binocular camera includes a first camera and a second camera, and a set of laser line images includes a first laser line image acquired by the first camera and a second laser line image acquired by the second camera;

[0006] The center point of each laser line image is extracted to obtain the coordinates of the center point of each laser line in each row of each laser line image.

[0007] For any set of laser line images, determine the candidate matching results between the center point in the first laser line image and the center point in the second laser line image in the set of laser line images;

[0008] Based on the positional relationship between the center points of the same row in the laser line image and the positional relationship between the light planes, the final matching result is determined from the candidate matching results between the center points in the first laser line image and the center points in the second laser line image.

[0009] Three-dimensional reconstruction is performed based on the final matching results.

[0010] According to a second aspect of the embodiments of this application, a three-dimensional reconstruction apparatus is provided, comprising:

[0011] The acquisition unit is used to acquire images of laser lines projected onto the object being measured using a binocular camera during the scanning process of the object being measured with a multi-line laser, so as to obtain multiple sets of laser line images; the binocular camera includes a first camera and a second camera, and a set of laser line images includes a first laser line image acquired by the first camera and a second laser line image acquired by the second camera;

[0012] The extraction unit is used to extract the center point of each laser line image to obtain the coordinates of the center point of each laser line in each row of each laser line image.

[0013] The first determining unit is used to determine, for any set of laser line images, the candidate matching result between the center point in the first laser line image and the center point in the second laser line image in the set of laser line images;

[0014] The second determining unit is used to determine the final matching result from the candidate matching results between the center points in the first laser line image and the center points in the second laser line image based on the positional relationship between the center points in the same row of the laser line image and the positional relationship between the light planes.

[0015] A reconstruction unit is used to perform three-dimensional reconstruction based on the final matching result.

[0016] According to a third aspect of the present application, an electronic device is provided, including a processor and a memory, the memory storing machine-executable instructions executable by the processor, the processor being configured to execute the machine-executable instructions to implement the method provided in the first aspect.

[0017] According to a fourth aspect of the embodiments of this application, a machine-readable storage medium is provided, wherein machine-executable instructions are stored therein, and when the machine-executable instructions are executed by a processor, the method provided in the first aspect is implemented.

[0018] The 3D reconstruction method of this application embodiment achieves 3D reconstruction by using a binocular camera to acquire images of laser lines projected onto the object during the scanning process of the object using a multi-line laser. This results in multiple sets of laser line images. The center point of each acquired laser line image is extracted to obtain the coordinates of the center point of each laser line in each row of each laser line image. For any set of laser line images, based on the intrinsic parameters of the binocular camera, candidate matching results are determined between the center point of the first laser line image and the center point of the second laser line image in that set of laser line images. Based on the positional relationship between the center points in the same row of the laser line images and the positional relationship between the light planes, the final matching result is determined from the candidate matching results between the center points of the first laser line image and the center points of the second laser line image. Then, 3D reconstruction is performed based on the final matching result. This achieves 3D reconstruction without the need for external coded patterns or a third camera, thus reducing the complexity of 3D reconstruction. Attached Figure Description

[0019] Figure 1 This is a schematic flowchart of a three-dimensional reconstruction method provided in an embodiment of this application;

[0020] Figure 2 This is a schematic diagram of the architecture of a multi-line laser sensor system provided in an embodiment of this application;

[0021] Figure 3 This is a schematic diagram of the triggering timing of a camera and a galvanometer provided in an embodiment of this application;

[0022] Figure 4 This is a schematic diagram of images captured by the left and right cameras provided in an embodiment of this application;

[0023] Figure 5 This is a schematic flowchart of a three-dimensional reconstruction method provided in an embodiment of this application;

[0024] Figure 6 This is a schematic diagram of longest growing subsequence matching provided in an embodiment of this application;

[0025] Figure 7 This is a schematic diagram of the structure of a three-dimensional reconstruction device provided in an embodiment of this application;

[0026] Figure 8 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0027] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0028] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0029] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, and to make the above-mentioned objectives, features and advantages of the embodiments of this application more apparent and understandable, the technical solutions in the embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0030] It should be noted that the sequence number of each step in the embodiments of this application does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0031] Please see Figure 1This is a flowchart illustrating a three-dimensional reconstruction method provided in an embodiment of this application, as shown below. Figure 1 As shown, the three-dimensional reconstruction method may include the following steps:

[0032] Step S100: During the scanning of the object under test using a multi-line laser, a binocular camera is used to acquire images of the laser lines projected onto the object under test, so as to obtain multiple sets of laser line images; the binocular camera includes a first camera and a second camera, and a set of laser line images includes a first laser line image acquired by the first camera and a second laser line image acquired by the second camera.

[0033] In this embodiment of the application, in order to achieve three-dimensional reconstruction of the object under test, a multi-line laser can be used to scan the object under test. During the scanning process of the object under test using a multi-line laser, a binocular camera is used to acquire images of the laser lines projected on the object under test to obtain multiple sets of laser line images. Based on the obtained laser line images, three-dimensional reconstruction is performed to obtain the point cloud of the surface of the object under test.

[0034] For example, one camera in a stereo camera can be referred to as the first camera, and the other camera can be referred to as the second camera.

[0035] For example, the first camera is the left eye camera (or left camera) in a binocular camera system, and the second camera is the right eye camera (or right camera) in a binocular camera system; or, the first camera is the right eye camera in a binocular camera system, and the second camera is the left eye camera in a binocular camera system.

[0036] A set of laser line images may include a laser line image captured by a first camera (which may be referred to as a first laser line image) and a laser line image captured by a second camera (which may be referred to as a second laser line image).

[0037] For example, the first camera can be the left eye camera in a binocular camera, and the second camera can be the right eye camera in a binocular camera. A set of laser line images can include a laser line image captured by the left eye camera (i.e., the first laser line image) and a laser line image captured by the right eye camera (i.e., the second laser line image).

[0038] It should be noted that, in this embodiment of the application, in order to avoid the influence of external ambient light on the quality of the laser line image, the above-mentioned binocular camera can be a monochrome camera, and filters can be deployed to filter out the light that affects the quality of the laser line image.

[0039] However, it should be recognized that in the embodiments of this application, the binocular camera is not limited to a black and white camera, but can also be a color camera, and the influence of the external environment can be reduced through other strategies (or no additional special processing may be required if the external environment is favorable). The embodiments of this application do not limit this.

[0040] Step S110: Extract the center point of each laser line image to obtain the coordinates of the center point of each laser line in each row of each laser line image.

[0041] In this embodiment of the application, the center point of the acquired laser line image can be extracted to obtain the coordinates of the center point of each laser line in each row of each laser line image.

[0042] For example, the grayscale centroid method or the Stger method can be used to extract the center point.

[0043] For example, the coordinates of the center point of the laser line can be subpixel level coordinates.

[0044] It should be noted that, in order to optimize the quality of the acquired laser line image and thus improve the effect of center point extraction in subsequent processes, the acquired laser line image can be preprocessed, such as image filtering and contrast enhancement.

[0045] In addition, epipolar correction can be performed on the acquired laser line images to remove image distortion from the left and right eye images and correct the same physical point to the same line in the image.

[0046] Step S120: For any set of laser line images, based on the intrinsic parameters of the binocular camera, determine the candidate matching result between the center point in the first laser line image and the center point in the second laser line image in the set of laser line images.

[0047] In this embodiment of the application, in order to achieve three-dimensional reconstruction of the object under test, when the laser line image is obtained in the above manner and the coordinates of the center point are extracted, the center points of the first laser line image and the second laser line image in the same group can be paired so that three-dimensional reconstruction can be performed based on the paired center points.

[0048] For example, for any set of laser line images, the center points in the first laser line image and the center points in the second laser line image can be paired based on the intrinsic parameters of the binocular camera to determine the candidate matching results between the center points in the first laser line image and the center points in the second laser line image.

[0049] For example, for any center point in the first laser line image, a candidate matching center point can be determined in the second laser line image based on the intrinsic parameters of the binocular camera, thereby obtaining the candidate matching result between the center point in the first laser line image and the center point in the second laser line image.

[0050] Alternatively, for any center point in the second laser line image, a candidate matching center point can be determined in the first laser line image based on the intrinsic parameters of the binocular camera, thereby obtaining the candidate matching result between the center point in the first laser line image and the center point in the second laser line image.

[0051] It should be noted that, in the embodiments of this application, center point extraction of laser line images is not limited to being performed after all laser line images have been acquired. For example, assuming that N (N≥2) sets of laser line images need to be acquired, center point extraction can be performed concurrently on the acquired laser line images after at least one set of laser line images has been acquired, and laser line image acquisition can continue.

[0052] Step S130: Based on the positional relationship between the center points of the same row in the laser line image and the positional relationship between the light planes, determine the final matching result from the candidate matching results between the center points in the first laser line image and the center points in the second laser line image.

[0053] In this embodiment of the application, it is considered that when the center point in the first laser line image is mapped to the second laser line image through the light plane, the positional relationship between the center points in the same row and the positional relationship between the corresponding light planes are regular.

[0054] For example, ignoring the hollowed-out object, when three-dimensional points on different light planes are projected onto the image, the relative positional relationship on the image is consistent with the relative positional relationship of the light planes.

[0055] Accordingly, the final matching result can be determined from the candidate matching results between the center points in the first laser line image and the center points in the second laser line image, based on the positional relationship between the center points in the same row of the laser line image and the positional relationship between the light planes.

[0056] Step S140: Perform three-dimensional reconstruction based on the final matching results.

[0057] In this embodiment of the application, after the center point in the first laser line image and the center point in the second laser line image are paired in the manner described above, three-dimensional reconstruction can be performed based on the final matching result to determine the three-dimensional point cloud of the surface of the object being measured.

[0058] As can be seen, by using a multi-line laser to scan the object under test, a binocular camera is used to acquire images of the laser lines projected onto the object to obtain multiple sets of laser line images. The center point of each acquired laser line image is extracted to obtain the coordinates of the center point of each laser line in each row of each laser line image. For any set of laser line images, based on the intrinsic parameters of the binocular camera, the candidate matching results between the center point of the first laser line image and the center point of the second laser line image in that set of laser line images are determined. Based on the positional relationship between the center points in the same row of the laser line images and the positional relationship between the light planes, the final matching result is determined from the candidate matching results between the center points of the first laser line image and the center points of the second laser line image. Then, based on the final matching result, three-dimensional reconstruction is performed. Three-dimensional reconstruction is achieved without the need for external coding patterns or a third camera, thus reducing the complexity of three-dimensional reconstruction.

[0059] In some embodiments, during the scanning of the object under test using a multi-line laser, acquiring images of the laser lines projected onto the object under test using a binocular camera may include:

[0060] Control the galvanometer to rotate to the set position;

[0061] For any set position of the galvanometer, use a binocular camera to capture an image of the laser line projected onto the object under test when the galvanometer is in that position.

[0062] For example, a galvanometer can be used to change the propagation direction of the laser lines emitted by a multi-line laser. By controlling the rotation of the galvanometer, the laser lines emitted by the multi-line laser can scan the outer surface of the object under test, thereby achieving three-dimensional reconstruction of the object under test.

[0063] Accordingly, the galvanometer can be controlled to rotate to a set position. For any set position of the galvanometer, a binocular camera is used to capture an image of the laser line projected onto the object under test when the galvanometer is in that position.

[0064] For example, the rotation position of the galvanometer can be determined based on the angle between adjacent laser lines emitted by the multi-line laser.

[0065] For example, assuming that the angle between adjacent laser lines emitted by a multi-line laser is θ, then during a complete scan, the angle covered by all positions during the rotation of the galvanometer must be no less than θ.

[0066] It should be noted that, in the embodiments of this application, the propagation direction of the laser line emitted by the multi-line laser is not limited to changing the propagation direction of the laser line emitted by the multi-line laser by using a galvanometer. The propagation direction of the laser line emitted by the multi-line laser can also be changed by controlling the rotation of the multi-line laser to achieve scanning of the object under test. The specific implementation is not limited here.

[0067] In some embodiments, determining the candidate matching result between the center point in the first laser line image and the center point in the second laser line image in any set of laser line images, based on the intrinsic parameters of the binocular camera, may include:

[0068] For any set of laser line images, based on the coordinates of the center point in the first laser line image of the target in the set of laser line images, and the pre-calibrated intrinsic parameters of the first camera and the light plane equation, the candidate matching center point of the center point in the first laser line image of the target in the second laser line image of the target in the set of laser line images is determined; wherein, the light plane equation is the equation of the light plane in the coordinate system of the first camera;

[0069] The determination of the final matching result from the candidate matching results between the center points of the first laser line image and the center points of the second laser line image, based on the positional relationship between the center points of the same row in the laser line image and the positional relationship between the light planes, may include:

[0070] Based on the positional relationship between the center points of the same row in the first laser line image and the positional relationship between the light planes, the final matching center point is determined from the candidate matching center points in the second laser image from the center point of the row, and the final matching result is obtained.

[0071] For example, for any set of laser line images, the center point coordinates in the first laser line image (which may be referred to as the target first laser line image) in the set of laser line images, as well as the pre-calibrated intrinsic parameters and light plane equation of the first camera, can be used to determine the candidate matching center point of the center point in the target first laser line image in the second laser line image (referred to as the target second laser line image in this document).

[0072] For example, when calibrating the light plane, the determined equation of the light plane can be the equation of the light plane in the coordinate system of the first camera.

[0073] For example, if the center point of the first laser line image in the same group of laser line images is determined as a candidate matching center point in the second laser line image in the manner described above, the final matching center point can be determined from the candidate matching center points in the second laser image based on the positional relationship between the center points of the same row in the first laser line image and the positional relationship between the light planes, thus obtaining the final matching result.

[0074] It should be noted that, in the embodiments of this application, for any set of laser line images, in the process of determining the candidate matching result between the center point in the first laser line image and the center point in the second laser line image in the set of laser line images based on the intrinsic parameters of the binocular camera, the coordinates of the center point in the target second laser line image in the set of laser line images, as well as the pre-calibrated intrinsic parameters of the second camera and the light plane equation, can be used to determine the candidate matching center point of the center point in the target first laser line image in the set of laser line images. Thus, the candidate matching result between the center point in the first laser line image and the center point in the second laser line image is obtained. The specific implementation method is similar to the method described in the above embodiments, and will not be repeated here.

[0075] In one example, the above-mentioned determination of the candidate matching center point of the center point in the first laser line image of the target in the set of laser line images, based on the coordinates of the center point in the first laser line image of the set of laser line images, and the pre-calibrated intrinsic parameters and light plane equation of the first camera, includes:

[0076] For any center point in any row of the first laser line image of the target, the three-dimensional coordinates corresponding to the center point are determined based on the coordinates of the center point, as well as the pre-calibrated intrinsic parameters of the first camera and the light plane equation.

[0077] Based on the three-dimensional coordinates and the pre-calibrated intrinsic parameters of the second camera, the projection coordinates of the three-dimensional coordinates in the second laser line image of the target are determined;

[0078] The center point in the second laser line image of the target, whose distance from the projected coordinates is within a preset distance range, is determined as the candidate matching center point.

[0079] For example, in the process of determining the matching center point of the center point in the first laser line image of the target and the center point in the second laser line image of the target, for any center point in any row of the first laser line image of the target, the three-dimensional coordinates corresponding to the center point can be determined based on the coordinates of the center point, as well as the pre-calibrated intrinsic parameters of the first camera and the light plane equation.

[0080] For example, the three-dimensional coordinates corresponding to the intersection of the line connecting the center point and the optical center with the optical plane can be determined as the three-dimensional coordinates corresponding to the center point.

[0081] For example, based on the three-dimensional coordinates corresponding to the center point in the first laser line image of the target, and the pre-calibrated intrinsic parameters of the second camera, the three-dimensional coordinates can be projected onto the second laser line image of the target to determine the projection coordinates of the three-dimensional coordinates in the second laser line image of the target. The center point in the second laser line image of the target that is within a preset distance range (which can be a pixel distance, and its specific value can be set according to the actual scene) is determined as the candidate matching center point.

[0082] In one example, determining the three-dimensional coordinates of the center point based on its coordinates, along with the pre-calibrated intrinsic parameters of the first camera and the light plane equation, can include:

[0083] Based on the scanning position corresponding to the first laser line image of the target, determine the target light plane equation corresponding to each laser line in the first laser line image of the target at that scanning position;

[0084] For any target light plane equation, the three-dimensional coordinates corresponding to the center point are determined based on the coordinates of the center point, the intrinsic parameters of the pre-calibrated first camera, and the target light plane equation.

[0085] For example, in order to determine the three-dimensional coordinates of each center point in the first laser line image of the target, the scanning position corresponding to the first laser line image of the target can be determined first.

[0086] For example, taking the change of the propagation direction of the laser line emitted by a multi-line laser by using a galvanometer as an example, the different positions of the galvanometer during the rotation process (i.e., the positions set above) correspond to different scanning positions. A binocular camera can be used to acquire laser line images at different scanning positions and record the correspondence between the acquired laser line images and the scanning positions.

[0087] For example, during the optical plane calibration process, for any scanning position, the optical plane equations corresponding to the multiple laser lines emitted by the multi-line laser can be calibrated.

[0088] Taking an n (n≥2) line laser as an example, for any scanning position, the equations of the light planes of n light planes can be determined.

[0089] For example, the light plane equation (which can be called the target light plane equation) corresponding to each laser line in the target first laser line image can be determined based on the scanning position corresponding to the target first laser line image.

[0090] For any target light plane equation, the three-dimensional coordinates corresponding to the center point are determined based on the coordinates of the center point, the intrinsic parameters of the pre-calibrated first camera, and the target light plane equation.

[0091] For example, the three-dimensional coordinates corresponding to the intersection of the line connecting the center point and the optical center with the target optical plane can be determined as the three-dimensional coordinates corresponding to the center point.

[0092] In one example, determining the center point in the target second laser line image whose distance from the projected coordinates is within a preset distance range as the candidate matching center point may include:

[0093] The center point of the target region in the second laser line image of the target, whose distance from the projected coordinates is within a preset distance range, is determined as the candidate matching center point; wherein, the target region is determined based on the measured distance.

[0094] For example, considering that the measurement distance is fixed, for any center point in the first laser line image of a set of laser line images, the range of the area where the matching center point in the second laser line image of the same set is located can be determined based on the measurement distance.

[0095] Accordingly, in order to improve the efficiency of center point matching, for any center point in the first laser line image of the target, the region in the second laser line image of the target (referred to as the target region in this paper) where the candidate matching center point of the center point is located can be determined based on the measured distance.

[0096] For any center point in the first laser line image of the target, after determining the projection coordinates of the center point in the second laser line image of the target in the manner described above, the center point in the target area of ​​the second laser line image of the target that is within a preset distance range from the projection coordinates can be determined as a candidate matching center point, so as to achieve the preliminary screening of candidate matching center points.

[0097] In some embodiments, determining the final matching center point from candidate matching center points in the second laser image based on the positional relationship between the center points of the same row in the first laser line image and the positional relationship between the light planes may include:

[0098] Based on the coordinates of the center point of the row, sort the center points of the row according to the target order; wherein, the target order includes the order of coordinates from smallest to largest, or the order of coordinates from largest to smallest;

[0099] Based on the candidate matching center points in the second laser image after the sorting of the center points in the row, and the positional relationship between the light planes corresponding to each candidate matching center point, the candidate matching center point combination with the largest number of center points is determined as the final matching center point combination, and the center point in the final matching center point combination is determined as the final matching center point; wherein, for any candidate matching center point combination, the positional relationship between the light planes corresponding to each center point in the candidate matching center point combination is consistent with the target order.

[0100] For example, considering that, ignoring the hollowed-out object, the relative positional relationship of three-dimensional points on different light planes projected onto the image is consistent with the relative positional relationship of the light planes.

[0101] For example, suppose there are two center points p1 and p2 in the same row of an image, and p1 is to the left of p2. Then the light plane corresponding to p1 should also be to the left of the light plane corresponding to p2.

[0102] Accordingly, after determining the candidate matching center points of each center point in the first laser line image in the second laser line image in the manner described above, for any center point in the first laser line image, the center points of that row can be sorted according to the target order based on the coordinates of the center points of that row.

[0103] It should be noted that the center point of any row in the first laser line image mentioned above includes the center point of all first laser line images in that row.

[0104] For example, assuming there are s scanning positions, the center point of any row in the first laser line image includes the center point of that row in s first laser line images.

[0105] For example, if the center points of a row in the first laser line image are sorted according to the target sorting, then the candidate matching center points in the second laser image can be selected from the candidate matching center points based on the candidate matching center points after the row sorting, and the positional relationship between the light planes corresponding to each candidate matching center point, and the center points whose positional relationship between the light planes is consistent with the target order, and a candidate matching center point combination can be constructed.

[0106] For example, for any candidate matching center point combination, the positional relationship between the light planes corresponding to each center point in the candidate matching center point combination is consistent with the target order.

[0107] For example, the center points of the same row in the first laser line image can be sorted in ascending order of their coordinates (coordinates in the image coordinate system) (i.e., from left to right). Similarly, the light planes corresponding to each laser line at different scanning positions can be numbered from left to right (numbering from smallest to largest).

[0108] For each center point after sorting in this row, when determining the center point to be added to the candidate matching center point combination from the candidate matching center points in the second laser line image according to the sorting results of each center point, it is necessary to ensure that the number of the light plane corresponding to the determined center point is increasing.

[0109] For example, according to the sorting result of each center point in the first laser line image, center points can be selected sequentially from the candidate matching center points in the second laser line image according to the target order to form a candidate matching center point combination. The candidate matching center point combination with the largest number of center points is determined as the final matching center point combination, and the center point in the final matching center point combination is determined as the final matching center point.

[0110] For example, suppose there are 10 center points in the same row of the first laser line image, and these 10 center points are arranged from left to right as P1, P2, ..., P10. Given the candidate matching center points for each center point in the second laser line image, we can iterate through the candidate matching center points of P1, select one center point (which can be called target center point 1), and then select a center point from the candidate matching center points of P2 whose corresponding light plane is to the right of the light plane corresponding to target center point 1 (which can be called target center point 2, i.e., the number of the light plane corresponding to target center point 2 is greater than that of the light plane corresponding to target center point 1). If target center point 2 exists, we continue to select a center point from the candidate matching center points of P3 whose corresponding light plane is to the right of the light plane corresponding to target center point 2 (which can be called target center point 3). If target center point 2 does not exist, we generate one candidate matching center point combination and reselect target center point 1.

[0111] If target center point 3 exists, then select a center point (which can be called target center point 4) to the right of the light plane corresponding to target center point 3 from the candidate matching center points of P4. If target center point 3 does not exist, generate a candidate matching center point combination and reselect target center point 2.

[0112] Similarly, the candidate matching center point combination with the largest number of center points is selected as the final matching center point combination, and each center point in the final matching center point combination is determined as the final matching center point.

[0113] For example, taking the center point from the previous example as an example, the center point can be selected sequentially from the candidate matching center points P1, P2, ..., P10 to form the final candidate set of matching center points.

[0114] For example, assuming the number of candidate matching center points for P1, P2, ..., P10 are n1, n2, ..., n10 respectively, then if there are no duplicate candidate matching center points for different center points, there are a total of n1*n2*...*n10 final matching center point candidate sets (if there are duplicates, the candidate sets including duplicate points need to be deleted). The center point in the corresponding candidate set with the increasing light plane number can be selected from these candidate sets to determine the final matching center point.

[0115] In some embodiments, the light plane equation is specified in the following manner:

[0116] The object under test is scanned using a multi-line laser at at least two different measurement distances.

[0117] During the scanning of the object under test using a multi-line laser at any measurement distance, a binocular camera is used to acquire images of the laser lines projected onto the object under test to obtain multiple sets of calibration laser line images; a set of calibration laser line images includes a first calibration laser line image acquired by a first camera and a second calibration laser line image acquired by a second camera.

[0118] The center point of each calibration laser line image is extracted to obtain the coordinates of the center point of each laser line in each row of each calibration laser line image;

[0119] Based on the auxiliary information, center point matching is performed on each group of calibration laser line images to obtain the matching center point pairs corresponding to each laser line;

[0120] For any laser line at any scanning position, the laser line is reconstructed in three dimensions based on the matching center point corresponding to the laser line to obtain three-dimensional points at multiple positions of the light plane. Then, the light plane is fitted based on the three-dimensional points at multiple positions to obtain the equation of the light plane corresponding to the laser line at the scanning position.

[0121] For example, auxiliary information during calibration could be the distance to the data acquisition location.

[0122] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, the technical solutions provided in the embodiments of this application are described below with reference to specific examples.

[0123] Please see Figure 2 This is a schematic diagram of the architecture of a multi-line laser sensor system provided in an embodiment of this application, as shown below. Figure 2 As shown, the system may include a galvanometer (including a rotating mechanism and a reflector), left and right cameras (imaging unit 1 and imaging unit 2 as shown in the figure), and a multi-line laser.

[0124] In this embodiment, based on Figure 2The system architecture shown allows a single position of the galvanometer to reconstruct a point cloud of n laser lines (corresponding to n (n≥2) line lasers). By rotating the galvanometer, the reconstructed point cloud covers the entire measurement field of view, thereby achieving a complete reconstruction of the object under test.

[0125] For example, the triggering sequence of the camera and galvanometer can be as follows: Figure 3 As shown, the galvanometer rotates to a set position, and collects a set of black and white image data each time it rotates to a new position.

[0126] For example, the rotation position of the galvanometer is consistent during repeated scans.

[0127] For example, for any rotational position of the galvanometer, the images captured by the left and right cameras can be as follows: Figure 4 As shown, by matching the laser points of the left and right cameras, the point cloud on the current n laser lines can be reconstructed.

[0128] For example, during the scanning process, due to issues such as the color, reflection, and height difference of the object being measured, not all extracted center points are laser points. Therefore, the extracted laser points cannot be directly matched.

[0129] In this embodiment, a laser point matching algorithm based on optical plane reprojection error and the longest growing subsequence can be used to solve the binocular matching problem of multi-line lasers, especially the binocular matching problem of ultra-multi-line lasers (e.g., 11 lines or more, i.e., n≥11) in complex environments (such as large depth-of-field occlusion, reflection from metal workpieces, etc.). The matching process includes optical plane reprojection screening and longest growing subsequence screening, and its flowchart can be shown as follows. Figure 5 As shown.

[0130] The following section explains some of the implementation details.

[0131] 1. Calibration

[0132] Calibration can include calibration of the intrinsic and extrinsic parameters of the left and right cameras and calibration of the light plane.

[0133] For example, the intrinsic and extrinsic parameter calibration is performed using publicly available methods such as the Zhang Zhengyou calibration method, by collecting data from the camera's checkerboard calibration board to obtain the camera's distortion coefficients, intrinsic parameter matrix, and extrinsic parameters (such as coordinate system transformation relationships between cameras).

[0134] Using the calibrated intrinsic and extrinsic parameters, laser line scanning data at multiple different distances are collected, such as scanning 68 positions (galvanometer rotation positions) with 11 lines, and collecting 11*68 sets of images.

[0135] Assuming data is collected at two different distances, a total of 2*11*68 sets of left and right eye laser line images will be collected.

[0136] During calibration, the acquisition distance or the position of the constrained light plane can be recorded (e.g., the first laser line always appears on the image) to assist in light plane matching. The center point coordinates of the left and right eye images are extracted and matched based on the auxiliary information to obtain the left and right eye matching points. Each laser line is reconstructed in three dimensions to obtain three-dimensional points at multiple positions on each light plane. Then, light plane fitting is performed on each light plane.

[0137] For example, performing light plane fitting on a light plane may include performing plane fitting (at least two different locations) or performing quadratic fitting (at least three different locations).

[0138] 2. Image preprocessing

[0139] For example, image preprocessing may include, but is not limited to, image filtering, contrast enhancement, and other processing.

[0140] 3. Polar line correction

[0141] For example, left and right eye epipolar line correction involves removing image distortion from the left and right eye images and correcting the same physical point to the same line in the image.

[0142] 4. Center point extraction

[0143] For laser lines in the left and right eye images, the center point can be extracted using methods such as gray-scale centroid method and Stger method. This extracts the center point of each multi-line laser and obtains the center point coordinates with sub-pixel accuracy.

[0144] 5. Light plane constraint matching

[0145] Light plane matching involves solving for a corresponding 3D coordinate for each center point in each row of the left image using the intrinsic parameters of the left camera and the light plane equation. This 3D coordinate is then projected onto the right image. If the point is projected near a center in the right image (within a pixel distance of less than a certain threshold), then the center point in the right image is a potential matching point for the corresponding point in the left image.

[0146] For example, the left image has h rows of pixels (image height), each row of pixels is extracted into n points (n laser lines), and each point can be matched with m potential matching points in the right image.

[0147] Optionally, Z-axis distance filtering can be performed during the optical plane matching process to filter out some candidate matches based on the Z-axis measurement range.

[0148] For example, if the measurement distance is constrained to 1m to 2m, the projection from the left image to the right image can only be onto a fixed interval, such as... Figure 4 The center point of the first line on the left can only be projected onto the position of the first or second laser line in the right figure.

[0149] 6. Longest Increasing Subsequence Matching

[0150] After the light plane matching of all scanned data is completed, there are a total of h rows and k laser lines (in the absence of interference, k = n, but in the presence of interference, k may also be greater than n or less than n) * s (galvanometer rotation positions) scanned positions of the light plane center points are matched to multiple center points in the left and right images. The ultimate goal of the matching is to obtain the three-dimensional points on each light plane of each row of pixels and match to the unique center points in the left and right images.

[0151] Exemplarily, in order to further screen the matching results, a light plane increasing constraint can be used. That is, when ignoring the hollow object, the three-dimensional points on different light planes are projected onto the image, and the relative position relationship on the image is consistent with the relative position relationship of the light planes. That is, if there are two center points p1 and p2 in the same row of the image and p1 < p2, then the light plane matched by p1 is on the left side of the light plane matched by p2. That is, among the 11 * 68 numbers from left to right, the light plane index corresponding to point p1 is less than the light plane index corresponding to point p2.

[0152] Similarly, it can also be interpreted that the pixel coordinates of the center points matched by the three-dimensional points on the left light plane are less than the pixel coordinates of the center points matched by the three-dimensional points on the right light plane.

[0153] Therefore, among the multiple center points in the left and right images that the light plane center points of h rows, k laser lines * s scanned positions are matched to, the pixel coordinates of the true matching center points should satisfy the constraint of increasing pixel coordinates. For the k * s light planes in each row, there are multiple possible potential matching results, but not every candidate match can satisfy the increasing constraint. Once there is an incorrect match, it will cause the matching pixel points to violate the increasing constraint. Through the longest increasing subsequence matching, by screening the matching points in each light plane, the matching result that constitutes the longest increasing subsequence is used as the final matching result, and its schematic diagram can be as Figure 6 [[ID=1X]]shown.

[0154] The method provided by this application has been described above. Next, the device provided by this application will be described:

[0155] Please refer to Figure 7 , which is a schematic structural diagram of a three-dimensional reconstruction device provided by an embodiment of this application. As Figure 7 shown, the three-dimensional reconstruction device may include:

[0156] The acquisition unit 710 is used to acquire images of laser lines projected onto the object being measured using a binocular camera during the scanning process of the object being measured using a multi-line laser, so as to obtain multiple sets of laser line images; the binocular camera includes a first camera and a second camera, and a set of laser line images includes a first laser line image acquired by the first camera and a second laser line image acquired by the second camera.

[0157] The extraction unit 720 is used to extract the center point of each laser line image to obtain the coordinates of the center point of each laser line in each row of each laser line image.

[0158] The first determining unit 730 is used to determine, for any set of laser line images, a candidate matching result between the center point in the first laser line image and the center point in the second laser line image, based on the intrinsic parameters of the binocular camera.

[0159] The second determining unit 740 is used to determine the final matching result from the candidate matching results between the center points in the first laser line image and the center points in the second laser line image based on the positional relationship between the center points in the same row in the laser line image and the positional relationship between the light planes.

[0160] Reconstruction unit 750 is used to perform three-dimensional reconstruction based on the final matching result.

[0161] In some embodiments, during the scanning of the object under test using a multi-line laser, the acquisition unit 710 acquires images of the laser lines projected onto the object under test using a binocular camera, including:

[0162] Control the galvanometer to rotate to the set position;

[0163] For any set position of the galvanometer, use a binocular camera to capture an image of the laser line projected onto the object under test when the galvanometer is in that position.

[0164] In some embodiments, for any set of laser line images, the first determining unit 730 determines, based on the intrinsic parameters of the binocular camera, a candidate matching result between the center point in the first laser line image and the center point in the second laser line image within the set of laser line images, including:

[0165] For any set of laser line images, based on the coordinates of the center point in the first laser line image of the target in the set of laser line images, and the pre-calibrated intrinsic parameters of the first camera and the light plane equation, the candidate matching center point of the center point in the first laser line image of the target in the second laser line image of the target in the set of laser line images is determined; wherein, the light plane equation is the equation of the light plane in the coordinate system of the first camera;

[0166] The second determining unit 740 determines the final matching result from the candidate matching results between the center points of the same row in the laser line image and the positional relationship between the light planes, including:

[0167] Based on the positional relationship between the center points of the same row in the first laser line image and the positional relationship between the light planes, the final matching center point is determined from the candidate matching center points in the second laser image from the center point of the row, and the final matching result is obtained.

[0168] In some embodiments, the first determining unit 730 determines, based on the coordinates of the center point in the target first laser line image in the set of laser line images, and the pre-calibrated intrinsic parameters and light plane equation of the first camera, a candidate matching center point in the target second laser line image in the set of laser line images, including:

[0169] For any center point in any row of the first laser line image of the target, the three-dimensional coordinates corresponding to the center point are determined based on the coordinates of the center point, as well as the pre-calibrated intrinsic parameters of the first camera and the light plane equation.

[0170] Based on the three-dimensional coordinates, and the pre-calibrated extrinsic parameters between the first and second cameras and the intrinsic parameters of the second camera, the projection coordinates of the three-dimensional coordinates in the second laser line image of the target are determined;

[0171] The center point in the second laser line image of the target, whose distance from the projected coordinates is within a preset distance range, is determined as the candidate matching center point.

[0172] In some embodiments, the first determining unit 730 determines the three-dimensional coordinates corresponding to the center point based on the coordinates of the center point, and the pre-calibrated intrinsic parameters and light plane equation of the first camera, including:

[0173] Based on the scanning position corresponding to the first laser line image of the target, determine the target light plane equation corresponding to each laser line in the first laser line image of the target at that scanning position;

[0174] For any target light plane equation, the three-dimensional coordinates corresponding to the center point are determined based on the coordinates of the center point, the intrinsic parameters of the pre-calibrated first camera, and the target light plane equation.

[0175] In some embodiments, the first determining unit 730 determines the center point in the target second laser line image whose distance from the projected coordinates is within a preset distance range as a candidate matching center point, including:

[0176] The center point of the target region in the second laser line image of the target, whose distance from the projected coordinates is within a preset distance range, is determined as the candidate matching center point; wherein, the target region is determined based on the projection error.

[0177] In some embodiments, the second determining unit 740 determines the final matching center point from candidate matching center points in the second laser image based on the positional relationship between the center points of the same row in the first laser line image and the positional relationship between light planes, including:

[0178] Based on the coordinates of the center point of the row, sort the center points of the row according to the target order; wherein, the target order includes the order of coordinates from smallest to largest, or the order of coordinates from largest to smallest;

[0179] Based on the candidate matching center points in the second laser image after the sorting of the center points in the row, and the positional relationship between the light planes corresponding to each candidate matching center point, the candidate matching center point combination with the largest number of center points is determined as the final matching center point combination, and the center point in the final matching center point combination is determined as the final matching center point; wherein, for any candidate matching center point combination, the positional relationship between the light planes corresponding to each center point in the candidate matching center point combination is consistent with the target order.

[0180] This application provides an electronic device including a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor executes the machine-executable instructions to implement the three-dimensional reconstruction method described above.

[0181] Please see Figure 8 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device may include a processor 801 and a memory 802 storing machine-executable instructions. The processor 801 and the memory 802 can communicate via a system bus 803. Furthermore, by reading and executing the machine-executable instructions corresponding to the 3D reconstruction logic in the memory 802, the processor 801 can execute the 3D reconstruction method described above.

[0182] The memory 802 mentioned in this document can be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, etc. For example, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0183] In some embodiments, a machine-readable storage medium, such as Figure 8 The memory 802 in the memory, which is a machine-readable storage medium, stores machine-executable instructions that, when executed by a processor, implement the three-dimensional reconstruction method described above. For example, the storage medium may be ROM, RAM, CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0184] It should be noted that, in this document, relational terms such as "objective" and "target" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0185] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A three-dimensional reconstruction method, characterized in that, include: During the scanning of the object under test using a multi-line laser, a binocular camera is used to acquire images of the laser lines projected onto the object under test, so as to obtain multiple sets of laser line images; the binocular camera includes a first camera and a second camera, and a set of laser line images includes a first laser line image acquired by the first camera and a second laser line image acquired by the second camera; The center point of each laser line image is extracted to obtain the coordinates of the center point of each laser line in each row of each laser line image. For any set of laser line images, based on the intrinsic parameters of the binocular camera, determine the candidate matching result between the center point in the first laser line image and the center point in the second laser line image in the set of laser line images; Based on the positional relationship between the center points of the same row in the laser line image and the positional relationship between the light planes, the final matching result is determined from the candidate matching results between the center points in the first laser line image and the center points in the second laser line image. Three-dimensional reconstruction is performed based on the final matching results; Specifically, for any set of laser line images, determining the candidate matching result between the center point in the first laser line image and the center point in the second laser line image, based on the intrinsic parameters of the binocular camera, includes: For any set of laser line images, based on the coordinates of the center point in the first laser line image of the target in the set of laser line images, and the pre-calibrated intrinsic parameters of the first camera and the light plane equation, the candidate matching center point of the center point in the first laser line image of the target in the second laser line image of the target in the set of laser line images is determined; wherein, the light plane equation is the equation of the light plane in the coordinate system of the first camera; The final matching result is determined from the candidate matching results between the center points of the first laser line image and the center points of the second laser line image based on the positional relationship between the center points of the same row in the laser line image and the positional relationship between the light planes, including: Based on the positional relationship between the center points of the same row in the first laser line image and the positional relationship between the light planes, the final matching center point is determined from the candidate matching center points in the second laser image from the center point of the row, and the final matching result is obtained.

2. The method according to claim 1, characterized in that, The process of scanning the object under test using a multi-line laser, and acquiring images of the laser lines projected onto the object using a binocular camera, includes: Control the galvanometer to rotate to the set position; For any set position of the galvanometer, use a binocular camera to capture an image of the laser line projected onto the object under test when the galvanometer is in that position.

3. The method according to claim 1, characterized in that, The step of determining the candidate matching center point of the center point in the first laser line image of the target in the set of laser line images, based on the coordinates of the center point in the first laser line image of the target and the pre-calibrated intrinsic parameters and light plane equation of the first camera, includes: For any center point in any row of the first laser line image of the target, the three-dimensional coordinates corresponding to the center point are determined based on the coordinates of the center point, as well as the pre-calibrated intrinsic parameters of the first camera and the light plane equation. Based on the three-dimensional coordinates, and the pre-calibrated extrinsic parameters between the first and second cameras and the intrinsic parameters of the second camera, the projection coordinates of the three-dimensional coordinates in the second laser line image of the target are determined; The center point in the second laser line image of the target, whose distance from the projected coordinates is within a preset distance range, is determined as the candidate matching center point.

4. The method according to claim 3, characterized in that, The process of determining the three-dimensional coordinates corresponding to the center point based on its coordinates, as well as the pre-calibrated intrinsic parameters and light plane equation of the first camera, includes: Based on the scanning position corresponding to the first laser line image of the target, determine the target light plane equation corresponding to each laser line in the first laser line image of the target at that scanning position; For any target light plane equation, the three-dimensional coordinates corresponding to the center point are determined based on the coordinates of the center point, the intrinsic parameters of the pre-calibrated first camera, and the target light plane equation.

5. The method according to claim 3, characterized in that, The step of determining the center point in the second laser line image of the target, whose distance from the projected coordinates is within a preset distance range, as the candidate matching center point includes: The center point of the target region in the second laser line image of the target, whose distance from the projected coordinates is within a preset distance range, is determined as the candidate matching center point; wherein, the target region is determined based on the projection error.

6. The method according to claim 1, characterized in that, The step of determining the final matching center point from candidate matching center points in the second laser image based on the positional relationship between the center points of the same row in the first laser line image and the positional relationship between the light planes includes: Based on the coordinates of the center point of the row, sort the center points of the row according to the target order; wherein, the target order includes the order of coordinates from smallest to largest, or the order of coordinates from largest to smallest; Based on the candidate matching center points in the second laser image after the sorting of the center points in the row, and the positional relationship between the light planes corresponding to each candidate matching center point, the candidate matching center point combination with the largest number of center points is determined as the final matching center point combination, and the center point in the final matching center point combination is determined as the final matching center point; wherein, for any candidate matching center point combination, the positional relationship between the light planes corresponding to each center point in the candidate matching center point combination is consistent with the target order.

7. A three-dimensional reconstruction device, characterized in that, include: The acquisition unit is used to acquire images of laser lines projected onto the object being measured using a binocular camera during the scanning process of the object being measured with a multi-line laser, so as to obtain multiple sets of laser line images; the binocular camera includes a first camera and a second camera, and a set of laser line images includes a first laser line image acquired by the first camera and a second laser line image acquired by the second camera; The extraction unit is used to extract the center point of each laser line image to obtain the coordinates of the center point of each laser line in each row of each laser line image. The first determining unit is used to determine, for any set of laser line images, a candidate matching result between the center point in the first laser line image and the center point in the second laser line image, based on the intrinsic parameters of the binocular camera. The second determining unit is used to determine the final matching result from the candidate matching results between the center points in the first laser line image and the center points in the second laser line image based on the positional relationship between the center points in the same row of the laser line image and the positional relationship between the light planes. A reconstruction unit is used to perform three-dimensional reconstruction based on the final matching result; Wherein, for any set of laser line images, the first determining unit determines, based on the intrinsic parameters of the binocular camera, a candidate matching result between the center point in the first laser line image and the center point in the second laser line image within that set of laser line images, including: For any set of laser line images, based on the coordinates of the center point in the first laser line image of the target in the set of laser line images, and the pre-calibrated intrinsic parameters of the first camera and the light plane equation, the candidate matching center point of the center point in the first laser line image of the target in the second laser line image of the target in the set of laser line images is determined; wherein, the light plane equation is the equation of the light plane in the coordinate system of the first camera; The second determining unit determines the final matching result from the candidate matching results between the center points of the same row in the laser line image and the positional relationship between the light planes, based on the positional relationship between the center points in the first laser line image and the center points in the second laser line image, including: Based on the positional relationship between the center points of the same row in the first laser line image and the positional relationship between the light planes, the final matching center point is determined from the candidate matching center points in the second laser image from the center point of the row, and the final matching result is obtained.

8. The apparatus according to claim 7, characterized in that, During the scanning process of the object under test using a multi-line laser, the acquisition unit uses a binocular camera to acquire images of the laser lines projected onto the object under test, including: Control the galvanometer to rotate to the set position; For any set position of the galvanometer, use a binocular camera to capture an image of the laser line projected onto the object under test when the galvanometer is in that position; The first determining unit determines, based on the coordinates of the center point in the first laser line image of the target in the set of laser line images, and the pre-calibrated intrinsic parameters and light plane equation of the first camera, the candidate matching center point of the center point in the first laser line image in the set of laser line images, including: For any center point in any row of the first laser line image of the target, the three-dimensional coordinates corresponding to the center point are determined based on the coordinates of the center point, as well as the pre-calibrated intrinsic parameters of the first camera and the light plane equation. Based on the three-dimensional coordinates, and the pre-calibrated extrinsic parameters between the first and second cameras and the intrinsic parameters of the second camera, the projection coordinates of the three-dimensional coordinates in the second laser line image of the target are determined; The center point in the second laser line image of the target, whose distance from the projected coordinates is within a preset distance range, is determined as the candidate matching center point; The first determining unit determines the three-dimensional coordinates corresponding to the center point based on the coordinates of the center point, as well as the pre-calibrated intrinsic parameters and light plane equation of the first camera, including: Based on the scanning position corresponding to the first laser line image of the target, determine the target light plane equation corresponding to each laser line in the first laser line image of the target at that scanning position; For any target light plane equation, the three-dimensional coordinates corresponding to the center point are determined based on the coordinates of the center point, the pre-calibrated intrinsic parameters of the first camera, and the target light plane equation. The first determining unit determines the center point in the target second laser line image whose distance from the projected coordinates is within a preset distance range as the candidate matching center point, including: The center point of the target region in the second laser line image of the target, whose distance from the projected coordinates is within a preset distance range, is determined as the candidate matching center point; wherein, the target region is determined based on the projection error; The second determining unit determines the final matching center point from candidate matching center points in the second laser image based on the positional relationship between the center points of the same row in the first laser line image and the positional relationship between the light planes, including: Based on the coordinates of the center point of the row, sort the center points of the row according to the target order; wherein, the target order includes the order of coordinates from smallest to largest, or the order of coordinates from largest to smallest; Based on the candidate matching center points in the second laser image after the sorting of the center points in the row, and the positional relationship between the light planes corresponding to each candidate matching center point, the candidate matching center point combination with the largest number of center points is determined as the final matching center point combination, and the center point in the final matching center point combination is determined as the final matching center point; wherein, for any candidate matching center point combination, the positional relationship between the light planes corresponding to each center point in the candidate matching center point combination is consistent with the target order.

9. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing machine-executable instructions that can be executed by the processor, the processor executing the machine-executable instructions to implement the method as described in any one of claims 1-6.

10. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores machine-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1-6.

Images