Three-dimensional inspection method for augmented reality auxiliary assembly

Abstract

The embodiment of the invention provides a three-dimensional inspection method for augmented reality auxiliary assembly. The technical problem that existing augmented reality auxiliary assembly is poor in automation can be solved. The method comprises the following steps: acquiring a real camera image and a three-dimensional digital model of a to-be-inspected assembly in the same state; rendering through a virtual camera according to the three-dimensional digital model to obtain a virtual rendered image; inputting the real camera image and the virtual rendering image into a cross-domain image registration network model, and training the cross-domain image registration network model; inputting a to-be-inspected image of a to-be-inspected assembly into the cross-domain image registration network model, and matching a first virtual image; obtaining a texture registration result according to the to-be-checked image and the first virtual image; obtaining a texture similarity score; according to a texture registration result, performing spatial pose registration on the to-be-inspected assembly and the three-dimensional digital model to obtain a spatial pose similarity score; and calculating an assembly score according to the texture similarity score and the spatial pose similarity score.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a three-dimensional inspection method for augmented reality-assisted assembly. Background Art

[0002] Augmented reality assisted assembly uses augmented reality technology to place virtual assembly instructions in the real assembly environment to assist operators in completing assembly tasks step by step.

[0003] At present, the augmented reality assisted assembly system controls the guidance program through the operator's voice or gesture interaction.

[0004] However, this will distract the operator's attention and reduce the efficiency of assembly. Therefore, a checking method is needed to determine the completion status of the assembly, that is, to determine whether the assembly is assembled correctly after each step of the assembly operation. Summary of the invention

[0005] The embodiment of the present application provides a three-dimensional inspection method for augmented reality-assisted assembly, which can solve the technical problem of poor automation of existing augmented reality-assisted assembly.

[0006] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0007] In a first aspect, an embodiment of the present application provides a three-dimensional inspection method for augmented reality-assisted assembly, the three-dimensional inspection method for augmented reality-assisted assembly comprising: obtaining a real camera image and a three-dimensional digital model of an assembly to be inspected in the same state; obtaining a virtual rendered image through virtual camera rendering according to the three-dimensional digital model; inputting the real camera image and the virtual rendered image into a cross-domain image registration network model, and training the cross-domain image registration network model; wherein the cross-domain image registration network model comprises a first matching layer, a second matching layer, a third matching layer and a fourth matching layer; the first matching layer is used to input a first size feature map of the real camera image and a first size feature map of the virtual rendering image to obtain a first deconvolution result; the second matching layer is used to input a second size feature map of the real camera image, a second size feature map of the virtual rendering image and a first deconvolution result to obtain a second deconvolution result; the third matching layer is used to input a third size feature map of the real camera image, a third size feature map of the virtual rendering image and a second deconvolution result As a result, a third deconvolution result is obtained; the fourth matching layer is used to input the fourth size feature map of the real camera image, the fourth size feature map of the virtual rendering image and the third deconvolution result to obtain a fourth deconvolution result; the fourth deconvolution result is used to obtain an estimated optical flow result; the estimated optical flow result is used to perform spatial pose registration; the size of the first size feature map is smaller than the size of the second size feature map; the size of the second size feature map is smaller than the size of the third size feature map; the size of the third size feature map is smaller than the size of the fourth size feature map; the image to be inspected of the assembly to be inspected is input into the cross-domain image registration network model to match the first virtual image; a texture registration result is obtained based on the image to be inspected and the first virtual image; and, based on the texture registration result, a texture similarity score is obtained; the texture registration result includes an estimated optical flow result; based on the texture registration result, the assembly to be inspected and the three-dimensional digital model are spatially aligned to obtain a spatial pose similarity score; the assembly score is calculated based on the texture similarity score and the spatial pose similarity score.

[0008] Based on the above description of the three-dimensional inspection method for augmented reality-assisted assembly provided in the embodiment of the present application, it can be known that the three-dimensional inspection method for augmented reality-assisted assembly includes training a cross-domain image registration network model, inputting the image to be inspected into the cross-domain image registration network model to obtain a texture registration result, and obtaining a spatial pose registration result of the image to be inspected. Furthermore, according to the texture similarity score and the spatial pose similarity score, the assembly score is obtained by calculation. The assembly score can be used to obtain the result of whether the operator has assembled correctly, so that the augmented reality-assisted assembly system can automatically jump to the next step of the boot program when the operator has assembled correctly, without any instructions from the operator, thereby improving the automation of augmented reality-assisted assembly.

[0009] In addition, the assembly status is checked by combining the texture registration results with the spatial pose registration results. The two are not simply spliced ​​together, and the spatial pose registration is performed on the basis of the texture registration results. In this way, the accuracy of the inspection results is improved.

[0010] Furthermore, both two-dimensional and three-dimensional features are taken into consideration to improve the convenience of actual operations.

[0011] In addition, it not only has high accuracy and does not rely on 3D technology, but also has small pose estimation error and short time consumption, making it suitable for various industrial assembly scenarios.

[0012] In a feasible implementation of the first aspect, the first matching layer includes a coarse correlation layer and a coarse decoder; the coarse correlation layer is built based on the matching layer; the coarse decoder is built based on the mapping decoder; the three-dimensional inspection method for augmented reality assisted assembly also includes: inputting the first size feature map of the real camera image and the first size feature map of the virtual rendering image into the coarse correlation layer and the coarse decoder in sequence to obtain a first optical flow result; deconvolving the first optical flow result to obtain a first deconvolution result.

[0013] In a feasible implementation of the first aspect, the second matching layer includes a deformation layer, a fine correlation layer and a fine decoder; the deformation layer is used to deform the feature map through a bilinear interpolation method; the fine correlation layer is built according to the cost volume layer; the fine decoder is built according to the optical flow estimator; the three-dimensional inspection method for augmented reality assisted assembly also includes: inputting the second size feature map and the first deconvolution result of the virtual rendering image into the deformation layer to obtain a first deformation result; inputting the second size feature map and the first deformation result of the real camera image into the fine correlation layer to obtain a first cost volume estimation result; inputting the first cost volume estimation result and the first deconvolution result into the fine decoder to obtain a first fine decoding result; obtaining a second optical flow result according to the first fine decoding result and the first deconvolution result; deconvolving the second optical flow result to obtain a second deconvolution result.

[0014] In a feasible implementation of the first aspect, the assembly to be inspected includes a first component and a second component; the three-dimensional inspection method for augmented reality-assisted assembly also includes: performing image segmentation on the real camera image and the virtual rendering image to obtain an image segmentation mask of the first component and an image segmentation mask of the second component; inputting the image segmentation mask of the first component and the image segmentation mask of the second component into a cross-domain image registration network model to obtain an estimated optical flow result.

[0015] In a feasible implementation of the first aspect, the three-dimensional inspection method for augmented reality-assisted assembly also includes: obtaining a first dense registration point pair based on the estimated optical flow result; the first dense registration point pair is a two-dimensional dense registration point pair corresponding to the image to be inspected and the first virtual image; obtaining a second dense registration point pair based on preset rendering parameters and the first dense registration point pair; the second dense registration point pair is a two-dimensional dense registration point pair corresponding to the image to be inspected and the three-dimensional digital model; the second dense registration point pair includes multiple pairs; randomly selecting some of the second dense registration point pairs, and iteratively calculating the six-dimensional pose; based on the six-dimensional pose, obtaining displacement similarity and rotation similarity to obtain a spatial pose similarity score.

[0016] In a feasible implementation of the first aspect, the calculation formula of displacement similarity includes:

[0017]

[0018] Where N represents the number of components; represents the estimated translation value of component i on the τ(X / Y / Z) axis; Indicates the relative translation value of components i and j on the τ(X / Y / Z) axis in the 3D digital model; ΔT i It represents the average displacement error of each component i, i.e., the displacement similarity, which is determined by the average value of its displacement error in the X, Y, and Z axes.

[0019] In a feasible implementation of the first aspect, the calculation formula of the rotation similarity includes:

[0020]

[0021] Where N represents the number of components, represents the estimated rotation value of the component on the axis, represents the estimated rotation value of component i on the τ(X / Y / Z) axis, Represents the relative rotation value of components i and j on the τ(X / Y / Z) axis in the 3D digital model, ΔR i The rotational mean error, i.e., the rotational similarity, of part i is determined by the average of its rotational errors in the X, Y, and Z axes.

[0022] In a feasible implementation of the first aspect, the formula for calculating texture similarity includes:

[0023]

[0024] For each component i, I matching point pairs are randomly selected and their similarities within L neighborhoods are calculated, N and They respectively represent the pixel points in the neighborhood of the matching point pairs in the original real camera image and the deformed real camera image.

[0025] In a second aspect, an embodiment of the present application provides a three-dimensional inspection system for augmented reality-assisted assembly, the three-dimensional inspection system for augmented reality-assisted assembly comprising: at least one processor; a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor so that the at least one processor can execute the method provided in the first aspect.

[0026] The three-dimensional inspection system for augmented reality assisted assembly trains a cross-domain image registration network model by executing the method provided in the first aspect, inputs the image to be inspected into the cross-domain image registration network model to obtain a texture registration result, and obtains a spatial pose registration result of the image to be inspected. Then, according to the texture similarity score and the spatial pose similarity score, an assembly score is obtained by calculation. The assembly score can be used to obtain the result of whether the operator has assembled correctly, so that the augmented reality assisted assembly system can automatically jump to the next step of the guidance program when the operator has assembled correctly, without any instructions from the operator, thereby improving the automation of augmented reality assisted assembly.

[0027] In a third aspect, an embodiment of the present application provides a computer-readable medium having computer program instructions stored thereon, and the computer program instructions can be executed by a processor to implement the method provided in the first aspect.

[0028] The computer program instructions in the computer-readable medium implement the method provided in the first aspect, train the cross-domain image registration network model, input the image to be checked into the cross-domain image registration network model to obtain the texture registration result, and obtain the spatial pose registration result of the image to be checked. Then, according to the texture similarity score and the spatial pose similarity score, the assembly score is obtained by calculation. The assembly score can be used to obtain the result of whether the operator has assembled correctly, so that the augmented reality assisted assembly system can automatically jump to the next step of the boot program when the operator has assembled correctly, without any instructions from the operator, thereby improving the automation of augmented reality assisted assembly. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic diagram of the structure of a three-dimensional inspection system for augmented reality-assisted assembly provided in an embodiment of the present application;

[0030] Figure 2 A schematic flow chart of a three-dimensional inspection method for augmented reality-assisted assembly provided in an embodiment of the present application;

[0031] Figure 3 A schematic diagram of the structure of a cross-domain image registration network in a three-dimensional inspection method for augmented reality-assisted assembly provided in an embodiment of the present application;

[0032] Figure 4 A schematic diagram of a cross-domain registration data set of a cross-domain image registration network in a three-dimensional inspection method for augmented reality-assisted assembly provided in an embodiment of the present application;

[0033] Figure 5 A schematic diagram of an implementation method of texture registration results in a three-dimensional inspection method for augmented reality-assisted assembly provided in an embodiment of the present application;

[0034] Figure 6 A schematic diagram of an implementation method of spatial posture registration in a three-dimensional inspection method for augmented reality-assisted assembly provided in an embodiment of the present application;

[0035] Figure 7 A schematic diagram of an implementation method of a six-dimensional pose estimation result for spatial pose registration in a three-dimensional inspection method for augmented reality-assisted assembly provided in an embodiment of the present application;

[0036] Figure 8 A schematic diagram of a visualization interface for implementing an augmented reality inspection of components of a three-dimensional inspection method for augmented reality-assisted assembly provided in an embodiment of the present application;

[0037] Fig. 9 A schematic diagram of a visualization interface for implementing augmented reality inspection of components in each step of an assembly process of a three-dimensional inspection method for augmented reality-assisted assembly provided in an embodiment of the present application;

[0038] Fig.10 Three views of the incorrect assembly of the components;

[0039] Fig.11 Three views of the template for the correct assembly of the component;

[0040] Fig.12 A schematic diagram of an implementation method of an incorrect assembly inspection result of a three-dimensional inspection method for augmented reality-assisted assembly provided in an embodiment of the present application;

[0041] Fig.13 A schematic diagram of a visualization interface for an implementation method of an error reminder for a three-dimensional inspection method for augmented reality-assisted assembly provided in an embodiment of the present application;

[0042] Fig.14 A step-by-step schematic diagram of a three-dimensional inspection method for augmented reality-assisted assembly for assisting in assembling a building block fox provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention. In the description of the embodiments of the present invention, unless otherwise specified, "multiple" refers to two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0044] In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, the words "first", "second" and the like are used to distinguish the same items or similar items with substantially the same functions and effects. Those skilled in the art will understand that the words "first", "second" and the like do not limit the quantity and execution order, and the words "first", "second" and the like do not necessarily limit the difference. At the same time, in the embodiments of the present invention, the words "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or design. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way for easy understanding.

[0045] The principles and features of the present application are described below. The examples given are only used to explain the present application and are not used to limit the scope of the present application.

[0046] Before describing the embodiments of the present application, the concepts involved are described.

[0047] Cost Volume is a concept used in computer vision to measure the similarity between left and right views, especially in optical flow estimation. It is a four-dimensional (4D) tensor, usually represented as [B, C, D, H]. Among them, B is the batch dimension, C is the channel dimension, D is the disparity dimension, and H is the height of the feature map. In optical flow estimation, Cost Volume is used to represent the data matching cost of a pixel and its corresponding pixel in the next frame.

[0048] Optical flow, in image registration, is represented as the displacement of corresponding pixels between the images to be registered.

[0049] The embodiment of the present application provides a three-dimensional inspection method for augmented reality-assisted assembly, which is applicable to the field of industrial assembly. The method is used for a three-dimensional inspection method for augmented reality-assisted assembly, and the assembly condition is inspected by combining the texture registration results and the spatial posture registration results. The two are not a simple splicing combination, and the spatial posture registration is performed on the basis of the texture registration results. In this way, the accuracy of the inspection results is improved.

[0050] The embodiment of the present application provides a three-dimensional inspection system for augmented reality-assisted assembly, which can execute the three-dimensional inspection method for augmented reality-assisted assembly provided by the embodiment of the present application. Figure 1 A schematic structural diagram of a three-dimensional inspection system for augmented reality-assisted assembly provided in an embodiment of the present application.

[0051] like Figure 1 As shown, the three-dimensional inspection system 001 for augmented reality-assisted assembly includes at least one processor 011 and a memory 012 communicatively connected to the at least one processor; wherein the memory 012 stores instructions executable by the at least one processor 011, and the instructions are executed by the at least one processor 011 so that the at least one processor 011 can execute the three-dimensional inspection method for augmented reality-assisted assembly provided in an embodiment of the present application.

[0052] Figure 2 A flowchart of a three-dimensional inspection method for augmented reality-assisted assembly provided in an embodiment of the present application. Figure 2 As shown, in some embodiments, the three-dimensional inspection method for augmented reality assisted assembly includes the following steps:

[0053] S1, obtain the real camera image and 3D digital model of the assembly to be inspected in the same state.

[0054] Use a camera to capture the entire assembly process of the assembly to be inspected, save the real camera image, and strictly calibrate the camera pose.

[0055] Collect and save real camera images and three-dimensional digital models in the same state. When the assembly to be inspected has multiple states, record the real camera images and three-dimensional digital models in each state respectively. In some embodiments, the assembly to be inspected has multiple components, and the assembly steps of the assembly to be inspected can be multiple process assembly steps. Each process assembly step is completed, which corresponds to a state. Multiple process assembly steps correspond to multiple states. In one implementation, one process assembly step can assemble multiple components, and the number of states of the assembly to be inspected is less than the number of components. In another implementation, one process assembly step can only assemble one component, and each component is successfully assembled, which corresponds to a state, and the number of states of the assembly to be inspected is equal to the number of components.

[0056] In one implementation, taking the assembly of building blocks as an example, which includes seven components, the whole process of assembling building blocks is filmed with a camera, the real camera image is saved, and the camera position and posture are strictly calibrated.

[0057] In order to enrich the training data set of the cross-domain image registration network model, in some embodiments, a YCB-Video data set is obtained and used as a training data set together with the assembly to be inspected. The YCB-Video data set includes six-dimensional (6D) pose parameters and camera parameters of a real camera.

[0058] S2, according to the three-dimensional digital model, rendering is performed through a virtual camera to obtain a virtual rendering image.

[0059] According to the calibrated camera pose, a virtual camera is set up to render the building block CAD model onto a two-dimensional image. During the building block assembly process, a virtual rendering image aligned with the real camera image is obtained, and the optical flow between them is recorded.

[0060] It is understandable that the real camera image and the virtual rendering image in each state correspond to each other. For example, the second step in the process assembly steps corresponds to a virtual rendering image. The third step in the process assembly steps corresponds to a virtual rendering image.

[0061] In order to enrich the training data set of the cross-domain image registration network model, in some embodiments, according to the YCB-Video data set, a three-dimensional digital model (such as a computer-aided design (CAD) model) is rendered onto a two-dimensional image to obtain a virtual rendered image aligned with the real camera image, and the optical flow between them is recorded.

[0062] In order to enrich the training data set of the cross-domain image registration network model, in some embodiments, the aligned YCB data set and the building block assembly data set are deformed using affine transformation, homogeneous transformation, and thin plate spline methods, and the optical flow between the real camera image and the virtual rendering image after deformation is recorded. In one implementation, the following is obtained: Figure 4 The enhanced dataset shown in FIG. 1 includes the YCB dataset, the assembly dataset to be inspected (real camera images, 3D digital models and virtual rendering images of the assembly to be inspected), the deformed real camera images and the deformed virtual rendering images.

[0063] By executing step S1 and step S2, a training data set of the cross-domain image registration network model is obtained.

[0064] S3, input the real camera image and the virtual rendering image into the cross-domain image registration network model, and train the cross-domain image registration network model.

[0065] In some embodiments, the real camera image and the virtual rendered image are preprocessed to obtain feature maps of different sizes. Preprocessing includes image segmentation and resizing. In some embodiments, four feature maps of different sizes are obtained, namely, a first size feature map, a second size feature map, a third size feature map, and a fourth size feature map. The size of the first size feature map is smaller than the size of the second size feature map. The size of the second size feature map is smaller than the size of the third size feature map. The size of the third size feature map is smaller than the size of the fourth size feature map. In one implementation, a real camera image of size "256×256" is input into a VGG-16 network to obtain a first size feature map of size "16×16", a second size feature map of size "32×32", a third size feature map of size "64×64", and a fourth size feature map of size "128×128". The virtual rendered image of size "256×256" is input into the VGG-16 network to obtain a first size feature map of size "16×16", a second size feature map of size "32×32", a third size feature map of size "64×64", and a fourth size feature map of size "128×128".

[0066] like Figure 3 As shown, in some embodiments, the cross-domain image registration network model includes a first matching layer, a second matching layer, a third matching layer, and a fourth matching layer. The first matching layer, the second matching layer, the third matching layer, and the fourth matching layer are pyramid structures. The first matching layer, the second matching layer, the third matching layer, and the fourth matching layer can also be called pyramid layers. The first matching layer is the top layer of the pyramid and can also be called the top pyramid.

[0067] It can be understood that the number of size types of feature maps is the same as the number of matching layers.

[0068] The first matching layer is used to input the first size feature map of the real camera image and the first size feature map of the virtual rendering image to obtain a first deconvolution result. In this way, the feature maps are roughly aligned.

[0069] In some embodiments, the first matching layer includes a coarse correlation layer and a coarse decoder. The coarse correlation layer is constructed according to the matching layer proposed by Rocco, and the coarse decoder is constructed according to a correspondence mapping decoder.

[0070] In some embodiments, the three-dimensional inspection method for augmented reality assisted assembly further includes:

[0071] S311, inputting the first size feature map of the real camera image and the first size feature map of the virtual rendering image into the coarse correlation layer and the coarse decoder in sequence to obtain a first optical flow result.

[0072] In some embodiments, when executing step S311, the three-dimensional inspection method for augmented reality assisted assembly further includes:

[0073] S3111, in the coarse correlation layer, by means of a global search, a first correlation result of each pixel point in the first size feature map of the real camera image and all the pixel points in the first size feature map of the virtual rendering image is calculated.

[0074] S3112, input the first correlation result into a coarse decoder, and output a coarse registration optical flow.

[0075] Roughly registered optical flow, that is, the first optical flow result.

[0076] S312: Deconvolute the first optical flow result to obtain a first deconvolution result.

[0077] The second matching layer is used to input the second size feature map of the real camera image, the second size feature map of the virtual rendering image and the first deconvolution result to obtain a second deconvolution result.

[0078] In some embodiments, the second matching layer includes a warp layer, a fine correlation layer, and a fine decoder. The warp layer is used to deform the feature map by a bilinear interpolation method. The fine correlation layer is built according to the cost volume layer of FlowNet2.0. The fine decoder is built according to the optical flow estimator of FlowNet2.0.

[0079] In some embodiments, the three-dimensional inspection method for augmented reality assisted assembly further includes:

[0080] S321, inputting the second size feature map of the virtual rendering image and the first deconvolution result into the deformation layer to obtain a first deformation result.

[0081] S322, input the second size feature map of the real camera image and the first deformation result into the fine correlation layer to obtain a first cost volume estimation result.

[0082] In the fine correlation layer, by means of local search, the correlation between each pixel point in the second size feature map of the real camera image and the neighboring pixel points in the first deformation result is obtained to obtain the first cost volume estimation result.

[0083] S323 , input the first cost volume estimation result and the first deconvolution result into a fine decoder to obtain a first fine decoding result.

[0084] S324: Obtain a second optical flow result according to the first precise decoding result and the first deconvolution result.

[0085] S325, deconvolve the second optical flow result to obtain a second deconvolution result.

[0086] The third matching layer is used to input the third size feature map of the real camera image, the third size feature map of the virtual rendering image, and the second deconvolution result to obtain a third deconvolution result.

[0087] The fourth matching layer is used to input the fourth size feature map of the real camera image, the fourth size feature map of the virtual rendering image, and the third deconvolution result to obtain a fourth deconvolution result.

[0088] The structure and data processing methods in the third and fourth matching layers can refer to those in the second matching layer, and will not be described in detail here. After three layers of fine registration layers (the second matching layer, the third matching layer and the fourth matching layer), the fine registration optical flow results are output.

[0089] The fourth deconvolution result is used to obtain an estimated optical flow result. In some embodiments, the fourth deconvolution result is upsampled to output the estimated optical flow result.

[0090] In one implementation, the batch size is set to 8, the Adam optimizer is used, the initial learning rate is set to 0.0001, and the weight decay is set to 0.0004. The loss weight of the first matching layer is α1=0.32. The loss weight of the second matching layer is α2=0.08. The loss weight of the third matching layer is α3=0.02. The loss weight of the fourth matching layer is α4=0.01. The search value of the fine correlation layer of the second matching layer is set to 81. The search value of the fine correlation layer of the third matching layer is set to 81. The search value of the fine correlation layer of the fourth matching layer is set to 81.

[0091] For example, experiments can be performed using Pytorch on a 3D inspection system for augmented reality-assisted assembly equipped with a central processing unit (Intel Core i9-13900KF CPU) and an image processing unit (NVIDIA GeForce RTX 4080 GPU).

[0092] The estimated optical flow results are used for spatial pose registration.

[0093] In some embodiments, the assembly to be inspected includes a first component and a second component, and the three-dimensional inspection method for augmented reality-assisted assembly further includes:

[0094] S331, performing image segmentation on the real camera image and the virtual rendering image to obtain an image segmentation mask of the first component and an image segmentation mask of the second component.

[0095] In some embodiments, MaskCNN is used to perform image segmentation to obtain an image segmentation mask for each component.

[0096] S332, input the image segmentation mask of the first component and the image segmentation mask of the second component into the cross-domain image registration network model to obtain an estimated optical flow result.

[0097] To better visualize the network results, such as Figure 5 As shown, using the estimated optical flow result, the real camera image (such as Figure 5 Each pixel in the image is remapped to the virtual rendered image (as shown in the first column) Figure 5 The corresponding pixel positions in the second column are shown in the figure, thereby converting the virtual rendered image to the domain where the real camera image is located, and obtaining the deformation result of the real camera image after the estimated optical flow result (as shown in the figure Figure 5 As shown in the fourth column). Among them, the deformation result of the real camera image after the real optical flow result (such as Figure 5 The third column shows the actual (relative to the estimated) deformation result, which is obtained in advance after calibrating the camera pose.

[0098] In order to obtain a better model, in some embodiments, endpoint error (EPE) is used as an evaluation indicator of optical flow, and a multi-scale training loss is used to train the network to obtain a trained model.

[0099] By executing steps S1 to S3, the training of the cross-domain image registration network model is completed.

[0100] S4, inputting the image of the assembly to be inspected into the cross-domain image registration network model to match the first virtual image.

[0101] In one implementation, Fig. 9 As shown, taking assembly blocks as an example, when the image to be inspected of the assembly to be inspected is the assembly image of the second step (step 2) in the process assembly steps, the first virtual image is the virtual rendering image of the second step (step 2) in the process assembly steps obtained according to the aforementioned steps S1 and S2.

[0102] S5, obtaining a texture registration result according to the image to be inspected and the first virtual image; and obtaining a texture similarity score according to the texture registration result.

[0103] The image to be inspected and the first virtual image are input into the cross-domain image registration network model. The processing of the image to be inspected and the first virtual image in the cross-domain image registration network model can be referred to step S3, which is not described here. After the image to be inspected and the first virtual image are calculated by the first matching layer, the second matching layer, the third matching layer and the fourth matching layer, a texture registration result is obtained.

[0104] The texture registration results include the estimated optical flow results.

[0105] In some embodiments, the real camera image is warped based on the estimated optical flow result to align the virtual rendered image.

[0106] In one implementation, the formula for calculating texture similarity includes:

[0107]

[0108] For each component i, I matching point pairs are randomly selected and their similarities within L neighborhoods are calculated, N and They respectively represent the pixel points in the neighborhood of the matching point pairs in the original real camera image and the deformed real camera image.

[0109] S6, based on the texture registration result, the spatial pose registration is performed on the assembly to be inspected and the three-dimensional digital model to obtain a spatial pose similarity score.

[0110] like Figure 6 As shown, in some embodiments, in a three-dimensional inspection system for augmented reality-assisted assembly, an object coordinate system is established and numbered for each component of the three-dimensional digital model. In one implementation, taking the assembly building blocks as an example, there are seven components, and an object coordinate system is established and numbered for each building block component. In order to simplify the calculation, it is exemplary to ensure that the coordinate axis direction of each component is consistent, which means that the relative rotation between each component is "0".

[0111] In order to obtain the spatial pose similarity score, before performing spatial pose registration on the assembly to be inspected and the three-dimensional digital model, the relative displacement template value between each component is first obtained. In some embodiments, the relative displacement template value is obtained based on the object coordinate system of the three-dimensional digital model.

[0112] As shown in Table 1, where |ΔT x |,|ΔT y |, and |ΔT z |Represents the relative translation of the two components on the X, Y and Z axes respectively. Figure 6 and Table 1, relative displacement template values ​​(mm) of each component of the building block car.

[0113] Table 1 Relative displacement template values ​​of each component of the building block car (mm)

[0114]

[0115]

[0116] In some embodiments, the three-dimensional inspection method for augmented reality assisted assembly further includes:

[0117] S611, obtaining a first dense registration point pair according to the estimated optical flow result.

[0118] The first dense registration point pair is a two-dimensional (2D-2D) dense registration point pair corresponding to the image to be inspected and the first virtual image.

[0119] S612: Obtain a second dense registration point pair according to the preset rendering parameters and the first dense registration point pair.

[0120] The second dense registration point pair is a two-dimensional to three-dimensional (2D-3D) dense registration point pair corresponding to the image to be inspected and the three-dimensional digital model. The second dense registration point pair includes multiple pairs.

[0121] S613, randomly select some second dense registration point pairs, and iteratively calculate the six-dimensional pose.

[0122] In some embodiments, a random sampling consensus (RANSAC) framework is used to randomly select some 2D-3D matching point pairs and input them into the Perspective-n-Point (PNP) algorithm to recover the pose of the 3D object from the 2D image and iteratively calculate the 6D pose of each component of the assembly. Figure 7 As shown, in one implementation, taking the assembly of building blocks as an example, the six-dimensional (6D) pose of each component is obtained.

[0123] S614, based on the six-dimensional posture, obtain the displacement similarity and the rotation similarity to obtain a spatial posture similarity score.

[0124] In some embodiments, the relative displacement is obtained based on the difference between the six-dimensional pose and the relative displacement template value. Figure 7 As shown in Table 1, the six-dimensional pose and relative displacement template values ​​are subtracted to obtain the relative displacement shown in Table 2.

[0125] Table 2 Estimated relative displacement of each component of the building block car

[0126] ①② ①③ ①④ ①⑤ ①⑥ ①⑦ ②③ <![CDATA[|ΔT x | est ]]> 59.8 4.11 39.957 40.52 33.249 85.598 55.69 <![CDATA[|ΔT y | est ]]> 4.779 26.833 33.356 44.936 9.994 9.939 22.054 <![CDATA[|ΔT z | est ]]> 0.744 6.767 1.478 6.879 3.537 4.068 6.023 ②④ ②⑤ ②⑥ ②⑦ ③④ ③⑤ ③⑥ <![CDATA[|ΔT x | est ]]> 19.843 19.28 26.551 25.798 35.847 36.41 29.139 <![CDATA[|ΔT y | est ]]> 28.577 40.157 5.215 5.16 6.523 18.103 16.839 <![CDATA[|ΔT z | est ]]> 0.734 6.135 2.793 3.324 5.289 0.112 3.23 ③⑦ ④⑤ ④⑥ ④⑦ ⑤⑥ ⑤⑦ ⑥⑦ <![CDATA[|ΔT x | est ]]> 81.488 0.563 6.708 45.641 7.271 45.078 52.349 <![CDATA[|ΔT y | est ]]> 16.894 11.58 23.362 23.417 34.942 34.997 0.055 <![CDATA[|ΔT z | est ]]> 2.699 5.401 2.059 2.59 3.342 2.811 0.531

[0127] By comparing the estimated relative displacement values ​​between components with the relative displacement template values ​​between components in the CAD model, the average displacement error of each component during the assembly process can be obtained, as shown in Table 3.

[0128] Table 3 Estimated average displacement errors of each component of the building block car

[0129] ① ② ③ ④ ⑤ ⑥ ⑦ <![CDATA[ΔT est ]]> 3.287 2.675 4.648 2.504 3.060 2.936 3.662

[0130] In one implementation, the calculation formula for displacement similarity includes:

[0131]

[0132] Where N represents the number of components; represents the estimated translation value of component i on the τ(X / Y / Z) axis; Indicates the relative translation value of components i and j on the τ(X / Y / Z) axis in the 3D digital model; ΔT i It represents the average displacement error of each component i, i.e., the displacement similarity, which is determined by the average value of its displacement error in the X, Y, and Z axes.

[0133] In one implementation, the calculation formula for rotation similarity includes:

[0134]

[0135] Where N represents the number of components, represents the estimated rotation value of the component on the axis, represents the estimated rotation value of component i on the τ(X / Y / Z) axis, Represents the relative rotation value of components i and j on the τ(X / Y / Z) axis in the 3D digital model, ΔR i The rotational mean error, i.e., the rotational similarity, of part i is determined by the average of its rotational errors in the X, Y, and Z axes.

[0136] In this way, texture registration and spatial pose registration are combined, and the "image texture registration + PnP 6D pose estimation" architecture is adopted to integrate the texture registration results with the spatial pose registration results. The two are not a simple splicing combination, and the spatial pose registration is performed on the basis of the texture registration results.

[0137] S7, calculating the assembly score according to the texture similarity score and the spatial pose similarity score.

[0138] In one implementation, the calculation formulas for the texture similarity score and the spatial pose similarity score include:

[0139] TS i , R.S. i , AMS i =Z_Score(ΔT i , ΔR i , A.M. i );

[0140] The texture similarity and spatial pose similarity (including displacement similarity and rotation similarity) results of each component i are combined, and the texture similarity score and spatial pose similarity score AMS are obtained using the Z-score normalization method. i TS i , RS i .

[0141] In one implementation, the calculation formula of the assembly inspection score TAI includes:

[0142] TAI i =αTS i +βRS i +(1-α-β)AMS i ;

[0143] The weighted average texture similarity score and the spatial pose similarity score are used to obtain the assembly inspection score TAI. When TAI is less than the threshold, the assembly is judged to be successful, otherwise, the assembly is judged to be failed.

[0144] There are many ways to implement the visual display of the assembly score.

[0145] for example,

[0146] In some embodiments, all components of the building block car are checked at the same time after assembly is completed. Figure 8 As shown in the figure, it shows the augmented reality inspection visualization interface, and the numbers in the figure represent the TAI score of each component. Figure 6 and Figure 8 , the TAI score of component ① is 2.928. The TAI score of component ② is 2.762. The TAI score of component ③ is 4.335. The TAI score of component ④ is 2.774. The TAI score of component ⑤ is 2.977. The TAI score of component ⑥ is 3.002. The TAI score of component ⑦ is 3.445.

[0147] For example,

[0148] In other embodiments, or, the method provided by the embodiment of the present application can check the components separately for each step of the assembly, and the effect is as follows: Fig. 9 As shown, it shows the augmented reality inspection visualization interface of the components at each assembly stage.

[0149] In addition, according to the TAI score, the method provided in the embodiment of the present application can find the incorrect assembly situation and display the error reminder in the visual interface. Fig.10 As shown in the figure, the building block component ② is incorrectly offset by 16mm along the z-axis, while the correct CAD assembly template is as follows Fig.11 The method provided in the embodiment of the present application can obtain an error detection result by combining texture registration and spatial posture registration, such as Fig.12 As shown, the visual inspection interface is as follows Fig.13 shown.

[0150] In addition to assembling the building block car, the method provided in the embodiment of the present application assembles the building block fox according to the virtual guidance provided by the augmented reality assisted assembly system, such as Fig.14As shown. (a)-(b) Augmented reality virtual guidance, in which augmented reality technology superimposes the virtual CAD model of the components required for each assembly step onto the real model. (c) Real model assembled according to augmented reality virtual guidance. (d) CAD assembly template. In this process, 100 images of correct assembly and 100 images of incorrect assembly are randomly taken from any angle to test the inspection accuracy. After testing, the method provided in the embodiment of the present application can add an inspection function to the augmented reality assisted assembly system to determine whether the assembly is correctly installed, and the inspection accuracy is 92%.

[0151] Compared with the related art, by extracting the speed-up robust (SURF) features from the video stream, matching the SURF features with the local features extracted from the reference image, and judging whether the assembly meets the standards based on the matching results, only two-dimensional features are considered, and the three-dimensional feature information of the assembly is lacking, resulting in low accuracy of the inspection results. The three-dimensional inspection method for augmented reality-assisted assembly provided in the embodiment of the present application comprehensively checks the assembly status with the results of texture registration and spatial posture registration. The two are not a simple splicing combination, and the spatial posture registration is performed on the basis of the texture registration results. In this way, the accuracy of the inspection results is improved.

[0152] Compared with the related art of joint inspection through two-dimensional features and multiple perspectives, which is more complex and time-consuming, the three-dimensional inspection method for augmented reality-assisted assembly provided in the embodiment of the present application takes into account both two-dimensional features and three-dimensional features, thereby improving the convenience of actual operation.

[0153] Compared with the related art, the three-dimensional digital model is aligned with the actual assembly component by using 3D point cloud. By aligning the assembly base, the point cloud of the remaining components can be spatially registered for inspection. The use of 3D point cloud relies on 3D vision technology, the algorithm complexity is high and time-consuming, and it is difficult to apply to actual industrial assembly scenarios. The three-dimensional inspection method for augmented reality-assisted assembly provided in the embodiment of the present application is not only highly accurate, independent of 3D technology, and time-saving, but also applicable to various industrial assembly scenarios.

[0154] Compared with the related art, the position and posture estimation error is at the level of 5 cm and 5 metric, which is difficult to be directly applied to the field of industrial assembly. The three-dimensional inspection method for augmented reality-assisted assembly provided in the embodiment of the present application has a small position and posture estimation error and can be applied to the field of industrial assembly.

[0155] Based on the same application concept, a three-dimensional inspection system for augmented reality-assisted assembly is also provided in an embodiment of the present application. The method corresponding to the three-dimensional inspection system for augmented reality-assisted assembly may be the three-dimensional inspection method for augmented reality-assisted assembly in the aforementioned embodiment, and its principle of solving the problem is similar to that of the method. The three-dimensional inspection system for augmented reality-assisted assembly provided in an embodiment of the present application includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the methods and / or technical solutions of the aforementioned multiple embodiments of the present application.

[0156] Another embodiment of the present application further provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of the present application described above.

[0157] Specifically, the present embodiment may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device, or device.

[0158] Computer readable signal media may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0159] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0160] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0161] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0162] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0163] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or page components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0164] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0165] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0166] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0167] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0168] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in a device claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

Claims

1. A three-dimensional inspection method for augmented reality assisted assembly, characterized in that: include: Obtain the real camera image and 3D digital model of the assembly to be inspected in the same state; According to the three-dimensional digital model, a virtual rendering image is obtained by rendering with a virtual camera; The real camera image and the virtual rendered image are input into a cross-domain image registration network model to train the cross-domain image registration network model; wherein the cross-domain image registration network model includes a first matching layer, a second matching layer, a third matching layer and a fourth matching layer; the first matching layer is used to input a first size feature map of the real camera image and a first size feature map of the virtual rendered image to obtain a first deconvolution result; the second matching layer is used to input a second size feature map of the real camera image, a second size feature map of the virtual rendered image and the first deconvolution result to obtain a second deconvolution result; the third matching layer is used to input a second size feature map of the real camera image, a second size feature map of the virtual rendered image and the first deconvolution result to obtain a second deconvolution result. The third deconvolution result is obtained by inputting the third size feature map of the virtual rendered image, the third size feature map of the virtual rendered image and the second deconvolution result; the fourth matching layer is used to input the fourth size feature map of the real camera image, the fourth size feature map of the virtual rendered image and the third deconvolution result to obtain a fourth deconvolution result; the fourth deconvolution result is used to obtain an estimated optical flow result; the estimated optical flow result is used to perform spatial pose registration; the size of the first size feature map is smaller than the size of the second size feature map; the size of the second size feature map is smaller than the size of the third size feature map; the size of the third size feature map is smaller than the size of the fourth size feature map; Inputting the image to be inspected of the assembly to be inspected into the cross-domain image registration network model to match the first virtual image; Obtaining a texture registration result according to the image to be inspected and the first virtual image; and obtaining a texture similarity score according to the texture registration result; the texture registration result includes the estimated optical flow result; According to the texture registration result, the assembly to be inspected and the three-dimensional digital model are spatially registered to obtain a spatial posture similarity score; An assembly score is calculated according to the texture similarity score and the spatial pose similarity score.

2. The three-dimensional inspection method for augmented reality assisted assembly according to claim 1, characterized in that: The first matching layer includes a coarse correlation layer and a coarse decoder; the coarse correlation layer is constructed according to the matching layer; The coarse decoder is constructed according to the mapping decoder; The three-dimensional inspection method for augmented reality assisted assembly also includes: Inputting the first size feature map of the real camera image and the first size feature map of the virtual rendering image into the coarse correlation layer and the coarse decoder in sequence to obtain a first optical flow result; Deconvolute the first optical flow result to obtain the first deconvolution result.

3. The three-dimensional inspection method for augmented reality assisted assembly according to claim 1 or 2, characterized in that: The second matching layer includes a deformation layer, a fine correlation layer and a fine decoder; the deformation layer is used to deform the feature map by a bilinear interpolation method; the fine correlation layer is constructed according to the cost volume layer; The fine decoder is built based on the optical flow estimator; The three-dimensional inspection method for augmented reality assisted assembly also includes: Inputting the second size feature map of the virtual rendering image and the first deconvolution result into the deformation layer to obtain a first deformation result; Inputting the second size feature map of the real camera image and the first deformation result into the fine correlation layer to obtain a first cost volume estimation result; Inputting the first cost volume estimation result and the first deconvolution result into the fine decoder to obtain a first fine decoding result; Obtaining a second optical flow result according to the first precise decoding result and the first deconvolution result; Deconvolute the second optical flow result to obtain the second deconvolution result.

4. The three-dimensional inspection method for augmented reality assisted assembly according to claim 1 or 2, characterized in that: The assembly to be inspected includes a first component and a second component; The three-dimensional inspection method for augmented reality assisted assembly also includes: Performing image segmentation on the real camera image and the virtual rendering image to obtain an image segmentation mask of the first component and an image segmentation mask of the second component; The image segmentation mask of the first component and the image segmentation mask of the second component are input into a cross-domain image registration network model to obtain the estimated optical flow result.

5. The three-dimensional inspection method for augmented reality assisted assembly according to claim 1 or 2, characterized in that: The three-dimensional inspection method for augmented reality assisted assembly also includes: According to the estimated optical flow result, a first dense registration point pair is obtained; the first dense registration point pair is a two-dimensional pair of two-dimensional dense registration point pairs corresponding to the image to be inspected and the first virtual image; According to the preset rendering parameters and the first dense registration point pair, a second dense registration point pair is obtained; the second dense registration point pair is a two-dimensional pair of three-dimensional dense registration point pairs corresponding to the image to be inspected and the three-dimensional digital model; the second dense registration point pair includes a plurality of pairs; Randomly selecting some of the second dense registration point pairs, and iteratively calculating the six-dimensional pose; Based on the six-dimensional posture, displacement similarity and rotation similarity are obtained to obtain the spatial posture similarity score.

6. The three-dimensional inspection method for augmented reality assisted assembly according to claim 5, characterized in that: The calculation formula of the displacement similarity includes: in, N Indicates the number of components; Representation Components i In τ (X / Y / Z) The estimated translation value on the axis; Represents components in a 3D digital model i and j In τ (X / Y / Z) Relative translation value on the axis; ΔT i Represents each component i The average displacement error of , i.e., displacement similarity, is determined by the average of its displacement errors in the X, Y, and Z axes.

7. The three-dimensional inspection method for augmented reality assisted assembly according to claim 5, characterized in that: The calculation formula of the rotation similarity includes: in, N Indicates the number of components, Represents the estimated rotation value of the component on the axis, indicating the component i The estimated rotation value about the τ(X / Y / Z) axis, Represents components in a 3D digital model i and j Relative rotation value on the τ(X / Y / Z) axis, ΔR i Representation Components i The rotational mean error, i.e., the rotational similarity, is determined by the average of its rotation errors on the X, Y, and Z axes.

8. The three-dimensional inspection method for augmented reality assisted assembly according to claim 1 or 2, characterized in that: The calculation formula for texture similarity includes: For each component i , randomly selected I Matching point pairs and calculate them L Similarity within the neighborhood, N and They respectively represent the pixel points in the neighborhood of the matching point pairs in the original real camera image and the deformed real camera image.

9. A three-dimensional inspection system for augmented reality assisted assembly, characterized in that: include: at least one processor; a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1 to 8.

10. A computer readable medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the method according to any one of claims 1 to 8.

Images