Self-supervised projection geometry correction method and device based on optimal transmission matching

Abstract

The application provides a self-supervised projection geometry correction method and device based on optimal transmission matching, which comprises the following steps: acquiring laser point array phase plane data and camera phase plane data of a projector in a real scene; and performing matching inference on the laser point array phase plane data and the camera phase plane data based on a matching network to obtain a matching relationship matrix; wherein the matching network is based on virtual laser point array phase plane data and virtual camera phase plane data in a virtual scene as training samples, and an initial matching matrix is trained by back propagation with a norm corresponding to a laser point inference matching relationship matrix and a true value matching matrix as a loss function; each matching point in the matching relationship matrix is sequentially subjected to matrix solving and structured light decoding processing, and a projector is subjected to projection geometry correction according to a projection matrix. The method of the application realizes the improvement of deployment efficiency in the multi-path special-shaped projection marker laser point matching link, and saves the human consumption of deployment personnel.

Description

Technical Field

[0001] This invention relates to the field of automatic image correction technology for projectors, and in particular to a self-supervised projection geometry correction method and apparatus based on optimal transmission matching. Background Technology

[0002] Automatic correction of multi-channel irregular projections is an important development direction in the intelligent exhibition, display, and entertainment industry. A fast, accurate, and easy-to-operate multi-channel projection geometric correction method is key to a high-quality intelligent projection method.

[0003] When correcting the projected image, the relevant technologies establish a matching relationship between the laser dot matrix and the camera's acquisition points by manually marking or lighting them sequentially. The projection matrix of the projector is calculated using the pixel correspondence between the projector and the camera and the spatial position of the laser points, thereby completing the projection geometry correction. Whether it is manual marking or sequential lighting, it will lead to a lot of time consumption and manual burden when there are many projection channels, affecting the deployment efficiency of multi-channel irregular projection. Summary of the Invention

[0004] This invention provides a self-supervised projection geometry correction method and apparatus based on optimal transmission matching, which solves the problem that the existing technology of establishing the matching relationship between laser dot matrix and camera acquisition points by manual marking or sequential lighting leads to a large amount of time consumption and manual burden, and improves the deployment efficiency of multi-channel irregular projection.

[0005] This invention provides a self-supervised projection geometry correction method based on optimal transmission matching, comprising:

[0006] Acquire phase plane data of the laser dot matrix of the projector and phase plane data of the camera in a real scene;

[0007] The matching network is used to perform matching inference on the laser dot matrix phase plane data and camera phase plane data to obtain a matching relationship matrix; wherein, the matching network is trained by backpropagation on the initial matching matrix using virtual laser dot matrix phase plane data and virtual camera phase plane data in the virtual scene as training samples, and using the norms corresponding to the laser dot inference matching relationship matrix and the true matching matrix as loss functions.

[0008] The matching points in the matching relationship matrix are sequentially solved by matrix calculation and structured light decoding to obtain the projection matrix, and the projection geometry of the projector is corrected according to the projection matrix.

[0009] According to the present invention, a self-supervised projection geometry correction method based on optimal transmission matching is provided, wherein the initial matching matrix includes a laser point feature extraction subnetwork, a camera point feature extraction subnetwork, an initial point similarity matrix generation module, a matching module, and a similarity metric feedback module;

[0010] The matching network is trained through the following steps:

[0011] The virtual scene is built based on the virtual engine UE, and virtual supervised training data of the projector in the virtual scene is collected. The virtual supervised training data includes the virtual laser dot matrix phase plane data and the virtual camera phase plane data.

[0012] The order of each data item in the virtual supervised training data is shuffled, and the truth matching matrix is ​​determined based on the shuffled virtual supervised training data.

[0013] The laser point feature extraction subnetwork extracts secondary features of the virtual laser point array from the virtual laser point array phase plane data, and the camera point feature extraction subnetwork extracts camera point features from the virtual camera phase plane data.

[0014] The initial point similarity matrix generation module performs normalization processing on the secondary features of the point matrix and the features of the camera points in sequence, and calculates the similarity of each normalized feature to obtain the initial point similarity matrix.

[0015] The initial point similarity matrix is ​​normalized based on the matching module, and the secondary features of the point array and the camera point features are iteratively optimized according to the normalized initial point similarity matrix to obtain the optimal transmission vector. The laser point inference matching relationship matrix is ​​calculated based on the optimal transmission vector and the normalized initial point similarity matrix.

[0016] Based on the similarity metric feedback module, the loss function is determined according to the norm corresponding to the difference between the laser point inferred matching relationship matrix and the true matching matrix. The initial matching network is then iteratively trained using the loss function to obtain the matching network.

[0017] According to the self-supervised projection geometry correction method based on optimal transmission matching provided by the present invention, the step of sequentially performing matrix solving and structured light decoding on each matching point in the matching relationship matrix to obtain the projection matrix includes:

[0018] The essential matrix is ​​obtained by solving the essential matrix based on each matching point in the matching relationship matrix;

[0019] The matching relationship between laser points and camera imaging points in the real scene is solved based on the epipolar line constraints corresponding to the epipolar geometric essential matrix.

[0020] The matching relationship is structured light decoded to obtain the coordinate information of the laser point on the projected image of the projector, and the projection matrix is ​​determined based on the coordinate information.

[0021] According to the present invention, a self-supervised projection geometry correction method based on optimal transmission matching is provided, wherein the virtual laser dot matrix phase plane data includes multiple laser dot matrix coordinates; the laser dot feature extraction subnetwork includes multiple convolutional layers, batch normalization layers, linear rectifier layers, fully connected layers, two-dimensional convolutional layers, and multi-structure residual networks;

[0022] The extraction of secondary features of the laser point points from the virtual laser point array phase plane data based on the laser point point feature extraction subnetwork includes:

[0023] Calculate the k nearest neighbor set of each laser dot matrix coordinate, and arrange the coordinates of each point in the k nearest neighbor set in order of their distance from the center point to obtain the input vector;

[0024] The input vector is processed sequentially through multiple convolutional layers, batch normalization layers, linear rectified layers, and fully connected layers to obtain primary features of the dot matrix.

[0025] The primary features of the lattice points are processed sequentially through the two-dimensional convolutional layer and the multi-structure residual network to obtain the secondary features of the lattice points.

[0026] According to the self-supervised projection geometry correction method based on optimal transmission matching provided by the present invention, the secondary features of the point matrix and the camera point features are iteratively optimized according to the normalized initial point similarity matrix, which is achieved by the following formula:

[0027] v = f2 / (M) norm u)

[0028] u=f1 / (M norm v);

[0029] Where f1 is the secondary feature of the point matrix, u is the optimal transmission vector corresponding to f1, f2 is the camera point feature, v is the optimal transmission vector corresponding to f2, and M norm This is the normalized initial point similarity matrix.

[0030] According to the self-supervised projection geometry correction method based on optimal transmission matching provided by the present invention, the loss function is expressed by the following formula:

[0031] L = ||M final -M gt ||2;

[0032] Where L is the loss function, Mfinal Infer the matching relationship matrix for the laser points, M gt Let be the truth-matching matrix; ||||2 is the 2-norm.

[0033] The present invention also provides a self-supervised projection geometry correction device based on optimal transmission matching, comprising:

[0034] The data acquisition module is used to acquire the laser dot matrix phase plane data and camera phase plane data of the projector in a real scene;

[0035] The matching module is used to perform matching inference on the laser dot matrix phase plane data and camera phase plane data based on the matching network to obtain a matching relationship matrix; wherein, the matching network is obtained by backpropagation training an initial matching matrix using virtual laser dot matrix phase plane data and virtual camera phase plane data in a virtual scene as training samples and using the norms corresponding to the laser dot inference matching relationship matrix and the ground truth matching matrix as loss functions;

[0036] The correction module is used to sequentially perform matrix solving and structured light decoding on each matching point in the matching relationship matrix to obtain the projection matrix, and to perform projection geometry correction on the projector based on the projection matrix.

[0037] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the self-supervised projection geometry correction method based on optimal transmission matching as described above.

[0038] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the self-supervised projection geometry correction method based on optimal transport matching as described above.

[0039] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the self-supervised projection geometry correction method based on optimal transport matching as described above.

[0040] The self-supervised projection geometry correction method and device based on optimal transmission matching provided by this invention uses a matching network pre-trained with virtual laser dot matrix phase plane data and virtual camera phase plane data in a virtual scene to perform matching inference on the laser dot matrix phase plane data and camera phase plane data, and obtain a matching relationship matrix. The matching points in the matching relationship matrix are then solved sequentially and processed by structured light decoding. Finally, the projection geometry of the projector is corrected according to the projection matrix. This improves deployment efficiency in the matching of multi-path irregular projection marker laser points and saves manpower for deployment personnel. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0042] Figure 1 This is one of the flowcharts of the self-supervised projection geometry correction method based on optimal transmission matching provided by the present invention;

[0043] Figure 2 This is the second flowchart of the self-supervised projection geometry correction method based on optimal transmission matching provided by the present invention;

[0044] Figure 3 This is the third flowchart of the self-supervised projection geometry correction method based on optimal transmission matching provided by the present invention;

[0045] Figure 4 This is a schematic diagram of the structure of the self-supervised projection geometry correction device based on optimal transmission matching provided by the present invention;

[0046] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0048] The following is combined with Figures 1-4 The present invention describes a self-supervised projection geometry correction method and apparatus based on optimal transmission matching.

[0049] Figure 1 This is one of the flowcharts illustrating the self-supervised projection geometry correction method based on optimal transmission matching provided by this invention, such as... Figure 1 As shown, the self-supervised projection geometry correction method based on optimal transmission matching includes the following steps:

[0050] Step 110: Obtain the laser dot matrix phase plane data and camera phase plane data of the projector in the real scene.

[0051] In this step, the laser dot matrix phase plane data in the real scene includes the coordinates of laser dots generated when at least one projector projects an image onto a wall or screen in the specified area, and the camera phase plane data includes the imaging points of the camera in the same scene.

[0052] Step 120: Based on the matching network, perform matching inference on the laser dot matrix phase plane data and the camera phase plane data to obtain the matching relationship matrix; wherein, the matching network is trained by backpropagation on the initial matching matrix using virtual laser dot matrix phase plane data and virtual camera phase plane data in the virtual scene as training samples, and using the norms corresponding to the laser dot inference matching relationship matrix and the true matching matrix as loss functions.

[0053] In this step, the matching network makes full use of the basic matrix relationship between corresponding points on the virtual laser dot matrix projection plane and the camera imaging plane, and obtains automatic matching between corresponding points through optimal transmission theory.

[0054] In this step, the virtual scene can be built using Unreal Engine (UE). Specifically, the virtual engine is used to set up a virtual irregular screen. According to the common placement positions of projectors, cameras, and laser dot arrays in different real scenes, multiple projectors, virtual cameras, and virtual laser dot arrays are randomly set up in a specified area. On the virtual irregular screen, the projection area of ​​each projector is divided, and virtual laser dot arrays are generated within the projection range of each projector.

[0055] For example, in the laser dot matrix coordinate system, a plane is set as the virtual phase plane of the laser dot matrix. The projection of each laser point on the virtual dot matrix plane is named the dot matrix imaging point, and its coordinates are the dot matrix point coordinates. The point of the virtual laser point in the virtual camera is named the camera imaging point, and its coordinates are the camera point coordinates. These virtual generated dot matrix point coordinates and camera point coordinates, and their correspondence, are used as the generated virtual supervised training data.

[0056] In this embodiment, the matching network is used to achieve automatic matching between laser dot matrix points and camera points; the matching network may include: a laser dot matrix point feature extraction subnetwork, a camera point feature extraction subnetwork, an initial point similarity matrix generation module, a matching module, and a similarity measurement feedback module; wherein, the matching module may be a Sinkhorn optimal transmission point matching module.

[0057] In this embodiment, the loss function can be determined by taking the 2-norm of the difference between the laser point inference matching relation matrix and the true value matching matrix.

[0058] Step 130: Perform matrix solving and structured light decoding on each matching point in the matching relationship matrix to obtain the projection matrix, and perform projection geometry correction on the projector based on the projection matrix.

[0059] In this step, solving the matrix for each matching point in the matching relationship matrix can be done by solving the row Ransac essential matrix to obtain the epipolar geometric essential matrix between the laser point and the camera point in the real scene.

[0060] In this embodiment, matrix solving and structured light decoding are performed sequentially on each matching point in the matching relationship matrix to obtain the projection matrix. This includes: solving the essential matrix based on each matching point in the matching relationship matrix to obtain the epipolar geometric essential matrix; solving the matching relationship between the laser point in the real scene and the camera imaging point based on the epipolar line constraint corresponding to the epipolar geometric essential matrix; performing structured light decoding on the matching relationship to obtain the coordinate information of the laser point on the projected image of the projector, and determining the projection matrix based on the coordinate information.

[0061] In this embodiment, a trained matching network is used in a real-world scene to perform matching inference on laser dot matrix points and camera points. After inferring the matching relationship matrix, the matching points obtained by discretizing this matrix are used to solve for the Ransac essential matrix, thereby obtaining the epipolar geometric essential matrix between these two sets of points. After obtaining the essential matrix, the epipolar constraint is used to accurately solve for the matching relationship between the two sets of point pairs. After obtaining the accurate point matching relationship, structured light decoding is used to determine the pixel correspondence between the projector and the camera to calculate the position coordinates of the laser points on the projector image, thereby obtaining the projection matrix of the projector and completing the projection geometry correction.

[0062] The self-supervised projection geometry correction method based on optimal transmission matching provided in this invention uses a matching network pre-trained with virtual laser dot matrix phase plane data and virtual camera phase plane data in a virtual scene to perform matching inference on the laser dot matrix phase plane data and camera phase plane data, obtaining a matching relationship matrix. Each matching point in the matching relationship matrix is ​​then solved sequentially using matrix solving and structured light decoding. Finally, the projection geometry of the projector is corrected based on the projection matrix. This improves deployment efficiency in the matching process of multi-path irregular projection marker laser points and saves manpower for deployment personnel.

[0063] In some embodiments, the initial matching matrix includes a laser point feature extraction subnetwork, a camera point feature extraction subnetwork, an initial point similarity matrix generation module, a matching module, and a similarity measurement feedback module. The matching network is trained through the following steps: a virtual scene is built based on the virtual engine UE, and virtual supervised training data of the projector in the virtual scene is collected. The virtual supervised training data includes virtual laser dot matrix phase plane data and virtual camera phase plane data. The order of each data item in the virtual supervised training data is shuffled, and the true matching matrix is ​​determined based on the shuffled virtual supervised training data. Secondary features of the laser dot matrix are extracted from the virtual laser dot matrix phase plane data based on the laser point feature extraction subnetwork, and phase features are extracted from the virtual camera phase plane data based on the camera point feature extraction subnetwork. The initial point similarity matrix generation module normalizes the secondary features of the matrix points and the camera point features sequentially, and calculates the similarity of each normalized feature to obtain the initial point similarity matrix. The matching module normalizes the initial point similarity matrix and iteratively optimizes the secondary features of the matrix points and the camera point features based on the normalized initial point similarity matrix to obtain the optimal transmission vector. The laser point inference matching relationship matrix is ​​then calculated based on the optimal transmission vector and the normalized initial point similarity matrix. Finally, the similarity metric feedback module determines the loss function based on the norm corresponding to the difference between the laser point inference matching relationship matrix and the ground truth matching matrix, and iteratively trains the initial matching network using the loss function to obtain the matching network.

[0064] In this embodiment, after acquiring virtual supervised training data and shuffling the order of each data point, the shuffled data is input into a matching network for inference matching; wherein, the shuffled matching relationship is represented by the true matching matrix M. gt .

[0065] Figure 2 This is the second flowchart of the self-supervised projection geometry correction method based on optimal transmission matching provided by this invention. Figure 2 In the illustrated embodiment, virtual supervised training data (corresponding to virtual self-supervised training data) is first acquired, and then the laser marker matching network is trained using the training data to obtain a usable matching network. Finally, the laser markers in the real scene are accurately matched using the usable matching network, thereby achieving geometric correction of the projector image.

[0066] The functions and working mechanisms of each module in the matching network are as follows:

[0067] (1) Secondary features of the laser point array are extracted from the phase plane data of the virtual laser point array through the laser point feature extraction subnetwork.

[0068] In this embodiment, the virtual laser dot matrix phase plane data includes multiple laser dot matrix coordinates; the laser dot feature extraction subnetwork includes multiple convolutional layers, batch normalization layers, linear rectifier layers, fully connected layers, two-dimensional convolutional layers, and a multi-structure residual network; extracting secondary features of the dot matrix points from the virtual laser dot matrix phase plane data based on the laser dot feature extraction subnetwork includes: calculating the k nearest neighbor set of each laser dot matrix coordinate, and arranging the coordinates of each point in the k nearest neighbor set in order of distance from the center point to obtain an input vector; processing the input vector sequentially through multiple convolutional layers, batch normalization layers, linear rectifier layers, and fully connected layers to obtain primary features of the dot matrix points; and processing the primary features of the dot matrix points sequentially through two-dimensional convolutional layers and a multi-structure residual network to obtain secondary features of the dot matrix points.

[0069] In this embodiment, the value of k can be set according to user needs, for example, k = 10.

[0070] In this embodiment, the laser dot matrix feature extraction subnetwork in the matching network takes the coordinates of all laser dot matrix points as input. For each point, a set of k nearest neighbors is calculated. The coordinates of the k nearest neighbors are arranged in order of their distance from the center point. The two-dimensional vector of the k nearest neighbor coordinates arranged in order is used as input and passes through a primary dot matrix feature generation network consisting of 5 layers of unit networks, including convolutional layers, batch normalization layers, linear rectified layers, and fully connected layers, to generate primary dot matrix features. Finally, the primary dot matrix features are taken as input and pass through a secondary dot matrix feature extraction network consisting of a two-dimensional convolutional network and 13 layers of ResNet to obtain secondary dot matrix features.

[0071] (2) Extract camera point features from the virtual camera phase plane data through the camera point feature extraction subnetwork.

[0072] Specifically, the input to the camera point feature extraction subnetwork is the coordinates of all laser points under the camera coordinate system. For each point, a set of k nearest neighbors is calculated (k is set to 10). The coordinates of the k nearest neighbors are arranged in order of their distance from the center point. The two-dimensional vector of the k nearest neighbor coordinates arranged in order is used as input and passed through a network with the same structure as the point matrix secondary feature generation network to generate camera point features.

[0073] (3) Obtain the initial point similarity matrix through the initial point similarity matrix generation module.

[0074] Specifically, the initial point similarity matrix generation module takes as input the secondary features of the point matrix and the features of the camera points. First, two sets of feature points are input and passed through a batch normalization layer. The L2 norm of each pair of combinations of the two sets of points after batch normalization is measured. The similarity of each combination constitutes the corresponding elements of the similarity matrix, thus generating the initial point similarity matrix.

[0075] (4) The matching relationship matrix is ​​inferred by calculating the laser points through the matching module.

[0076] Specifically, the matching network includes the Sinkhorn optimal transmission point matching module. The inputs to this module are: secondary features f1 of the matrix points, features f2 of the camera points, and the initial point similarity matrix M. M is first normalized according to the following formula, as shown below:

[0077]

[0078] Among them, M norm Let i be the row of M and j be the column of M after normalization of the initial point similarity matrix; then initialize the optimal transmission vector: u = f1.

[0079] In this embodiment, the secondary features of the point matrix and the camera point features are iteratively optimized based on the normalized initial point similarity matrix, as achieved by the following formula:

[0080] v = f2 / (M) norm u)

[0081] u=f1 / (M norm v);

[0082] Where f1 is the secondary feature of the raster points, u is the optimal transmission vector corresponding to f1, f2 is the camera point feature, v is the optimal transmission vector corresponding to f2, and M norm The initial point similarity matrix is ​​normalized; this continues until the convergence conditions for u and v are met (the change in u and v is less than a given threshold, or the number of iterations reaches a set maximum value); finally, the final laser point inference matching relationship matrix is ​​calculated according to the following formula:

[0083] M final =M norm u T v;

[0084] M final Infer the matching relationship matrix for the laser points.

[0085] (5) The network parameters are updated according to the loss function through the similarity metric feedback module to obtain the trained matching network.

[0086] In this embodiment, the loss function is expressed by the following formula:

[0087] L = ||M final -M gt ||2;

[0088] Where L is the loss function, M final To infer the matching relationship matrix for laser points, M gt is the truth-matching matrix; ||||2 is the 2-norm.

[0089] The self-supervised projection geometry correction method based on optimal transmission matching provided in this invention utilizes the basic matrix relationship between corresponding points on the virtual laser dot matrix projection plane and the camera imaging plane, and obtains automatic matching between corresponding points by using optimal transmission theory, thereby training a laser dot matching network and improving the accuracy of projection geometry correction in real-world environments.

[0090] Figure 3 This is the third flowchart of the self-supervised projection geometry correction method based on optimal transmission matching provided by this invention. Figure 3 In the illustrated embodiment, virtual projection points of the dot matrix are input into the laser dot matrix feature extraction subnetwork to obtain secondary features of the dot matrix points, and virtual camera projection points are input into the camera point feature extraction subnetwork to obtain camera point features. The dot matrix point secondary features and camera point features are then normalized and similarity calculated sequentially by the initial point similarity matrix generation module to obtain the initial point similarity matrix. The initial point similarity matrix is ​​then normalized by the Sinkhorn optimal transmission point matching module, and the two types of features are iteratively optimized to obtain the optimal transmission vector. The laser point inference matching relationship matrix is ​​calculated based on the optimal transmission vector and the normalized initial point similarity matrix. Finally, the similarity metric feedback module determines the loss function based on the norm corresponding to the difference between the laser point inference matching relationship matrix and the ground truth matching matrix to train the initial matching network, thus obtaining the matching network.

[0091] The self-supervised projection geometry correction device based on optimal transmission matching provided by the present invention will be described below. The self-supervised projection geometry correction device based on optimal transmission matching described below can be referred to in correspondence with the self-supervised projection geometry correction method based on optimal transmission matching described above.

[0092] Figure 5 This is a schematic diagram of the structure of the self-supervised projection geometry correction device based on optimal transmission matching provided by the present invention, as shown below. Figure 4 As shown, the self-supervised projection geometry correction device based on optimal transmission matching includes a data acquisition module 410, a matching module 420, and a correction module 430.

[0093] Data acquisition module 410 is used to acquire laser dot matrix phase plane data and camera phase plane data of the projector in a real scene;

[0094] The matching module 420 is used to perform matching inference on the laser dot matrix phase plane data and the camera phase plane data based on the matching network to obtain the matching relationship matrix; wherein, the matching network is trained by backpropagation on the initial matching matrix using virtual laser dot matrix phase plane data and virtual camera phase plane data in the virtual scene as training samples, and using the norms corresponding to the laser dot inference matching relationship matrix and the true matching matrix as loss functions.

[0095] The correction module 430 is used to sequentially solve the matrix and decode the structured light for each matching point in the matching relationship matrix to obtain the projection matrix, and to perform projection geometry correction on the projector based on the projection matrix.

[0096] The self-supervised projection geometry correction device based on optimal transmission matching provided in this invention uses a matching network pre-trained with virtual laser dot matrix phase plane data and virtual camera phase plane data in a virtual scene to perform matching inference on the laser dot matrix phase plane data and camera phase plane data, obtaining a matching relationship matrix. Each matching point in the matching relationship matrix is ​​then solved sequentially using matrix solving and structured light decoding. Finally, the projection geometry of the projector is corrected according to the projection matrix. This improves deployment efficiency in the multi-path irregular projection marker laser point matching process and saves manpower for deployment personnel.

[0097] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 5 As shown, the electronic device may include a processor 510, a communication interface 520, a memory 530, and a communication bus 540. The processor 510, communication interface 520, and memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a self-supervised projection geometry correction method based on optimal transmission matching. This method includes: acquiring laser dot matrix phase plane data and camera phase plane data of the projector in a real scene; performing matching inference on the laser dot matrix phase plane data and camera phase plane data based on a matching network to obtain a matching relationship matrix; wherein the matching network is trained by backpropagation on an initial matching matrix using virtual laser dot matrix phase plane data and virtual camera phase plane data in a virtual scene as training samples, and using the norms corresponding to the laser dot inference matching relationship matrix and the true matching matrix as loss functions; sequentially performing matrix solving and structured light decoding on each matching point in the matching relationship matrix, and performing projection geometry correction on the projector according to the projection matrix.

[0098] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0099] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the self-supervised projection geometry correction method based on optimal transmission matching provided by the above methods. The method includes: acquiring laser dot matrix phase plane data and camera phase plane data of the projector in a real scene; performing matching inference on the laser dot matrix phase plane data and camera phase plane data based on a matching network to obtain a matching relationship matrix; wherein the matching network is obtained by backpropagation training an initial matching matrix using virtual laser dot matrix phase plane data and virtual camera phase plane data in a virtual scene as training samples, and using the norms corresponding to the laser dot inference matching relationship matrix and the true matching matrix as loss functions; performing matrix solving and structured light decoding on each matching point in the matching relationship matrix in sequence, and performing projection geometry correction on the projector according to the projection matrix.

[0100] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the self-supervised projection geometry correction method based on optimal transmission matching provided by the above methods. The method includes: acquiring laser dot matrix phase plane data and camera phase plane data of the projector in a real scene; performing matching inference on the laser dot matrix phase plane data and camera phase plane data based on a matching network to obtain a matching relationship matrix; wherein the matching network is obtained by backpropagation training an initial matching matrix using virtual laser dot matrix phase plane data and virtual camera phase plane data in a virtual scene as training samples, and using the norms corresponding to the laser dot inference matching relationship matrix and the true matching matrix as loss functions; performing matrix solving and structured light decoding on each matching point in the matching relationship matrix in sequence, and performing projection geometry correction on the projector according to the projection matrix.

[0101] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0102] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A self-supervised projection geometry correction method based on optimal transmission matching, characterized in that, include: Acquire phase plane data of the laser dot matrix of the projector and phase plane data of the camera in a real scene; The matching network is used to perform matching inference on the laser dot matrix phase plane data and camera phase plane data to obtain a matching relationship matrix; wherein, the matching network is obtained by backpropagation training of an initial matching network using virtual laser dot matrix phase plane data and virtual camera phase plane data in a virtual scene as training samples, and using the norms corresponding to the laser dot inference matching relationship matrix and the ground truth matching matrix as loss functions. The matching points in the matching relationship matrix are sequentially subjected to matrix solving and structured light decoding to obtain the projection matrix, and the projection geometry of the projector is corrected based on the projection matrix; the initial matching network includes a laser point feature extraction subnetwork, a camera point feature extraction subnetwork, an initial point similarity matrix generation module, a matching module, and a similarity measurement feedback module; The matching network is trained through the following steps: The virtual scene is built based on the virtual engine UE, and virtual supervised training data of the projector in the virtual scene is collected. The virtual supervised training data includes the virtual laser dot matrix phase plane data and the virtual camera phase plane data. The order of each data item in the virtual supervised training data is shuffled, and the truth matching matrix is ​​determined based on the shuffled virtual supervised training data. The laser point feature extraction subnetwork extracts secondary features of the virtual laser point array from the virtual laser point array phase plane data, and the camera point feature extraction subnetwork extracts camera point features from the virtual camera phase plane data. The initial point similarity matrix generation module performs normalization processing on the secondary features of the point matrix and the features of the camera points in sequence, and calculates the similarity of each normalized feature to obtain the initial point similarity matrix. The initial point similarity matrix is ​​normalized based on the matching module, and the secondary features of the point array and the camera point features are iteratively optimized according to the normalized initial point similarity matrix to obtain the optimal transmission vector. The laser point inference matching relationship matrix is ​​calculated based on the optimal transmission vector and the normalized initial point similarity matrix. Based on the similarity metric feedback module, the loss function is determined according to the norm corresponding to the difference between the laser point inferred matching relationship matrix and the true matching matrix. The initial matching network is then iteratively trained using the loss function to obtain the matching network. The process of sequentially solving the matrix and performing structured light decoding on each matching point in the matching relationship matrix to obtain the projection matrix includes: The essential matrix is ​​obtained by solving the essential matrix based on each matching point in the matching relationship matrix; The matching relationship between laser points and camera imaging points in the real scene is solved based on the epipolar line constraints corresponding to the epipolar geometric essential matrix. The matching relationship is structured light decoded to obtain the coordinate information of the laser point on the projected image of the projector, and the projection matrix is ​​determined based on the coordinate information.

2. The self-supervised projection geometry correction method based on optimal transmission matching according to claim 1, characterized in that, The virtual laser dot matrix phase plane data includes multiple laser dot matrix coordinates; the laser dot feature extraction subnetwork includes multiple convolutional layers, batch normalization layers, linear rectifier layers, fully connected layers, two-dimensional convolutional layers, and multi-structured residual networks; The extraction of secondary features of the laser point points from the virtual laser point array phase plane data based on the laser point point feature extraction subnetwork includes: Calculate the k nearest neighbor set of each laser dot matrix coordinate, and arrange the coordinates of each point in the k nearest neighbor set in order of their distance from the center point to obtain the input vector; The input vector is processed sequentially through multiple convolutional layers, batch normalization layers, linear rectified layers, and fully connected layers to obtain primary features of the dot matrix. The primary features of the lattice points are processed sequentially through the two-dimensional convolutional layer and the multi-structure residual network to obtain the secondary features of the lattice points.

3. The self-supervised projection geometry correction method based on optimal transmission matching according to claim 1, characterized in that, The step of iteratively optimizing the secondary features of the point matrix and the camera point features based on the normalized initial point similarity matrix is ​​achieved through the following formula: ； in, f 1 represents the secondary feature of the lattice points. u for f The optimal transmission vector corresponding to 1 f 2 represents the camera point features. v for f The optimal transmission vector corresponding to 2, M norm This is the normalized initial point similarity matrix.

4. The self-supervised projection geometry correction method based on optimal transmission matching according to claim 1, characterized in that, The loss function is expressed by the following formula: L =|| M final - M gt ||2； in, L Let the loss function be... M final Infer the matching relationship matrix for the laser points. M gt Let be the truth-matching matrix; || ||2 is the 2-norm.

5. A self-supervised projection geometry correction device based on optimal transmission matching, applied to the self-supervised projection geometry correction method based on optimal transmission matching as described in claim 1, characterized in that, include: The data acquisition module is used to acquire the laser dot matrix phase plane data and camera phase plane data of the projector in a real scene; The matching module is used to perform matching inference on the laser dot matrix phase plane data and camera phase plane data based on the matching network to obtain a matching relationship matrix; wherein, the matching network is obtained by backpropagation training an initial matching network using virtual laser dot matrix phase plane data and virtual camera phase plane data in a virtual scene as training samples, and using the norms corresponding to the laser dot inference matching relationship matrix and the ground truth matching matrix as loss functions; The correction module is used to sequentially perform matrix solving and structured light decoding on each matching point in the matching relationship matrix to obtain the projection matrix, and to perform projection geometry correction on the projector based on the projection matrix.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the self-supervised projection geometry correction method based on optimal transmission matching as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the self-supervised projection geometry correction method based on optimal transmission matching as described in any one of claims 1 to 4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the self-supervised projection geometry correction method based on optimal transmission matching as described in any one of claims 1 to 4.

Images