A projection fusion method based on neural network

Abstract

The present invention relates to the field of artificial intelligence technology, and specifically to a projection fusion method based on a neural network, including: obtaining an illumination image, a projection image, and a target image, obtaining a projection overlap area according to the illumination image, obtaining a homography matrix of each first pixel according to the projection image, obtaining a homography matrix of each second pixel and the degree of accuracy according to the homography matrix of each first pixel, obtaining a first fused target image and a second fused target image of each second pixel according to the homography matrix of the second pixel and a sub-projection image, and finally obtaining a projection offset feature vector; obtaining a projector adjustment parameter using a neural network according to the projection offset feature vector, and adjusting the projector. The present invention obtains a parameter adjustment method of a projector through a projection image and a target image, so that the adjustment process is simple and fast, and an accurate projection fusion result can be obtained on a curved screen, avoiding erroneous texture information in the projection image.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a projection fusion method based on a neural network. Background Art

[0002] In command and monitoring centers, network management centers, video conferences, academic reports, technical lectures, multi-functional conference rooms and other scenes, in order to improve visual experience, the demand for large screen, multi-color, high brightness, and high-resolution display effects is becoming more and more intense. In the past, it was difficult to project high-resolution images through a single projector. Nowadays, through projection fusion technology, images projected by multiple projectors can be spliced ​​and fused together to achieve high-resolution display effects and improve visual experience.

[0003] However, most existing projection fusion methods require manual adjustment of the projector's posture and parameters to align the textures of images projected by different projectors, and then use edge fusion algorithms to obtain the final projection effect. However, the manual adjustment process is very troublesome. If the projector position is offset or the equipment is replaced, or the projection facilities are rearranged or repaired, the parameters need to be manually adjusted, leading to various problems, such as low meeting efficiency, delayed meeting time, or abnormal images.

[0004] Some existing automated projection fusion methods and systems require heavy work such as building a three-dimensional model of the projection scene, setting control points, measuring the scene, calibrating equipment, etc., and are very easy to introduce errors. Summary of the invention

[0005] In order to solve the above technical problems, the purpose of the present invention is to provide a projection fusion method based on a neural network, and the technical solution adopted is as follows:

[0006] The present invention provides a projection fusion method based on a neural network, the method comprising the following steps:

[0007] Collect the illumination images projected by all projectors, and obtain the overlapping area of ​​the projector projections through the threshold segmentation algorithm;

[0008] Each pixel on the projection image and outside the projection overlap area is regarded as each first pixel, and a sub-projection image consisting of all pixels in a preset window is obtained with each first pixel as the center, and a homography matrix of each first pixel is obtained according to the corner points of the sub-projection image of each first pixel and the corner points of the target image;

[0009] Consider each pixel on the projected image and in the projection overlap area as each second pixel, obtain a preset number of first pixels having the smallest Euclidean distance to each second pixel, and obtain the homography matrix and accuracy of each second pixel according to the clustering results of the homography matrices of all the preset number of first pixels;

[0010] Taking each second pixel as the center, a sub-projection image composed of all pixels in a preset window is obtained, and a first fused target image of each second pixel is obtained by using the homography matrix and the sub-projection image of each second pixel and combining with the target image;

[0011] Obtaining a second fused target image for each second pixel according to the sub-projection image of each second pixel and its first fused target image;

[0012] Combined with the accuracy of all second pixels, the projection offset feature vector of the projector is obtained according to the first fusion target image and the second fusion target image of all second pixels. The projection offset feature vector is input into the neural network, and the network outputs the adjustment parameters of the projector. The projector is adjusted according to the adjustment parameters and re-projects the image.

[0013] Furthermore, the step of acquiring the projection offset feature vector includes:

[0014] Obtaining a first fused mean value of all pixel coordinates in the first fused target image of each second pixel, and a second fused mean value of all pixel coordinates in the second fused target image of each second pixel, obtaining a displacement vector formed by the first fused mean value to the second fused mean value, and merging the displacement vector and the first fused mean value into a vector, which is called an offset vector of each second pixel;

[0015] The mean shift algorithm is used to cluster the offset vectors of all second pixels to obtain all categories, and a weighted sum is performed on the offset vectors in each category, and the result obtained is called the reference offset vector of each category; the weight of the weighted sum is the accuracy of the second pixel corresponding to each offset vector in each category; the reference offset vectors of all categories are merged into a high-dimensional vector, which is called a projection offset feature vector.

[0016] Furthermore, the step of acquiring the homography matrix and accuracy of each second pixel includes:

[0017] For a preset number of first pixels having the smallest Euclidean distance to each second pixel, obtaining a set of all homography matrices of the preset number of first pixels, flattening each homography matrix into a one-dimensional vector, clustering the homography matrices in the set to obtain all categories, obtaining a category containing the largest number of homography matrices among all categories, calculating a first mean of all homography matrices in the category, and using the first mean as the homography matrix of each second pixel;

[0018] Obtaining a first pixel set corresponding to all homography matrices in the category, calculating a second mean of pixel coordinates in the first pixel set, calling the Euclidean distance between the coordinate of each second pixel and the second mean a distance factor of each second pixel, and then calculating an L2 norm of all homography matrices in the category and the first mean, and the variance of all the L2 norms is called a distribution factor of each second pixel;

[0019] The reciprocal of the product of the distance factor of each second pixel and the distribution factor of each second pixel is used as the accuracy of each second pixel.

[0020] Furthermore, the step of acquiring the first fusion target image of each second pixel includes:

[0021] Obtaining homogeneous coordinates of all pixels on the projected sub-image of each second pixel, multiplying the homogeneous coordinates of all pixels by the homography matrix of each second pixel, and thereby obtaining all new pixel coordinates;

[0022] Then, a brightness map of the target image is obtained, and the grayscale values ​​corresponding to the new pixel coordinates on the brightness map of the target image are obtained. The new pixel coordinates and the corresponding grayscale values ​​constitute a first fused target image for each second pixel.

[0023] Furthermore, the step of acquiring the second fused target image of each second pixel includes:

[0024] Acquire the pixel coordinates of the sub-projection image of each second pixel and the grayscale value of the pixel corresponding to the pixel coordinates on the first fused target image of each second pixel, and construct a temporary image of each second pixel with the pixel coordinates and the grayscale value;

[0025] Obtaining a difference between the sub-projection image of each second pixel and the temporary image of each second pixel, called a difference image, obtaining the minimum grayscale value in the difference image, subtracting the grayscale value of each pixel in the difference image from the minimum grayscale value, and then normalizing the difference image;

[0026] Using a corner point detection algorithm to obtain all corner points of the difference image, using a corner point matching algorithm to match all corner points on the difference image with all corner points on the target image, obtaining all matched pairs of corner points, and then fitting a homography matrix using all the corner point pairs as data samples;

[0027] Obtaining homogeneous coordinates of all pixels on the difference image, multiplying the homogeneous coordinates of all pixels by the homography matrix, and then obtaining all new pixel coordinates;

[0028] The grayscale value corresponding to the new pixel coordinates on the target image brightness map is obtained, and the new pixel coordinates and the corresponding grayscale value constitute a second fused target image for each second pixel.

[0029] Furthermore, the step of acquiring the sub-projection image of each first pixel and the sub-projection image of each second pixel comprises:

[0030] A rectangular preset window is established with each first pixel as the center, and all pixels in the window and belonging to the projected image are obtained; if there are pixels belonging to the projection overlap region among all the pixels, the pixel values ​​of the pixels belonging to the projection overlap region are set to zero vectors; then the pixel coordinates and corresponding pixel values ​​of all the pixels on the projected image are obtained, and the brightness map of the image constructed by the coordinates and pixel values ​​of all the pixels is called a sub-projection image of each first pixel;

[0031] A rectangular window is established with each second pixel as the center and a preset neighborhood range as the size, and all pixels within the window and belonging to the projected image are obtained. If there are pixels outside the projection overlap area among all the pixels, the pixel values ​​of the pixels outside the projection overlap area are set to a zero vector; then the pixel coordinates and corresponding pixel values ​​of all the pixels on the projected image are obtained, and the brightness map of the image constructed by all the pixel coordinates and pixel values ​​is called the sub-projection image of each second pixel.

[0032] Furthermore, the step of acquiring the homography matrix of each first pixel includes:

[0033] A corner detection algorithm is used to obtain all corner points of the sub-projection image of each first pixel, a corner detection algorithm is used to obtain all corner points of the target image, a corner matching algorithm is used to match all corner points on the sub-projection image of each first pixel with all corner points on the target image to obtain all matched pairs of corner points, and then all corner point pairs are used as data samples to fit a homography matrix, which is the homography matrix of each first pixel.

[0034] Furthermore, the step of obtaining the projection overlap area includes:

[0035] All projectors project white light onto the screen, use a camera to capture the white light image on the screen, convert the image into a grayscale image, use the Otsu threshold segmentation algorithm to segment the grayscale image, obtain all connected domains, calculate the mean of the grayscale values ​​of all pixels in each connected domain, and obtain the connected domain with the largest mean as the projection overlap area;

[0036] Among all the projectors, one is a projector whose parameters have been adjusted, and the others are projectors whose parameters have not been adjusted.

[0037] Furthermore, the neural network is a fully connected neural network, the data set of the neural network is generated by a simulator, the loss function of the neural network is a mean square error loss function, and the neural network is trained using a stochastic gradient descent algorithm based on the loss function combined with the data set, so that the neural network converges.

[0038] Furthermore, the target image refers to a high-resolution image that needs to be projected onto the screen by the projector, and the target image has texture information; the projected image refers to an image obtained by capturing the camera after the target image is projected onto the screen by the projector, and the camera is an RGB camera installed on the projector.

[0039] The present invention has the following beneficial effects:

[0040] The present invention obtains adjustment parameters of the projector through the target image to be projected and the projection image captured by the camera installed on the projector. After the projector adjusts the parameters, the texture of the image projected by the projector can be aligned to obtain a better projection fusion effect. There is no need to introduce other heavy processing work and a large amount of auxiliary data. The equipment is simple, and only a camera is added to the projector. No additional equipment is required. The implementation is convenient and applicable to a variety of scenarios.

[0041] The present invention estimates the homography matrix of the projection image and the target image within the projection fusion area through the homography matrix of the projection image and the target image outside the projection fusion area, so that the present invention can perform projection fusion on a curved screen; further, the present invention obtains the offset vector of the target fusion image projected by the projector at different positions relative to the target fusion image projected by other projectors based on the homography matrix of the projection image and the target image within the projection fusion area, fuses the offset vectors at different positions, obtains the final projection offset feature vector, avoids errors when projecting on the curved screen, and ensures the accuracy of the calculation results.

[0042] The present invention obtains adjustment parameters of the projector by projecting an offset feature vector, so that projection fusion is fast and accurate and automation is easy to achieve. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 A flowchart of a projection fusion method based on a neural network provided by an embodiment of the present invention; DETAILED DESCRIPTION

[0045] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the specific implementation, structure, features and effects of a projection fusion method based on a neural network proposed by the present invention in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures, or characteristics in one or more embodiments may be combined in any suitable form.

[0046] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0047] The following is a detailed description of a specific scheme of a projection fusion method based on a neural network provided by the present invention in conjunction with the accompanying drawings.

[0048] See also Figure 1 , which shows a projection fusion method based on a neural network of the present invention, characterized in that the method comprises the following steps:

[0049] Step S001, obtaining a projection overlap area according to the light projected by two projectors.

[0050] The present invention is described by taking two projectors projecting images onto a curved screen as an example. The present invention first manually adjusts the parameters of one of the projectors, and the parameters include the posture viewing angle of the projector and the focal length of the projector projected image, so that one of the projectors can project a clear image at the correct position of the screen. Next, the present invention only needs to project and fuse the image projected by the other projector with the image projected by the projector with adjusted parameters. Therefore, the present invention mainly analyzes the other projector next. For the convenience of describing the present invention, the projector described later refers to the other projector described above unless otherwise specified.

[0051] First, two projectors project white light onto a screen, the screen is illuminated, and the light in the overlapping area of ​​the two projectors' projections is relatively bright; the present invention installs an RGB camera on the projector, the camera's viewing angle is toward the screen, and the image on the screen can be captured. When the two projectors project white light onto the screen, the image captured by the camera is an illumination image; the illumination image is grayed to obtain a grayscale image, the grayscale image is segmented using the Otsu threshold segmentation algorithm, all connected domains are obtained, the mean of the grayscale values ​​of all pixels in each connected domain is calculated, and the connected domain with the largest mean is obtained as the projection overlapping area.

[0052] Step S002: Obtain the homography matrix of the pixel points outside the projection overlap area according to the projection image and the target image.

[0053] Two projectors need to project a high-resolution and clear image onto a screen. The present invention refers to this high-resolution image as a target image. The target image has rich texture details. The target image is divided into two sub-images. The two sub-images have overlapping pixel areas. The two projectors project the two sub-images onto the screen respectively, and the two images are displayed on the screen.

[0054] The present invention expects that the two images display the same texture at the same position in the projection overlap area, so as to ensure that the projected image does not produce erroneous texture. However, in reality, since the projector parameters have not been set properly, the two images display different textures at the same position in the projection overlap area, resulting in the image displayed on the screen having erroneous texture.

[0055] The present invention first uses the camera on the projector to capture the image displayed on the screen, which is called the projection image, that is, the texture information of the projection image in the projection overlap area is wrong; in summary, the texture information of each position on the projection image in the projection overlap area is the superposition of the texture information of two positions on the two sub-images into which the target image is divided, or it can be said that the texture information of each position on the projection image in the projection overlap area is the superposition of the texture information of two positions on the target image. Then the present invention needs to obtain the two positions and areas corresponding to the position or area on the target image when a certain position or area on the projection image within the projection overlap area is given, but because the texture information on the projection image within the projection overlap area is wrong, the above result cannot be directly obtained. The present invention provides the following method:

[0056] Considering that the screen is curved, but the local area of ​​the screen is considered to be flat, based on this, the present invention obtains each pixel on the projected image and outside the projection overlap area. These pixels are collectively referred to as first pixels. For any first pixel p, a rectangular preset window is established with the first pixel p as the center. The window size is K×K (K=127×127 in the present invention). This window corresponds to a local area on the screen, and this local area on the screen is considered to be flat. Obtain all pixels within the window and belonging to the projected image. These pixels are recorded as a pixel set ,if When there are pixels in the projection overlap area, the pixel values ​​of the pixels in the projection overlap area are set to zero vectors; then get The pixel coordinates and corresponding pixel values ​​of the pixels in the image. It should be noted that since the projected image is an RGB image, the pixel value of each pixel is a three-dimensional vector, and each dimension of the three-dimensional vector corresponds to three RGB channels; these pixel coordinates and pixel values ​​are constructed into an image, which is also an RGB image, and the brightness map of the image is obtained. ,Will It is called the sub-projection image of the first pixel p. It should be noted that. The coordinates of the pixels on the image are the same as the corresponding pixel coordinates on the projected image, but It is a single-channel grayscale image.

[0057] The method for obtaining the brightness map of the RGB image is: converting the RGB image to the HSV color space, obtaining the brightness component in the HSV color space, and the brightness component is the brightness map of the RGB image.

[0058] Similarly, obtain each pixel on the projected image and in the projection overlap area. These pixels are collectively referred to as second pixels. For any second pixel q, take the second pixel q as the center and establish a rectangular preset window with a window size of K×K (K=127×127 in the present invention). This window corresponds to a local area on the screen. This local area on the screen is considered to be a plane. Obtain all pixels within the window and belonging to the projected image. These pixels are recorded as a pixel set ,if When there are pixels outside the projection overlap area in , the pixel values ​​of the pixels outside the projection overlap area are set to zero vectors; then get The pixel coordinates and corresponding pixel values ​​of the pixels in the image are constructed into an image, and the brightness map of the image is obtained. ,Will It is called the sub-projection image of the second pixel q.

[0059] It should be noted that, as described in step S002, the texture information of each position on the projected image within the projection overlap area is formed by superimposing the texture information of two positions on the target image; then it can be known that the sub-projection image of the second pixel q The texture information of each position on the target image is the superposition of the texture information of two positions on the target image.

[0060] For the sub-projected image of the first pixel p , It is a part of the projected image obtained after the target image is projected onto the screen by the projector, that is, The texture in the target image is the same as the texture of a certain area on the target image. In the present invention, the area with the same texture on the target image cannot be directly obtained in consideration of the projector and camera viewing angle. The present invention provides the following method:

[0061] Use Harris corner detection algorithm to obtain All corner points on the image and all corner points on the target image are matched using a corner matching algorithm based on normalized cross correlation (NCC). The corner points on the image are matched with the corner points on the target image to obtain all the matched corner point pairs. The corner point coordinates of all corner point pairs are used as data samples, and a homography matrix is ​​fitted using the RANSC algorithm. , the matrix satisfies , Represents the homogeneous coordinates of any pair of corner point coordinates, where v represents the homogeneous coordinates of the corner point of the projected image, and u represents the coordinates of the corner point of the target image. It represents the mapping relationship or affine transformation relationship between the local pixel area on the projected image and the local pixel area with the same texture on the target image. It is called the homography matrix of the first pixel p.

[0062] Similarly, the homography matrix of all first pixels is obtained.

[0063] Step S003: Obtain a homography matrix of a second pixel according to the homography matrix of the first pixel.

[0064] What the present invention actually hopes to obtain is the mapping relationship between the local area within the projection fusion area and on the projected image and the target image, that is, the homography matrix of the second pixel; however, since the texture of the local area on the projected image within the projection fusion area is wrong, it is impossible to obtain the homography moment of the second pixel like obtaining the homography matrix of the first pixel as mentioned above.

[0065] The present invention believes that the homography matrices of adjacent pixels on the projected image are almost equal, because the screen is a regular curved surface, and there will be no sudden changes in the curvature of adjacent positions on the curved surface. Based on this, the present invention uses the homography matrix of the first pixel to obtain the homography matrix of the second pixel.

[0066] For any second pixel q, obtain the N (N=64 in the present invention) first pixels closest to the second pixel q on the projected image. The present invention considers that the homography matrix of the second pixel q is related to the homography matrix of the N first pixels. The method for obtaining the homography matrix of the second pixel q is:

[0067] Get the set of homography matrices of these N first pixels , flatten the homography matrix into a one-dimensional vector, and use the mean shift clustering algorithm to Cluster the homography matrices in to obtain all categories, and obtain the category with the largest number of homography matrices among all categories, recorded as , the homography matrix in this category contains many similar homography matrices that are concentrated together, so the homography matrix of the second pixel q is ,in The category The number of homography matrices contained in , represents the i-th homography matrix in this category, Indicates that The mean of all homography matrices in the first pixel q, that is, the present invention considers that the homography matrix of the second pixel q is the mean of a large number of homography matrices in the homography matrices of the first pixels closest to the second pixel q, so that the homography matrix of the second pixel q is as accurate as possible to avoid introducing noise; however, the present invention needs to further describe the accuracy of the homography matrix obtained by the second pixel q;

[0068] The accuracy of the second pixel q is ,in The calculation method is: All homography matrices in The L2 norm of , the variance of all L2 norms is , The larger the The more discrete the homography matrix distribution in is, the more The clustering results obtained The lower the accuracy; The calculation method is: Get The pixel coordinates of the first pixel corresponding to all homography matrices in , calculate the mean of these pixel coordinates, then It represents the Euclidean distance between the mean and the coordinates of the second pixel p. The larger the value, the farther the distance between the second pixel p and the nearest first pixel where the homography matrix is ​​concentrated is. The lower the accuracy.

[0069] So far, the present invention uses the accuracy of the second pixel to describe the accuracy of the homography matrix of the second pixel, so as to avoid introducing too many errors later.

[0070] Similarly, the homography matrix and accuracy of all second pixels are obtained.

[0071] Step S004: Obtain a first fused target image and a second fused target image of the second pixel according to the homography matrix of the second pixel.

[0072] For the second pixel p, its sub-projection image is , then according to its homography matrix You can get the target image with Pixel areas with the same texture, the specific method is: Get The homogeneous coordinate a of any pixel of , b also represents a homogeneous coordinate, obtain the two-dimensional coordinate represented by b, and obtain the pixel at the coordinate on the target image, that is, according to Available The pixel corresponding to any pixel in the target image can be obtained The corresponding pixels of all pixels on the target image, the pixel coordinates of these pixels and the grayscale values ​​of these pixels on the brightness map of the target image constitute a sub-image, which is a single-channel grayscale image and is called the first fused target image of the second pixel q .

[0073] As described in step S002, the texture information obtained at each position on the sub-projection image of the second pixel q is formed by superimposing the texture information of two positions on the target image. In other words, the texture information of the sub-projection image of the second pixel q is formed by superimposing the texture information of two sub-images on the target image. It is one of the two sub-images on the target image, and the other is called the second fused target image of the second pixel q , The method to obtain is:

[0074] First of all, it should be made clear that The pixel coordinates of After the affine transformation, we get The pixel coordinates of Pixels and The pixels of The gray value of the pixel above is obtained from the projected image. The grayscale value of is obtained from the target image.

[0075] The present invention The pixel coordinates on Reconstruct an image using the grayscale value of the pixel corresponding to the pixel coordinate , that is, image The pixel coordinates on The pixel coordinates on are the same, but the grayscale value is The gray value of the corresponding pixel on ; So far, we can know Is and The projector with the adjusted parameters will superimpose the target image. Projected onto the screen, the display result on the screen is , the other projector that has not adjusted the parameters will Projected on the screen, the result displayed on the screen is ;because and are projected into the same screen area, so the final result displayed on the screen in this area is and The superposition result of This is the working and imaging process of the projector. How to obtain:

[0076] First of all Preprocessing: Get The smallest gray value in The gray value of each pixel in is subtracted from the minimum gray value, and then the difference image is normalized to ensure The minimum grayscale value is 0.

[0077] Using corner detection algorithm to obtain All corner points of All corner points on the image are matched with all corner points on the target image to obtain all matched pairs of corner points. Then, all the corner point pairs are used as data samples to fit a homography matrix using the RANSC algorithm to obtain The homogeneous coordinates of all pixels on the target image are obtained by multiplying the homogeneous coordinates of all pixels with the homography matrix to obtain all new pixel coordinates. These pixel coordinates and the grayscale values ​​of these pixel coordinates on the target image brightness map constitute .

[0078] So far, the first fusion target image and the second fusion target image of the second pixel q are obtained, and similarly, the first fusion target images and the second fusion target images of all the second pixels are obtained.

[0079] Step S005: Obtain a projection offset feature vector according to the first fusion target image and the second fusion target image of each second pixel.

[0080] As described in step S004, for the second pixel p, obtain its first fusion target image and the second fusion target image , these two images are sub-images of the target image. Because the parameters of the projector are not adjusted properly, and It represents the texture information of different areas on the target image. The mean of the pixel coordinates of all pixels in , get The mean of the pixel coordinates of all pixels in , the present invention will and Combine into one vector , called the offset vector of the second pixel p, is used to represent the offset of the image to be projected by the projector relative to the image to be projected by another projector with set parameters.

[0081] Mean shift clustering is performed on the offset vectors of all second pixels to obtain all clustering results. Similar offset vectors in the same category indicate the same offset situation. Weighted summation is performed on the offset vectors in each category, and the result obtained is called the reference offset vector of each category; the weight of the weighted summation is the accuracy of the second pixel corresponding to each offset vector in each category; the reference offset vectors of all categories are merged into a high-dimensional vector, called a projection offset feature vector. The projection offset feature vector is used to indicate the offset situation of the projector when projecting at different positions.

[0082] Step S006, obtaining adjustment parameters of the projector according to the projection offset feature vector combined with the neural network.

[0083] Construct a fully connected neural network whose input is a P-dimensional vector and whose output is the parameter adjustment of the projector.

[0084] The method for obtaining the data set for training the network is as follows: a virtual projector is constructed in a computer simulator (such as an Unreal game engine), the projection results and adjustment parameters of the virtual projector can be controlled by a computer program without human intervention, the two virtual projectors are allowed to project completely overlapping images, and then the computer changes the parameters of one of the projectors so that the projection results become non-overlapping, the offset vectors of all non-overlapping positions are obtained, a projection offset feature vector is constructed, and the projection offset feature vector and the parameter adjustment amount of the virtual projector are used as data and labels; a large number of data sets can be quickly obtained by continuously changing the parameters of the virtual projector and the shape of the screen without human intervention.

[0085] The neural network is trained using the data set using a mean square error loss function and a stochastic gradient descent algorithm, so that the neural network converges.

[0086] The projection offset feature vector is input into the trained neural network to output the adjustment parameters of the projector. If the dimension of the projection offset feature vector is less than P dimensions, it is necessary to fill the projection offset feature vector with 0. If the dimension of the projection offset feature vector is greater than P dimensions, the redundant dimensions can be deleted without affecting the output results of the network.

[0087] Finally, the projector automatically adjusts parameters such as the posture viewing angle and focal length of the projector according to the output adjustment parameters, so that the images projected on the screen by the two projectors have the same and overlapping texture in the overlapping projection area.

[0088] Then, the edge fusion algorithm is used to make the brightness of the overlapping projection area consistent with the brightness of other areas, thereby obtaining a high-resolution projection image. At this point, the automatic adjustment of the projector is completed, so that a good projection fusion effect can be achieved in subsequent command and monitoring centers, network management centers, video conferences, academic reports, technical lectures, multi-functional conference rooms and other scenarios.

[0089] In order to improve the accuracy of the present invention, the present invention can be implemented multiple times in succession.

[0090] It should be noted that the sequence of the embodiments of the present invention described above is for description only and does not represent the advantages and disadvantages of the embodiments. The above describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0091] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0092] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A projection fusion method based on neural network, It is characterized in that The method comprises the following steps: Collect the illumination images projected by all projectors, and obtain the overlapping area of ​​the projector projections through the threshold segmentation algorithm; Each pixel on the projection image and outside the projection overlap area is regarded as each first pixel, and a sub-projection image consisting of all pixels in a preset window is obtained with each first pixel as the center, and a homography matrix of each first pixel is obtained according to the corner points of the sub-projection image of each first pixel and the corner points of the target image; Consider each pixel on the projected image and in the projection overlap area as each second pixel, obtain a preset number of first pixels having the smallest Euclidean distance to each second pixel, and obtain the homography matrix and accuracy of each second pixel according to the clustering results of the homography matrices of all the preset number of first pixels; Taking each second pixel as the center, a sub-projection image composed of all pixels in a preset window is obtained, and a first fused target image of each second pixel is obtained by using the homography matrix and the sub-projection image of each second pixel and combining with the target image; Obtaining a second fused target image for each second pixel according to the sub-projection image of each second pixel and its first fused target image; Combined with the accuracy of all second pixels, the projection offset feature vector of the projector is obtained according to the first fusion target image and the second fusion target image of all second pixels. The projection offset feature vector is input into the neural network, and the network outputs the adjustment parameters of the projector. The projector is adjusted according to the adjustment parameters and re-projects the image.

2. According to the projection fusion method based on neural network in claim 1, It is characterized in that The step of obtaining the projection offset feature vector comprises: Obtaining a first fused mean value of all pixel coordinates in the first fused target image of each second pixel, and a second fused mean value of all pixel coordinates in the second fused target image of each second pixel, obtaining a displacement vector formed by the first fused mean value to the second fused mean value, and merging the displacement vector and the first fused mean value into a vector, which is called an offset vector of each second pixel; The mean shift algorithm is used to cluster the offset vectors of all second pixels to obtain all categories, and a weighted sum is performed on the offset vectors in each category, and the result obtained is called the reference offset vector of each category; the weight of the weighted sum is the accuracy of the second pixel corresponding to each offset vector in each category; the reference offset vectors of all categories are merged into a high-dimensional vector, which is called a projection offset feature vector.

3. According to the projection fusion method based on neural network in claim 1, It is characterized in that The step of acquiring the homography matrix and accuracy of each second pixel includes: For a preset number of first pixels having the smallest Euclidean distance to each second pixel, obtaining a set of all homography matrices of the preset number of first pixels, flattening each homography matrix into a one-dimensional vector, clustering the homography matrices in the set to obtain all categories, obtaining a category containing the largest number of homography matrices among all categories, calculating a first mean of all homography matrices in the category, and using the first mean as the homography matrix of each second pixel; Obtaining a first pixel set corresponding to all homography matrices in the category, calculating a second mean of pixel coordinates in the first pixel set, calling the Euclidean distance between the coordinate of each second pixel and the second mean a distance factor of each second pixel, and then calculating an L2 norm of all homography matrices in the category and the first mean, and the variance of all the L2 norms is called a distribution factor of each second pixel; The reciprocal of the product of the distance factor of each second pixel and the distribution factor of each second pixel is used as the accuracy of each second pixel.

4. The projection fusion method based on a neural network according to claim 1, It is characterized in that The step of acquiring the first fusion target image of each second pixel includes: Obtaining homogeneous coordinates of all pixels on the projected sub-image of each second pixel, multiplying the homogeneous coordinates of all pixels by the homography matrix of each second pixel, and thereby obtaining all new pixel coordinates; Then, a brightness map of the target image is obtained, and the grayscale values ​​corresponding to the new pixel coordinates on the brightness map of the target image are obtained. The new pixel coordinates and the corresponding grayscale values ​​constitute a first fused target image for each second pixel.

5. The projection fusion method based on a neural network according to claim 1, It is characterized in that The step of acquiring the second fused target image of each second pixel comprises: Acquire the pixel coordinates of the sub-projection image of each second pixel and the grayscale value of the pixel corresponding to the pixel coordinates on the first fused target image of each second pixel, and construct a temporary image of each second pixel with the pixel coordinates and the grayscale value; Obtaining a difference between the sub-projection image of each second pixel and the temporary image of each second pixel, called a difference image, obtaining the minimum grayscale value in the difference image, subtracting the grayscale value of each pixel in the difference image from the minimum grayscale value, and then normalizing the difference image; Using a corner point detection algorithm to obtain all corner points of the difference image, using a corner point matching algorithm to match all corner points on the difference image with all corner points on the target image, obtaining all matched pairs of corner points, and then fitting a homography matrix using all the corner point pairs as data samples; Obtaining homogeneous coordinates of all pixels on the difference image, multiplying the homogeneous coordinates of all pixels by the homography matrix, and then obtaining all new pixel coordinates; The grayscale value corresponding to the new pixel coordinates on the target image brightness map is obtained, and the new pixel coordinates and the corresponding grayscale value constitute a second fused target image for each second pixel.

6. The projection fusion method based on a neural network according to claim 1, It is characterized in that The step of acquiring the sub-projection image of each first pixel and the sub-projection image of each second pixel comprises: A rectangular preset window is established with each first pixel as the center, and all pixels in the window and belonging to the projected image are obtained; if there are pixels belonging to the projection overlap region among all the pixels, the pixel values ​​of the pixels belonging to the projection overlap region are set to zero vectors; then the pixel coordinates and corresponding pixel values ​​of all the pixels on the projected image are obtained, and the brightness map of the image constructed by the coordinates and pixel values ​​of all the pixels is called a sub-projection image of each first pixel; A rectangular window is established with each second pixel as the center and a preset neighborhood range as the size, and all pixels within the window and belonging to the projected image are obtained. If there are pixels outside the projection overlap area among all the pixels, the pixel values ​​of the pixels outside the projection overlap area are set to a zero vector; then the pixel coordinates and corresponding pixel values ​​of all the pixels on the projected image are obtained, and the brightness map of the image constructed by all the pixel coordinates and pixel values ​​is called the sub-projection image of each second pixel.

7. The projection fusion method based on a neural network according to claim 1, It is characterized in that The step of acquiring the homography matrix of each first pixel comprises: A corner detection algorithm is used to obtain all corner points of the sub-projection image of each first pixel, a corner detection algorithm is used to obtain all corner points of the target image, a corner matching algorithm is used to match all corner points on the sub-projection image of each first pixel with all corner points on the target image to obtain all matched pairs of corner points, and then all corner point pairs are used as data samples to fit a homography matrix, which is the homography matrix of each first pixel.

8. The projection fusion method based on a neural network according to claim 1, It is characterized in that The step of obtaining the projection overlap area includes: All projectors project white light onto the screen, use a camera to capture the white light image on the screen, convert the image into a grayscale image, use the Otsu threshold segmentation algorithm to segment the grayscale image, obtain all connected domains, calculate the mean of the grayscale values ​​of all pixels in each connected domain, and obtain the connected domain with the largest mean as the projection overlap area; Among all the projectors, one is a projector whose parameters have been adjusted, and the others are projectors whose parameters have not been adjusted.

9. The projection fusion method based on a neural network according to claim 1, It is characterized in that The neural network is a fully connected neural network, the data set of the neural network is generated by a simulator, the loss function of the neural network is a mean square error loss function, and the neural network is trained using a stochastic gradient descent algorithm based on the loss function combined with the data set, so that the neural network converges.

10. The projection fusion method based on a neural network according to claim 1, It is characterized in that The target image refers to a high-resolution image that needs to be projected onto the screen by a projector, and the target image has texture information; the projected image refers to an image acquired by a camera after the target image is projected onto the screen by a projector, and the camera is an RGB camera installed on the projector.

Images