Projector automatic focusing method based on gradient calculation and convolutional neural network

Abstract

The invention discloses a projector automatic focusing method based on gradient calculation and a convolutional neural network, and the method comprises the steps: S100, enabling a projector to comprise a stepping motor, a camera, and a projection lens, and enabling the stepping motor to control the focal length of projection; the camera is used for shooting a picture projected on a wall surface or a curtain; the projection lens is used for projecting pictures or videos to a wall surface or a curtain; s200, using a projection lens to project a specific chessboard graph, using a camera to shoot a picture, extracting a Y channel component of the picture, using a gradient calculation mode to calculate the definition of the image, judging conditions and executing corresponding steps; and S300, constructing a convolutional neural network, performing training by using a chessboard graph shot by a camera, and performing last-step judgment by using the trained convolutional neural network for prediction of a high-definition picture obtained by gradient calculation.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a projector automatic focusing method based on gradient calculation and convolutional neural network. Background Art

[0002] A projector is a device that amplifies images or video signals and projects them onto a screen or wall. It is often used in many scenarios such as home, theater, education, and business presentations. The projector autofocus function is the most basic and practical function of a projector in daily use. Autofocus can improve the convenience of use and reduce the trouble of manual adjustment. After turning on and moving the projector, it can automatically adjust the picture to the clearest state.

[0003] The current autofocus solutions mainly include obtaining the current state of the projector's picture clarity through an external camera to control the movement of the motor to adjust the focal length of the projector to achieve autofocus, using an external camera to shoot a fixed calibration picture to determine the distance between the projector and the projection wall or screen to directly control the motor to adjust the focal length of the projector to achieve autofocus, or using an external sensor, such as a distance sensor, to directly obtain the distance between the projector and the wall or screen to adjust the focal length to achieve autofocus.

[0004] The camera-based solution is currently the most cost-effective and practical solution. In addition to the autofocus function, the camera-based solution can also realize functions such as automatic trapezoid, automatic obstacle avoidance and automatic screen entry. However, the cost of external sensors is higher and the versatility is lower. The current camera-based technical solution is based on projecting a specific picture, using the camera to obtain the image information to calculate the relative clarity of the current focal length or calculate the distance between the current camera and the image, so as to control the change of the projector's projection focal length.

[0005] Through certain instructions, the stepper motor can be controlled to change the focal length of the projector, thereby changing the size and clarity of the projected image. The camera-based solution can use the camera to capture the specific image information of the current projection to calculate the relative image clarity, and adjust the focal length of the projector according to the stepper motor, thereby affecting the size of the relative image clarity, so as to find the position with the highest clarity within the projectable range of the projector. The relative image clarity mentioned here instead of the absolute image clarity is because the image clarity calculated for the same image is different under different brightness environments, different distances, or different projection walls, so it can only be compared in a certain environment. The relative clarity within the projectable range of the projector can be selected to achieve the clearest focal length in that environment.

[0006] The camera solution can also cause great problems. For example, the brightness of the current environment will directly affect the exposure of the camera. The camera sets certain exposure parameters, and the pictures taken in different brightness environments are very different. For example, if the exposure parameters are set in a daytime environment to obtain pictures, the pictures taken in a completely dark environment at night may no longer be suitable. Similarly, different projection distances of the projector and the projection wall or screen of the projector will also result in different picture clarity values. The autofocus program has fixed the corresponding parameters and thresholds when setting. Under different conditions, the focal length of the maximum clarity position is also different, and the error is also large when the environmental difference is too large.

[0007] Deep learning can automatically learn features from data and can achieve high accuracy. It also has excellent generalization ability, can process unseen data, and has super adaptability. Using deep learning technology in the autofocus function of the projector has a huge advantage in accuracy over other technologies. It can achieve more accurate picture clarity calculations and maintain relatively stable picture clarity calculations in complex scenes such as brightness changes, inconsistent projection distances, or different surface textures. When using deep learning to calculate picture clarity, it often requires higher computing power than other ordinary models, and the chips on general projectors often do not have very good computing power, so it is unrealistic to continue using deep learning models to calculate picture clarity. Summary of the invention

[0008] In order to solve the above problems, the present invention provides a projector autofocus method based on gradient calculation and convolutional neural network. First, a relatively large area of ​​clarity is calculated by a simple gradient algorithm, and then a deep learning algorithm is used to calculate the picture clarity of the current picture, so as to determine whether to select the best picture clarity for the current projector. In this way, a wide range of fixed threshold parameters can be used to quickly find the area where the maximum clarity value is located, and then a deep learning method can be used to more accurately find the maximum clarity value in the environment, so as to determine the projection focal length of the projector. The method of directly using a fixed threshold value will be greatly affected by the environment in the step of accurately finding the maximum clarity value, so that the clearest position cannot be found. The present invention realizes the calculation of the clarity value of the picture by a simple gradient algorithm, accurately finds the area where the maximum clarity value of the projector is located in a certain environment, and then uses a deep learning algorithm to search for the focal length with the maximum clarity, and uses the method of gradient calculation and deep learning combination to realize a projector autofocus algorithm that combines speed and accuracy. The entire algorithm implementation flow chart can be seen Figure 5 shown.

[0009] In order to achieve the above-mentioned purpose, the present invention provides the following technical solutions:

[0010] S100: The projector includes a stepper motor, a camera and a projection lens, wherein the stepper motor is used to control the focal length of the projection; the camera is used to shoot a picture projected on a wall or a screen; and the projection lens is used to project a picture or a video onto the wall or the screen;

[0011] S200: Project a specific chessboard image using a projection lens, shoot the image using a camera, extract the Y channel component of the image, calculate the clarity of the image using a gradient calculation method, determine conditions and execute corresponding steps;

[0012] S300: Construct a convolutional neural network, use the chessboard image taken by the camera for training, and use the trained convolutional neural network to predict the high-definition image obtained by gradient calculation for the final judgment;

[0013] In step S100, there are the following instructions:

[0014] S101: The image captured by the camera installed on the projector is generally a grayscale image;

[0015] S102: The screen size ratio of the projector is generally 16:9; the projection distance of the projector is generally 1-4 meters

[0016] S103: a stepper motor of the projector, used to control the extension and retraction of the lens of the projector, thereby controlling the change of the focal length of the projector;

[0017] S104: The schematic diagram of the projector lens, stepper motor and camera on the projector can be seen Figure 1 , the method of the present invention can be implemented on both methods;

[0018] S105: The stepper motor of the projector has a limit switch, that is, whether the stepper motor controls the lens to extend or retract to the most extreme position can be read through the program;

[0019] S106: The specific chessboard image mentioned in step S200 is projected by a projector. Figure 2 shown.

[0020] In step S100, the relative definition of the current image is calculated as follows:

[0021] S107: Initialization tol_sum=0;

[0022] The final Tol_sum refers to the clarity of the image;

[0023] S108: Using the window traversal method, for each window position (ii, jj), traverse the height and width of the image with steps of size_w and size_1. ii∈[0,h,size_w] jj∈[0,w,size_l]

[0024] Where h is the height of the image, w is the width of the image, size_w is the height of the window, and size_l is the width of the window.

[0025] Here (ii, jj) refers to the position in the Y channel component of the original image;

[0026] S109: For each window (i, j), the absolute difference between the current pixel and the pixels to its right and below is calculated, that is, the horizontal and vertical gradient differences are calculated. The distance here in the present invention is L pixels, that is, the gradient difference is directly calculated for the adjacent L+1 positions. current=frameByte[ii+i][jj+j] next_hor=frameByte[ii+i][jj+j+L] next_ver=frameByte[ii+i+L][jj+j] tmp1_tab[k+j]=|current-next_hor| tmp2_tab[k+j]=|current-next_ver|

[0027] Where frameByte[i][j]: the camera obtains the pixel value of the i-th row and j-th column of a grid in the image, both of which refer to the Y channel component;

[0028] Where k is the variable used for indexing;

[0029] The obtained tmp1_tab and tmp2_tab are the horizontal and vertical gradient values ​​in the image;

[0030] Here (i, j) refers to the position in the size_w*size_l grid we divided;

[0031] S110: Perform the following operations on each tmp1_tab and tmp2_tab: if(tmp>P&&k1 <K)→(tmp_tab[k1]=tmmp,k1+=1)

[0032] Where tol_grad refers to the total gradient of the grid, including longitudinal and transverse gradients, P refers to the gradient threshold, k1 is the current array subscript, and K refers to the array length;

[0033] S111: tol_sum += tol_grad

[0034] When the window sliding calculation of the entire image is completed, the final accumulated tol_sum is the gradient clarity of the current image;

[0035] The structure diagram of the image gradient clarity method can be seen Figure 3 .

[0036] The conditions in step S200 are the following:

[0037] S20 1: Condition a: SharPness_Delta>SharPness_limit;

[0038] Where SharPness_Delta is the difference between the image clarity of the previous image and the maximum image clarity, i.e., MAXSharPness-SharPness cur, and SharPness limit is the clarity difference threshold, which is used to control the direction judgment of the motor;

[0039] Condition b: The motor is retracted to the limit, i.e., motor_limit = 1, and cannot be retracted or extended any further.

[0040] Condition c: SharPness_Delta <limit_thr;

[0041] Wherein limit_thr is the maximum sharpness difference threshold, which means that when limit_thr>Sharpness Delta in step S300, a convolutional neural network is used to detect the picture sharpness.

[0042] The steps in step S200 are as follows:

[0043] S202: Step A: Set MAX_SharPness=SharPness_pre=0; if the current image SharPness_cur>MAX_SharPness, MAX_SharPness=SharPness_cur;

[0044] The maximum image sharpness is MAX_SharPness, the current image sharpness is SharPness_cur, and the previous image sharpness is SharPness_pre.

[0045] Step B: When SharPness_Delta < limit_thr, use a convolutional neural network to predict the current image to determine whether it is clear, and the result obtained is whether the image is clear.

[0046] The structural diagram of the convolutional neural network of the present invention can be seen Figure 4 as shown.

[0047] According to the automatic focusing method for a projector based on gradient calculation and convolutional neural network described in claim 1, it is characterized in that the convolutional neural network in step S300 is as follows:

[0048] Since it is used on a projector, the projector is basically configured with an android system, the computing power of the chips varies, and the dimension of the image data is not large. Therefore, the CNN model of the convolutional neural network of the present invention is constructed by 3 convolutional layers, 3 max pooling layers, 1 flattening layer, and 2 fully connected layers;

[0049] Among them, the convolutional layer uses a convolutional kernel of size 3*3, and one layer uses 32 convolutional kernels. The pooling layer window can use a size of 2*2;

[0050] Due to the classification function of the convolutional neural network used in the present invention, the softmax activation function is used in the output layer to output the category: whether it is clear.

[0051] Input layer: The input shape is (480, 640, 1) (the Y-channel luminance component of the present invention, and the image size is 480*640);

[0052] Convolutional layer: Use 3 convolutional layers to extract features and use the ReLU activation function;

[0053] The mathematical description of the convolutional layer can be seen as follows Y is the feature output after convolution, W is the convolutional kernel, and W(m, n) refers to the weight of the m-th row and n-th column in the convolutional kernel; while X is the feature input, that is, the Y-channel component of the image mentioned above, k is the convolutional kernel size, and i and j are the coordinates of the feature output;

[0054] Pooling layer: Add a pooling layer (MaxPooling) after the convolutional layer to reduce dimensionality calculation, improve efficiency, and reduce consumption.

[0055] Flattening layer: Flatten the multi-dimensional feature map into a one-dimensional array.

[0056] Fully connected layer: Add a fully connected layer.

[0057] Output layer: In fact, it is also a fully connected layer in essence. The present invention uses the softmax activation function to output the category probability

[0058] After building the model mentioned in step S302, and before using the convolutional neural network to predict the clarity in step S302, the model needs to be trained, including the following steps:

[0059] S303: First, a projector is used to project a specific chessboard image onto a wall or a screen, wherein the specific chessboard image is visible. Figure 2 As shown;

[0060] S304: using a projector to project a chessboard image in different environments, controlling a stepper motor to rotate and change the focal length of the projector, and using an external camera of the projector to take pictures in different environments, at different distances, with different focal lengths, and on different screen walls;

[0061] S305: each of the captured chessboard images needs to be manually labeled with a category: clear / unclear;

[0062] S306: extracting the Y channel component of the image and the corresponding label calibration as the final data set;

[0063] S307: Divide the data set into a training set, a validation set and a test set, usually with a ratio of 80% training, 10% validation and 10% test;

[0064] S308: Use the convolutional neural network constructed in step S302 to train and predict the data, and use appropriate training strategies to improve the accuracy of the model to a certain level, such as 95% or 98%, before use.

[0065] A projector autofocus method based on gradient calculation and convolutional neural network is characterized in that a faster gradient calculation method is used to calculate the Y channel component of a specific chessboard image as image clarity calculation, steps S100 and S200 are used to quickly find the area where the maximum image clarity may exist, and in step S300, a more accurate convolutional neural network is used to accurately predict whether the current chessboard image is clear, thereby more accurately finding the optimal focal length of the projector in the environment.

[0066] By executing all the above steps, the maximum clarity focal length of the projector in the current environment can be found, thereby realizing a projector automatic focusing method that combines speed and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] For clearer reading, understanding and implementation, the present invention is described in detail below with reference to the accompanying drawings;

[0068] Figure 1This is an example top view of the camera and lens motor module of the projector in the embodiment of the present invention being placed in a horizontal position and projected;

[0069] Figure 2 This is a plan view showing an example of a camera and a lens motor module of a projector in a horizontal position in an embodiment of the present invention;

[0070] Figure 3 This is an example top view of the camera and lens motor module of the projector in the embodiment of the present invention, which are placed in a vertical position and projected;

[0071] Figure 4 This is a plan view showing an example of a camera and a lens motor module of a projector in a vertical position in an embodiment of the present invention;

[0072] Figure 1-Figure 4 The two relative positions in can use the projector automatic focusing method based on gradient calculation and convolutional neural network in the present invention;

[0073] Figure 5 It is an example diagram of a specific chessboard diagram projected by a projector mentioned in an example of the present invention;

[0074] The chessboard image has obvious edges and contrast. The size of the gradient value reflects the sharpness of the edge. A larger gradient value usually means that the image is clearer and the edges are more obvious; a smaller gradient value may indicate blur or lack of details.

[0075] Figure 6 is a schematic diagram of a method for obtaining image clarity based on gradient calculation in an example of the present invention;

[0076] Figure 7 It is a schematic diagram of the structure of the convolutional neural network constructed in the example of the present invention;

[0077] Figure 8 It is a flow chart of a projector auto-focusing method based on gradient calculation and convolutional neural network in an example of the present invention.

[0078] Fig. 9 This is a diagram showing some examples of image data collected in an example of the present invention, where clear images are marked with a label: 1, and unclear images are marked with a label: 0;

[0079] Among them, 1-projection wall or screen; 2-camera; 3-motor and lens; DETAILED DESCRIPTION

[0080] The present invention will be further described below with reference to the exemplary figures.

[0081] First, the projector used in this example is equipped with a camera and a lens motor module. The camera is used to capture the image projected by the projector, and the lens motor module is a stepper motor that controls the extension and retraction of the lens, thereby changing the focal length of the projector to achieve the effect of changing the clarity of the projected image. The relative installation of the camera and the lens motor module can be seen below. Figure 1 As shown;

[0082] Step S101: In this example, the resolution of the camera is 480*640;

[0083] Step S102: In this example, the projection screen size is 1280*720, the specific ratio is 16:9, and the projection distance of the projector is between 1.5 meters and 4.0 meters;

[0084] Step S104: the projector in this example is equipped with a motor, and the rotation of the motor can be controlled to extend or retract the lens;

[0085] The communication protocol used by the camera in this example is the YUV protocol, specifically YUV420;

[0086] Wherein Y refers to the brightness component, U refers to the chrominance component, and V refers to the chrominance component. The chessboard image we project is a black and white image, so we only need to extract the Y channel component, and the other channel components are not very meaningful in this example and the present invention;

[0087] Step S105: the motor in this example has a limit position, that is, whether the motor is extended or retracted to the lens can be read by the program, that is, motor_limit=1 or motor_limit=0;

[0088] Step 106: In this example, the photo projected by the projector is a chessboard pattern. Since the chessboard pattern has obvious edges and contrast, it can be seen in the example that Figure 2

[0089] Step S100: The motor is started, the camera is turned on, and the projector projects a specific chessboard photo onto a wall or a screen. The stepper motor is controlled to retract the lens, and the external camera is used to continuously obtain the current image information. The Y channel component is used to calculate the relative clarity of the current image, and step A is executed;

[0090] In this example, several operations are involved:

[0091] 1. Control the motor to rotate, retract the lens, and reduce the focal length;

[0092] 2. Control the projector to project a specific chessboard image;

[0093] 3. Use the camera to continuously capture the chessboard image projected on the wall or screen;

[0094] 4. Extract the Y channel component of the image and calculate the gradient to obtain the image clarity of the current projector at the focal length;

[0095] 5. Use the obtained picture clarity to execute step A;

[0096] The clarity of the current image in this example can be seen in the following steps:

[0097] Step S107: Initialization tol_sum = 0;

[0098] Step S108: Using the window traversal method, for each window position (ii), traverse the image in steps of size_w and size_l within the height and width range of the image. ii∈[0,h,size_w] jj∈[0,w,size_l]

[0099] Where h is the height of the image, w is the width of the image, size_w is the height of the window, and size_l is the width of the window. Here (ii, jj) refers to the position in the Y channel component of the original 480*640 image;

[0100] In this example, h is 640, w is 480, size_w is 80, and size_l is 80, that is, a 480*640 image is divided into 48 80*80 grids;

[0101] Step S109: For each window (i, j) obtained by the above segmentation, the absolute difference between the current pixel and the pixels on its right and below is calculated, that is, the horizontal and vertical gradient differences are calculated. The distance between the current pixel and the pixels on its right and below is 1 pixel in this example and the present invention, that is, the gradient difference is directly calculated between two adjacent positions. current=frameByte[ii+i][jj+j] next_hor=frameByte[ii+i][jj+j+1] next_ver=frameByte[ii+i+1][jj+j] tmp1_tab[k+j]=|current-next_hor| tmp2_tab[k+j]=|current-next_ver|

[0102] Where frameByte[i][j]: the pixel value of the i-th row and j-th column of a grid, both refer to the Y channel component; k is the variable used for indexing; the obtained tmp1_tab and tmp2_tab are the horizontal and vertical gradient values ​​in the image;

[0103] Step S110: Perform the following operations on tmp1_tab and tmp2_tab in each grid: if(tmp>P&&k1 <K)→(tmp_tab[k1]=tmmp,k1+=1)

[0104] Where tol_grad refers to the total gradient of the grid, including the longitudinal and transverse gradients. In this example, P is 20, K is 3000, and k1 is initially 0.

[0105] Step S111: tol_sum += tol_grad

[0106] In this example, all calculated gradients are accumulated to obtain the final tol_sum;

[0107] When the window sliding calculation of the entire image is completed, the final tol_sum is the gradient clarity SharPness_cur of the current image;

[0108] The structure diagram of the image gradient clarity method can be seen Figure 3 .

[0109] After obtaining the image clarity SharPness_cur of the current image in this example, the operation of step A needs to be performed

[0110] Step A in step S200 is the following steps:

[0111] Step S202: Set MAX_SharPness=SharPness_pre=0; if the current image SharPness_cur>MAX_SharPness, MAX_SharPness=SharPness_cur;

[0112] The maximum image sharpness is MAX_SharPness, the current image sharpness is SharPness_cur, and the previous image sharpness is SharPness_pre.

[0113] In this example, the operation of step A in step S200 is mainly to find the maximum image clarity value that occurs during the process of retracting the projector, so as to control the direction of the motor in step S200. Because the starting position of the motor is different, if the motor needs to rotate to the limit position every time, it cannot meet the daily needs in terms of time efficiency;

[0114] Step S200: When one or more of the two conditions a / b are met, the motor will be extended, and the external camera will be used to continuously obtain the current image information, calculate the relative clarity of the current image, and execute step A;

[0115] Condition a / b in this example is step S201:

[0116] Condition a: SharPness_Delta>30000, used to control the motor direction;

[0117] Condition b: The motor is retracted to the limit, i.e., motor_limit = 1, and cannot be retracted or extended any further.

[0118] In this example, the clarity value obtained during the motor rotation process is regular. When the motor rotates to a certain position, assuming that the maximum clarity value is found, then the clarity value obtained at this time will be smaller than the current clarity value regardless of whether the motor rotates inward or outward. The greater the distance of continuous rotation, the smaller the clarity value will be.

[0119] Therefore, when searching for clarity values, the difference between the maximum clarity value and the current clarity value can be used to determine whether to continue rotating in one direction, because if a certain threshold is exceeded, the area obtained by continuing to rotate will not be the area where the maximum clarity value may appear;

[0120] In this example, it is assumed that condition a is met, that is, the difference between the maximum clarity value and the current clarity value is greater than 30000. Even if the motor has not rotated to the limit position, it cannot continue to rotate in this direction because it is meaningless and the motor needs to be extended;

[0121] In this example, it is assumed that condition b is met, which means that the motor has reached its limit and cannot rotate any further. It is necessary to reverse and extend the motor.

[0122] In this example, if one or more of the conditions a / b are met, the motor needs to be controlled to be extended, and the operation of step S200 is continued;

[0123] Step S300: When one or more of the two conditions a / b are met again, the motor will be retracted, and the external camera will be used to continuously obtain image information. The picture clarity is obtained through the above-mentioned clarity calculation steps. If condition c is met, step B is used, and the obtained Y_SP=1; the motor stops rotating, the camera stops shooting, and the focusing ends; if condition a / b is met, a control command is used to rotate the motor in another direction, that is, extend or retract, until condition c is met, and finally the motor stops, the camera is turned off, and the focusing ends.

[0124] In this example, when one or more of the conditions a / b are met again in step S300, it means that after step S200 is executed, the lens is extended, and the motor limit is encountered, which meets condition b, or the difference between the current definition value and the maximum definition value that has been searched is greater than our 30000 threshold, which meets condition a, then the motor is retracted again in step S300 directly. At this point, we can accurately find the area where the maximum definition value may appear;

[0125] Condition c in step S300 is:

[0126] Step S301: Condition c: SharPness_Delta<10000;

[0127] In step S300, when SharPness_Delta<10000, the trained convolutional neural network is used to predict the image to determine whether the current image is clear;

[0128] Step B in step S300 is:

[0129] Use the trained convolutional neural network (CNN) to predict the current image and determine whether it is clear. The result is Y_SP=0 / 1; where Y_SP refers to whether the image is clear.

[0130] The convolutional neural network structure diagram of this example can be seen Figure 4 shown.

[0131] Input layer: The Y channel component size of the image in this example is 480*640, so the input layer size is set to 480*640;

[0132] Convolutional layer: In this example, three convolutional layers are used to extract features;

[0133] The mathematical description of the convolutional layer can be seen as follows Y is the feature output after convolution, W is the convolution kernel, W(m,n) refers to the weight of the m rows and n columns in the convolution kernel; X is the feature input, that is, the Y channel component of the image mentioned above, k is the convolution kernel size, and i and j are the coordinates of the feature output;

[0134] The convolution kernel size selected in this example is 3*3. The first convolution layer uses 32 convolution kernels, the second convolution layer uses 64 convolution kernels, and the third convolution layer uses 128 convolution kernels. The activation function used in the convolution kernel layer is the ReLU activation function: .

[0136] Pooling layer: Add a pooling layer (MaxPooling) after the convolution layer to reduce dimensional calculations, improve efficiency, and reduce consumption. It is worth noting that a pooling layer needs to be added after each convolution layer;

[0137] The window size of the pooling layer is set to 2*2, which reduces the size of the feature map by half, retains the main features and reduces the amount of calculation.

[0138] The pooling layer is also divided into average pooling layer and maximum pooling layer, which extracts the maximum value or average value in the window;

[0139] The pooling layer in this example selects the maximum pooling layer, which retains the most prominent features of each grid in the chessboard image and can well preserve the texture features;

[0140] The mathematical expression of the maximum pooling layer can be seen as follows: P(i,j)=maxX(i+m,j+n)

[0141] Where P is the pooled feature and X is the input feature

[0142] Flattening layer: Flattens the multidimensional feature map into a one-dimensional array for input into the fully connected layer. The expression can be seen as follows: Z = flatten(Y)

[0143] Fully connected layer: Add a fully connected layer, the expression can be seen as follows: A=f(WZ+b)

[0144] Where W is the weight matrix, b is the bias, and f is the activation function. The activation function used in the fully connected layer of this example is still the ReLU activation function;

[0145] Output layer: In fact, it is also a fully connected layer. In this example, the output layer uses the softmax activation function to output the category probability, which can be expressed as follows:

[0146] Where e is the base of natural logarithm, approximately 2.71828, z i is the i-th element in the input vector, K is the number of categories, 2 in this example, σ(z i ) refers to the i-th element of the softmax function output, which represents the probability of category i;

[0147] Before step B in step S300 uses the convolutional neural network for prediction, the following steps are required:

[0148] Step S303: First, a projector is used to project a specific chessboard image onto a white wall, where the specific chessboard image is visible. Figure 2 As shown;

[0149] Step S304: Projecting using a projector in different environments Figure 2 The chessboard diagram controls the rotation of the stepper motor to change the focal length of the projector, and uses the external camera of the projector to take pictures in different environments, at different distances, with different focal lengths and different white wall surfaces;

[0150] Step S305: each chessboard image captured needs to be labeled with a category: clear / unclear;

[0151] In this example, steps S303, 304 and 305 can be seen in Example Figure 6 As shown;

[0152] Step S306: extracting the Y channel component of the image and the corresponding data label as the final data set;

[0153] Step S307: Divide the data set into a training set, a validation set and a test set. In this example, the ratio is 80% training, 10% validation and 10% test.

[0154] Among them, it is worth noting that the number of training data categories should be equal, that is, the number of clear categories is approximately equal to the number of unclear categories;

[0155] An imbalance in the number of categories can easily cause the model to tend to predict the majority category, distort the evaluation of model performance, and cause unstable training.

[0156] In this example, 1000 pieces of data of clear categories and 1000 pieces of data of unclear categories are finally selected as the data set, with a total of 2000 pieces of data. According to the division in step S307, the training set is 1600 pieces of data, the validation set is 200 pieces of data, and the test set is 200 pieces of data;

[0157] In deep learning model training, the more data the better. The fact that 2000 pieces of data are selected as the data set in this example does not mean that the present invention also uses only 2000 pieces of data as the data set.

[0158] Step S308: Use the convolutional neural network constructed in step S302 to train and predict the data, use appropriate training strategies to improve the accuracy of the model, and stop training and put it into use when the accuracy is >95%.

[0159] Through the above implementation, the projector auto-focus method based on gradient calculation and convolutional neural network can be realized;

[0160] The camera described in this example is not limited to a size of 480*640, and can also be implemented with cameras of other resolution sizes;

[0161] The convolutional neural network structure described in this example is not limited to Figure 4 The structure of the network can be modified according to the dimension and characteristics of the data to improve the effect of the model;

[0162] The present invention is not limited to the above-described embodiments, which are only one of several specific implementations of the present invention. Any deformation and modification based on the method of the application scope of the present invention are deemed to be within the application scope of the present invention. We intend to protect all innovative implementations that are consistent with the spirit of the present invention to ensure its wide applicability and effectiveness.

Claims

1. A projector auto-focus method based on gradient calculation and convolutional neural network, characterized in that: The method comprises: S100: The projector includes a stepper motor, a camera and a projection lens, wherein the stepper motor is used to control the focal length of the projection; the camera is used to shoot a picture projected on a wall or a screen; and the projection lens is used to project a picture or a video onto the wall or the screen; S200: Project a specific chessboard image using a projection lens, shoot the image using a camera, extract the Y channel component of the image, calculate the clarity of the image using a gradient calculation method, determine conditions and execute corresponding steps; S300: Construct a convolutional neural network, use the chessboard image taken by the camera for training, and use the trained convolutional neural network to predict the high-definition image obtained by gradient calculation for the final judgment.

2. The projector auto-focusing method based on gradient calculation and convolutional neural network according to claim 1, characterized in that: In step S100, there are the following instructions: S101: The image captured by the camera installed on the projector is generally a grayscale image; S102: The screen size ratio of the projector is generally 16:9; the projection distance of the projector is generally 1-4 meters; S103: a stepper motor of the projector, used to control the extension and retraction of the lens of the projector, thereby controlling the change of the focal length of the projector; S104: installing a lens, a stepping motor and a camera of the projector on the projector, and the method can be implemented in both methods; S105: The stepper motor of the projector has a limit switch, that is, whether the stepper motor controls the lens to extend or retract to the most extreme position can be read through the program; S106: Projecting the specific chessboard pattern using a projector as mentioned in the above step S200.

3. The projector auto-focusing method based on gradient calculation and convolutional neural network according to claim 1, characterized in that: In step S100, the definition of the current image calculated using the gradient is as follows: S107: Initialization Where tol_sum refers to the clarity of the image; S108: Using the window traversal method, for each window position (ii, jj), traverse the height and width of the image with steps of size_w and size_l. Where h: image height, w: image width, size_w: window height, size_l: window width; Here (ii, jj) refers to the position in the Y channel component of the original image; S109: For each window (i, j), the absolute difference between the current pixel and the pixels to its right and below is calculated, that is, the horizontal and vertical gradient differences are calculated, and the interval is L pixels, that is, the gradient difference is directly calculated for the adjacent L+1 positions; Where frameByte[i][j]: the camera obtains the pixel value of the i-th row and j-th column of a grid in the image, both of which refer to the Y channel component; k is a variable used for indexing; the obtained tmp1_tab and tmp2_tab are the horizontal and vertical gradient values ​​in the image; and here (i, j) refers to the position in the size_w*size_l grid we divided; S110: Perform the following operations on each tmp1_tab and tmp2_tab: Where tol_grad refers to the total gradient of the grid, including vertical and horizontal gradients, P refers to the gradient threshold, k1 is the current array subscript, and K refers to the array length; S111： When the window sliding calculation of the entire image is completed, the finally accumulated tol_sum is the gradient sharpness of the current image. The structure diagram of the image gradient sharpness method can be seen in Figure 3.

4. The projector auto-focusing method based on gradient calculation and convolutional neural network according to claim 1, characterized in that: The condition in step S200 is the following condition: S201： Condition a: SharPness_Delta > SharPness_limit; where SharPness_Delta is the difference between the image sharpness of the previous image and the maximum image sharpness, that is, MAX_SharPness - SharPness_cur, and SharPness_limit is the threshold of the sharpness difference, which is used to control the direction judgment of the motor; Condition b: The motor is retracted to the limit, that is, motor_limit = 1, and it cannot be retracted / extended anymore; Condition c: SharPness_Delta < limit_thr; where limit_thr is the maximum sharpness difference threshold, which means that when limit_thr > SharPness_Delta in step S300, other operations are performed, such as using a deep learning network to detect the image sharpness or directly stopping the motor rotation.

5. The projector auto-focusing method based on gradient calculation and convolutional neural network according to claim 1, characterized in that: The steps in step S200 are the following steps: S202： Step A: Set MAX_SharPness = SharPness_pre = 0; if the current image SharPness_cur > MAX_SharPness, then MAX_SharPness = SharPness_cur; where the maximum image sharpness: MAX_SharPness, the current image sharpness: SharPness_cur, and the previous image sharpness: SharPness_pre; Step B: When SharPness_Delta < limit_thr, use a convolutional neural network to predict the current image to determine whether it is clear, and the result obtained is whether the image is clear; the structure diagram of the convolutional neural network of the present invention can be seen in Figure 4.

6. The projector auto-focusing method based on gradient calculation and convolutional neural network according to claim 1, characterized in that: The convolutional neural network in step S300 is as follows: S301: Since it is used on a projector, the projector is basically configured with an android system, the computing power of the chip varies, and the dimension of the image data is not large. Therefore, the convolutional neural network CNN model of the present invention is constructed by 3 convolutional layers, 3 max pooling layers, 1 flattening layer, and 2 fully connected layers. Among them, the convolutional layer uses a 3*3 convolutional kernel, one layer uses 32 convolutional kernels, and the pooling layer window uses a 2*2 size. Since the classification function of the convolutional neural network used in the present invention is used, the softmax activation function is used in the output layer to output the category: whether it is clear; Input layer: The input shape is (480, 640, 1); Convolutional layer: Use 3 convolutional layers to extract features and use the ReLU activation function; The mathematical description of the convolutional layer can be seen as follows Y is the feature output after convolution, W is the convolution kernel, W(m,n) refers to the weight of the m rows and n columns in the convolution kernel; X is the feature input, that is, the Y channel component of the image mentioned above, k is the convolution kernel size, i and j are the coordinates of the feature output; Pooling layer: Add a pooling layer (MaxPooling) after the convolution layer to reduce dimensional calculations, improve efficiency, and reduce consumption; Flattening layer: flattens the multidimensional feature map into a one-dimensional array; Fully connected layer: Add a fully connected layer; Output layer: In fact, it is also a fully connected layer. The present invention uses a softmax activation function to output category probabilities.

7. The projector auto-focusing method based on gradient calculation and convolutional neural network according to claim 6, characterized in that: After building the model mentioned in step S301, the model needs to be trained, including the following steps: S302: First, a projector is used to project a specific chessboard image onto a wall or a screen, wherein the specific chessboard image is shown in FIG2 ; S303: using a projector to project a chessboard image in different environments, controlling a stepper motor to rotate and change the focal length of the projector, and using an external camera of the projector to take pictures in different environments, at different distances, with different focal lengths, and on different screen walls; S304: manually label the captured chessboard image with a category label: clear / unclear; S305: extracting the Y channel component of the image and the corresponding label as the final data set; S306: Divide the data set into a training set, a validation set, and a test set, usually with a ratio of 80% training, 10% validation, and 10% test; S307: Use the convolutional neural network constructed in step S301 to train and predict the data, and use appropriate training strategies to improve the accuracy of the model to a certain level, such as 95% or 98%, before use.

8. A projector auto-focus method based on gradient calculation and convolutional neural network, characterized in that Using a faster gradient calculation method to calculate the Y channel component of a specific chessboard image as image clarity calculation, using the device of step S100 and the gradient calculation method of step S200 to quickly find the area where the maximum image clarity may exist, and using a more accurate convolutional neural network in step S300 to accurately predict whether the current chessboard image is clear, so as to more accurately find the optimal focal length of the projector in the environment; By executing steps S100, S200, and S300, the maximum clarity focal length of the projector in the current environment can be found, thereby realizing the automatic focus function of the projector.

Images