Projection method and device

Abstract

The invention provides a projection method and device. The projection method comprises the steps that a first predistortion correction parameter is acquired, and the first predistortion correction parameter is acquired in advance based on a first prediction attitude of projection equipment; performing predistortion correction on a current image according to the first predistortion correction parameter to obtain a first target image; and projecting the first target image on a target plane. According to the invention, the requirement of high frame rate realization can be met, and the projection picture is always kept square.

Description

Technical Field

[0001] The present application relates to projection technology, and in particular to a projection method and device. Background Art

[0002] A common situation in projection scenes is that the image projected by the projection device on the screen or wall is not the expected rectangle, but a trapezoid, which creates a need to correct this trapezoidal distortion. A method for digitally performing trapezoidal correction can pre-distort the original input image and then project the transformed image to project the desired rectangular image.

[0003] In the related technology, the deflection angle of the projection optical axis in various directions is detected by sensors to obtain the degree of image distortion, calculate the pre-distortion correction parameters, and perform pre-distortion transformation on the input image based on the pre-distortion correction parameters before outputting it. This method can perform trapezoidal correction in real time during the posture change of the projection device, so that the image always remains rectangular.

[0004] However, the above method is difficult to meet the requirements of high frame rate implementation, the projected image may not always remain square, and even the user may still be able to perceive keystone distortion. Summary of the invention

[0005] The present application provides a projection method and device to meet the requirements of high frame rate implementation so that the projected image remains square at all times.

[0006] In a first aspect, the present application provides a projection method, comprising: obtaining a first pre-distortion correction parameter, wherein the first pre-distortion correction parameter is obtained in advance based on a first predicted posture of a projection device; performing pre-distortion correction on a current image according to the first pre-distortion correction parameter to obtain a first target image; and projecting the first target image onto a target plane.

[0007] In the embodiment of the present application, by adopting a prediction method to obtain pre-distortion correction parameters, the pre-distortion correction parameters can be made closer to the posture of the projection device during distortion correction, so as to meet the requirements of high frame rate implementation and keep the projected image square at all times.

[0008] The distortion correction parameters can be used to perform reverse keystone distortion on the input current image, so that the keystone distortion during the projection process can adjust the distorted image back to a normal rectangular picture.

[0009] The pre-distortion correction parameters can be obtained by the following two methods:

[0010] One method adopts actual measurement method, whereby the projection device uses sensors to detect the deflection angles of the projection optical axis in various directions, such as pitch angle α, roll angle β, yaw angle θ, etc., to obtain the posture of the projection device, and obtain the degree of distortion of the picture according to the posture of the projection device, thereby calculating the pre-distortion correction parameters.

[0011] In this case, the pre-distortion correction parameters are obtained based on the actual measured data of the sensor. Figure 3 The embodiments shown will not be described in detail here.

[0012] Another method uses a prediction method. The projection device predicts the posture of the projection device based on historical data measured by the sensor or the historical posture of the projection device (the historical posture can be obtained based on historical data), obtains the degree of distortion of the picture based on the posture of the projection device, and thus calculates the pre-distortion correction parameters.

[0013] In this case, the pre-distortion correction parameters are predicted based on historical data / historical postures.

[0014] Optionally, the projection device can adopt a machine learning model, input historical data / historical posture, output the posture of the projection device, and then obtain the degree of distortion of the picture according to the posture of the projection device, so as to calculate the pre-distortion correction parameters.

[0015] The technology of the above machine learning model can be referred to the above description and will not be repeated here.

[0016] Optionally, the projection device may adopt a traditional solution, such as a particle filter, to obtain the posture of the projection device, and then obtain the degree of distortion of the picture according to the posture of the projection device, so as to calculate the pre-distortion correction parameters.

[0017] Particle filtering methods can include:

[0018] (1) Initialization phase: Read sensor data at time t and obtain the posture at time t. Based on this, initialize some randomly distributed particles;

[0019] (2) State prediction stage: The velocity and direction of movement calculated from the sensor data are assigned to the above particles, and the posture of the particles at the future time t+1 is predicted. Optionally, the average posture of all particles can be directly used as the posture at time t+1;

[0020] At this point, the predicted posture at the future time t+1 can be obtained. The next two steps are to update the particle weights to make subsequent predictions more accurate.

[0021] (3) Weight update stage: particle weights can be updated based on sensor data at time t+1;

[0022] (4) Resampling stage: New particles are extracted according to the updated particle weights. After resampling, particles with higher weights have a greater probability of being retained, and particles with smaller probabilities have a greater probability of being eliminated;

[0023] (5) Enter the next state prediction stage.

[0024] It should be noted that, in addition to the above two optional methods, the projection device may also use other methods to predict the pre-distortion correction parameters, which is not specifically limited in the embodiments of the present application.

[0025] In an embodiment of the present application, the projection device can execute two processing logics, wherein processing logic 1 periodically obtains pre-distortion correction parameters, and processing logic 2 performs pre-distortion correction on the image to be projected according to the pre-distortion correction parameters. The pre-distortion correction parameters required by processing logic 2 come from processing logic 1, and the closer the acquisition time of the pre-distortion correction parameters obtained by processing logic 2 is to the execution time of processing logic 2, the more the accuracy of the pre-distortion correction can be improved, that is, the shorter the effective delay (the time interval from its acquisition to its use) of the pre-distortion correction parameters obtained by processing logic 2, the more the accuracy of the pre-distortion correction can be improved. This is because the posture of the moving projection device is changing, and in order to ensure that the picture is always "square" (i.e., a rectangular picture) in the eyes of the user, the pre-distortion correction parameters obtained based on the posture are also changing. For example, the posture of the projection device changes from time t to time t+n, and the pre-distortion correction parameters are obtained once at these two times. When the projection device executes processing logic 2 at time t+m (m>n), the posture of the projection device is closer to the posture at time t+n. Therefore, using the pre-distortion correction parameters at time t+n is more accurate than using the pre-distortion correction parameters at time t.

[0026] Based on this, in the above processing logic 1, the projection device needs to record the latest pre-distortion correction parameters to ensure that when executing the processing logic 2, the pre-distortion correction parameters are closer to the current posture.

[0027] Optionally, in processing logic 1, each time the projection device obtains a pre-distortion correction parameter, it stores it in a fixed address of the cache and overwrites the pre-distortion correction parameter stored last time, so that in processing logic 2, the pre-distortion correction parameter extracted by the projection device from the fixed address of the cache is always the latest parameter at the current moment.

[0028] Optionally, in processing logic 1, each time the projection device obtains a pre-distortion correction parameter, it stores it in the latest address of the cache, so that in processing logic 2, the pre-distortion correction parameter extracted by the projection device from the latest address of the cache is always the latest parameter at the current moment.

[0029] It should be noted that, in addition to the above method, the projection device may also save and record the latest pre-distortion correction parameters in other ways, which is not specifically limited in the embodiments of the present application.

[0030] As mentioned above, in order to improve the accuracy of pre-distortion correction, the projection device needs to save and record the latest pre-distortion correction parameters. However, if only the above-mentioned actual measurement method is used, due to some factors such as detection frame rate, processing delay, and transmission time consumption, even if the pre-distortion correction parameters are the latest, their acquisition time is still far away from the execution time of processing logic 2, so that they do not fit the execution time of processing logic 2, thereby affecting the effect of pre-distortion correction, so that users can still feel the distortion.

[0031] Based on this, the embodiments of the present application may adopt the above-mentioned prediction method, or combine the actual measurement method with the prediction method. The goal is to make the acquisition time of the latest pre-distortion correction parameter and the execution time interval of the processing logic 2 shorter (for example, the time difference between the two is less than a preset threshold).

[0032] In a possible implementation, the projection device inserts one or more times of obtaining the predistortion correction parameters by prediction between two times of obtaining the predistortion correction parameters by actual measurement. This can increase the number of times the predistortion correction parameters are obtained, improve the frequency of obtaining the predistortion correction parameters, and make up for the defect of the sensor's low detection frame rate, so that when processing high frame rate video images, more and more suitable predistortion correction parameters can be used.

[0033] In a possible implementation, after obtaining the pre-distortion correction parameters by actual measurement, the projection device may obtain the pre-distortion correction parameters of the execution time of the corresponding processing logic 2 by prediction based on the parameters. In this way, the acquisition time of the pre-distortion correction parameters can be closer to the execution time of the processing logic 2, thereby shortening the effective delay of the pre-distortion correction parameters.

[0034] In a possible implementation, the projection device inserts one or more times of obtaining the predistortion correction parameters by prediction between two times of obtaining the predistortion correction parameters by actual measurement, and after obtaining the predistortion correction parameters by actual measurement, obtains the predistortion correction parameters of the execution time of the corresponding processing logic 2 by prediction based on the parameters. In this way, the number of times of obtaining the predistortion correction parameters can be increased, the frequency of obtaining can be improved, and the time of obtaining the predistortion correction parameters can be closer to the time of processing the corresponding image.

[0035] It should be noted that in order to improve the accuracy of the pre-distortion correction parameters, the embodiment of the present application may also adopt other methods, which are not specifically limited.

[0036] According to the pre-distortion correction parameters, the current image can be pre-distorted (also called pre-distortion transformation) to obtain a pre-distorted image (also called a target image). Relative to the rectangular current image, the pre-distorted image is its distorted appearance, but this distortion is the opposite of the trapezoidal distortion generated by the projection device. After the trapezoidal distortion of the projection, a rectangular image can be finally seen on the target plane. The principle is similar to "negative equals positive", that is, after the current image is pre-distorted, a pre-distorted image with reverse trapezoidal distortion is first obtained, and then the pre-distorted image is projected by the projection device, so that the trapezoidal distortion is generated again, thereby obtaining a rectangular current image on the target plane.

[0037] In a possible implementation, if the difference between the sensor data read twice is small, the projection device may consider not calculating new pre-distortion correction parameters.

[0038] The projection device can read the sensor data multiple times and compare the data from two adjacent readings. If the difference between the two is small, for example, the difference between the two is less than a preset threshold, then it can be considered that the displacement itself is small, or no displacement may have occurred. In order to save computing power and improve efficiency, the old pre-distortion correction parameters can still be used for pre-distortion correction.

[0039] In a possible implementation, the projection device may be triggered to perform keystone correction based on the following situations:

[0040] (1) User-triggered keystone correction: For example, a keystone correction switch is set, and the user can actively turn on the switch, thereby triggering the projection device to automatically perform keystone correction.

[0041] (2) Triggering keystone correction when the device is turned on. After the projection device is turned on, the keystone correction function is automatically turned on.

[0042] (3) The projection device detects a change in posture and triggers trapezoidal correction. The projection device may be in a stationary or moving state, both of which can be detected by reading sensor data. If it is in a stationary state, then a trapezoidal correction can be triggered once; if it is in a moving state, the projection method provided in this application can be used to perform trapezoidal correction.

[0043] In a second aspect, the present application provides a projection device, comprising: an acquisition module, used to acquire a first pre-distortion correction parameter, wherein the first pre-distortion correction parameter is obtained in advance based on a first predicted posture of a projection device; a correction module, used to perform pre-distortion correction on a current image according to the first pre-distortion correction parameter to obtain a first target image; and a projection module, used to project the first target image onto a target plane.

[0044] In a possible implementation, it also includes: a prediction module, used to obtain a historical posture, where the historical posture is obtained based on previously collected sensor data; predicting based on the historical posture to obtain the first predicted posture; and obtaining the first pre-distortion correction parameter based on the first predicted posture.

[0045] In a possible implementation, the prediction module is specifically configured to perform prediction based on the historical posture to obtain at least one predicted posture corresponding to at least one moment; and determine the predicted posture corresponding to the at least one moment that is closest to the current moment as the first predicted posture.

[0046] In a possible implementation, the prediction module is specifically configured to input the historical posture into a machine learning model to obtain the at least one predicted posture.

[0047] In a possible implementation, the prediction module is specifically configured to perform particle filtering processing according to the historical posture to obtain the at least one predicted posture.

[0048] In a possible implementation, the acquisition module is further used to acquire sensor data collected twice adjacently; when the similarity of the sensor data collected twice adjacently is greater than a preset threshold, a second pre-distortion correction parameter is acquired, and the second pre-distortion correction parameter is the pre-distortion correction parameter used last time; the correction module is further used to perform pre-distortion correction on the current image according to the second pre-distortion correction parameter to obtain a second target image; the projection module is further used to project the second target image onto a target plane.

[0049] In a possible implementation, the acquisition module is further used to acquire a third pre-distortion correction parameter, where the third pre-distortion correction parameter is obtained based on sensor data collected in real time; the correction module is further used to perform pre-distortion correction on the current image based on the third pre-distortion correction parameter to obtain a third target image; and the projection module is further used to project the third target image onto a target plane.

[0050] In a third aspect, the present application provides a projection device, comprising: a sensor and a spatial light modulator (SLM) driver chip; the sensor and the SLM driver chip are connected via a bus; the sensor is used to obtain corresponding sensor data according to the posture of the projection device; the SLM driver chip is used to implement the method as described in any one of the above-mentioned first aspects.

[0051] In a fourth aspect, the present application provides a computer-readable storage medium, comprising a computer program, wherein when the computer program is executed on a computer, the computer executes any one of the methods described in the first aspect.

[0052] In a fifth aspect, the present application provides a computer program product, wherein the computer program product comprises a computer program code, and when the computer program code is executed on a computer, the computer executes any one of the methods in the first aspect.

[0053] In a sixth aspect, the present application provides a chip comprising a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute a method as described in any one of the above-mentioned first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 Here are some examples of keystone distortion:

[0055] Figure 2 This is a comparison chart of keystone correction;

[0056] Figure 3 A schematic diagram of an implementation method for automatic keystone correction of a projection device;

[0057] Figure 4 A schematic block diagram of a projection device 400 according to an embodiment of the present application is shown;

[0058] Figure 5 A flowchart of a process 500 of a projection method according to an embodiment of the present application;

[0059] Figure 6 This is a schematic diagram of an embodiment of the present application implemented in a driver chip;

[0060] Figure 7 A schematic diagram of interpolation calculation in an embodiment of the present application;

[0061] Figure 8 A schematic diagram of posture pre-compensation according to an embodiment of the present application;

[0062] Fig. 9 A schematic diagram of parallel computing of an embodiment of the present application;

[0063] Fig.10 Schematic diagram of the structure of the projection device 1000 according to the embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0065] The terms "first", "second", etc. in the specification embodiments, claims, and drawings of the present application are only used for the purpose of distinguishing descriptions, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, including a series of steps or units. The method, system, product, or device is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0066] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0067] Before introducing the technical solution of the present application, the relevant technologies involved in the embodiments of the present application are first described.

[0068] 1. Artificial Intelligence (AI)

[0069] (1) Neural Network

[0070] Neural Network (NN) is a machine learning model. A neural network can be composed of neural units. A neural unit can refer to an operation unit with xs and intercept 1 as input. The output of the operation unit can be:

[0071]

[0072] Where s=1, 2, ...n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolution layer. The activation function can be a sigmoid function. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the characteristics of the local receptive field. The local receptive field can be an area composed of several neural units.

[0073] (2) Deep Neural Networks

[0074] Deep Neural Network (DNN), also known as multi-layer neural network, can be understood as a neural network with many hidden layers. There is no special metric for "many" here. From the position of different layers of DNN, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. Although DNN looks complicated, the work of each layer is actually not complicated. Simply put, it is the following linear relationship expression: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since DNN has many layers, the coefficient W and the offset vector The definition of these parameters in DNN is as follows: Taking coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the output third layer index 2 and the input second layer index 4. In summary, the coefficients from the kth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as It should be noted that the input layer does not have a W parameter. In a deep neural network, more hidden layers allow the network to better describe complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (the weight matrix formed by many layers of vector W).

[0075] (3) Convolutional Neural Network

[0076] Convolutional Neuron Network (CNN) is a deep neural network with a convolutional structure. It is a deep learning architecture. A deep learning architecture refers to multiple levels of learning at different levels of abstraction through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network in which each neuron can respond to the image input into it. The convolutional neural network contains a feature extractor consisting of a convolution layer and a pooling layer. The feature extractor can be regarded as a filter, and the convolution process can be regarded as using a trainable filter to convolve an input image or convolution feature plane (feature map).

[0077] A convolutional layer is a layer of neurons in a convolutional neural network that performs convolution on the input signal. A convolutional layer can include many convolution operators, also known as kernels. In image processing, the convolution operator is essentially a weight matrix, which is usually predefined. During the convolution operation on the image, the weight matrix is ​​usually processed horizontally on the input image one pixel after another (or two pixels after two pixels... depending on the value of the stride) to complete the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image. During the convolution operation, the weight matrix will extend to the entire depth of the input image. Therefore, convolution with a single weight matrix will produce a convolution output with a single depth dimension, but in most cases, instead of using a single weight matrix, multiple weight matrices of the same size (rows × columns) are applied, that is, multiple isotype matrices. The output of each weight matrix is ​​stacked to form the depth dimension of the convolution image, where the dimension can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features in the image, for example, one weight matrix is ​​used to extract image edge information, another weight matrix is ​​used to extract specific colors of the image, and another weight matrix is ​​used to blur unnecessary noise points in the image. The multiple weight matrices have the same size (rows × columns), and the size of the feature maps extracted by the multiple weight matrices of the same size is also the same. The extracted multiple feature maps of the same size are then merged to form the output of the convolution operation. The weight values ​​in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, so that the convolutional neural network can make correct predictions. When the convolutional neural network has multiple convolutional layers, the initial convolutional layer often extracts more general features, which can also be called low-level features; as the depth of the convolutional neural network deepens, the features extracted by the later convolutional layers become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

[0078] Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer. It can be a convolution layer followed by a pooling layer, or multiple convolution layers followed by one or more pooling layers. In the image processing process, the only purpose of the pooling layer is to reduce the spatial size of the image. The pooling layer can include an average pooling operator and / or a maximum pooling operator to sample the input image to obtain an image of smaller size. The average pooling operator can calculate the pixel values ​​in the image within a specific range to produce an average value as the result of average pooling. The maximum pooling operator can take the pixel with the largest value in the range as the result of maximum pooling within a specific range. In addition, just as the size of the weight matrix used in the convolution layer should be related to the image size, the operator in the pooling layer should also be related to the image size. The size of the image output after processing by the pooling layer can be smaller than the size of the image input to the pooling layer. Each pixel in the image output by the pooling layer represents the average value or maximum value of the corresponding sub-region of the image input to the pooling layer.

[0079] After being processed by the convolution layer / pooling layer, the convolution neural network is not sufficient to output the required output information. Because as mentioned above, the convolution layer / pooling layer will only extract features and reduce the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolution neural network needs to use the neural network layer to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer may include multiple hidden layers, and the parameters contained in the multiple hidden layers can be pre-trained according to the relevant training data of the specific task type. For example, the task type may include image recognition, image classification, image super-resolution reconstruction, etc.

[0080] Optionally, after the multiple hidden layers in the neural network layer, an output layer of the entire convolutional neural network is also included. The output layer has a loss function similar to the classification cross entropy, which is specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network is completed, the back propagation will begin to update the weight values ​​and biases of the aforementioned layers to reduce the loss of the convolutional neural network and the error between the result output by the convolutional neural network through the output layer and the ideal result.

[0081] (4) Recurrent Neural Network

[0082] Recurrent Neural Networks (RNN) are used to process sequence data. In the traditional neural network model, from the input layer to the hidden layer and then to the output layer, the layers are fully connected, and the nodes within each layer are disconnected. Although this ordinary neural network has solved many difficult problems, it is still powerless for many problems. For example, if you want to predict the next word in a sentence, you generally need to use the previous word, because the previous and next words in a sentence are not independent. The reason why RNN is called a recurrent neural network is that the current output of a sequence is also related to the previous output. The specific manifestation is that the network will remember the previous information and apply it to the calculation of the current output, that is, the nodes between the hidden layers are no longer disconnected but connected, and the input of the hidden layer includes not only the output of the input layer but also the output of the hidden layer at the previous moment. In theory, RNN can process sequence data of any length. The training of RNN is the same as the training of traditional CNN or DNN. The same error back propagation algorithm is used, but there is a difference: that is, if the RNN is expanded, the parameters, such as W, are shared; this is not the case with the traditional neural network mentioned above. And when using the gradient descent algorithm, the output of each step depends not only on the network at the current step, but also on the state of the network at the previous steps. This learning algorithm is called the Back Propagation Through Time (BPTT).

[0083] Since we already have convolutional neural networks, why do we need recurrent neural networks? The reason is simple. In convolutional neural networks, there is a premise that the elements are independent of each other, and the input and output are also independent, such as cats and dogs. But in the real world, many elements are interconnected, such as the change of stocks over time, or someone said: I like traveling, and my favorite place is Yunnan. I must go there if I have the chance in the future. Fill in the blank here, humans should all know to fill in "Yunnan". Because humans will make inferences based on the content of the context, but how can machines do this? RNN came into being. RNN is designed to give machines the ability to remember like humans. Therefore, the output of RNN needs to rely on the current input information and historical memory information.

[0084] (5) Loss Function

[0085] In the process of training a deep neural network, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible.

[0086] (6) Back propagation algorithm

[0087] Convolutional neural networks can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial super-resolution model during the training process, so that the reconstruction error loss of the super-resolution model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the error loss information is back-propagated to update the parameters in the initial super-resolution model, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, aiming to obtain the optimal parameters of the super-resolution model, such as the weight matrix.

[0088] (7) Generative Adversarial Networks

[0089] Generative Adversarial Networks (GAN) is a deep learning model. The model includes at least two modules: one module is the generative model, and the other module is the discriminative model. Through the game learning between these two modules, better output is generated. Both the generative model and the discriminative model can be neural networks, specifically deep neural networks or convolutional neural networks. The basic principle of GAN is as follows: Taking the GAN that generates pictures as an example, suppose there are two networks, G (Generator) and D (Discriminator), where G is a network that generates pictures. It receives a random noise z and generates pictures through this noise, denoted as G(z); D is a discriminative network, which is used to determine whether a picture is "real". Its input parameter is x, x represents a picture, and the output D(x) represents the probability that x is a real picture. If it is 1, it means that it is 100% a real picture, and if it is 0, it means that it cannot be a real picture. In the process of training the generative adversarial network, the goal of the generative network G is to generate realistic images as much as possible to deceive the discriminative network D, while the goal of the discriminative network D is to distinguish the images generated by G from the real images as much as possible. In this way, G and D constitute a dynamic "game" process, which is also the "adversary" in the "generative adversarial network". The final result of the game is that, under ideal conditions, G can generate images G(z) that are "real" enough, while D has difficulty determining whether the images generated by G are real, that is, D(G(z)) = 0.5. In this way, an excellent generative model G is obtained, which can be used to generate images.

[0090] 2. Keystone distortion

[0091] When the projection optical axis of the projection device is not perpendicular to the target plane (for example, a screen, a wall, etc.), the projected image will not be the expected rectangle, but a trapezoid. For example, when there is a vertical angle between the projection optical axis and the target plane, the image is a vertical trapezoid; when there is a horizontal angle between the projection optical axis and the target plane, the image is a horizontal trapezoid; when the projection optical axis rotates, the image also rotates; when the aforementioned three angles exist at the same time, the image presents an arbitrary quadrilateral; when the projection lens is distorted or the screen is spherical, cylindrical or any other shape, the projection will also appear in an arc, spherical or even arbitrary curved shape.

[0092] Generally, one-dimensional vertical trapezoids and horizontal trapezoids can be defined as one-dimensional (1D) keystone distortion (KST), arbitrary quadrilateral distortion can be defined as two-dimensional keystone distortion (2D KST), and arc-shaped, spherical or arbitrary shaped distortion can be defined as three-dimensional keystone distortion (3D KST). Figure 1 are several examples of trapezoidal distortion. The trapezoidal correction method in this application is mainly for 1D KST and 2D KST, that is, Figure 1 Vertical trapezoids, horizontal trapezoids, and any quadrilaterals in .

[0093] 3. Keystone correction

[0094] Manual trapezoidal correction means that the user moves the control points (for example, the four vertices of a trapezoid) in the projected image through a remote control, etc. The processor calculates the pre-distortion correction parameters based on the front and rear coordinates of the control points, performs pre-distortion correction on the original image, and finally projects the corrected image for output.

[0095] Automatic keystone correction refers to the use of sensors to detect the deflection angles of the projection optical axis in various directions, such as the pitch angle α, roll angle β, yaw angle θ, etc., to obtain the degree of image distortion, calculate the pre-distortion correction parameters, perform pre-distortion correction on the original image, and finally project the corrected image for output.

[0096] A method of performing trapezoidal correction digitally can pre-distort the original input image and then project the transformed image to project the desired rectangular image. Pre-distortion processing is a perspective transformation. Figure 2 This is a comparison chart of keystone correction, such as Figure 2 As shown, when no trapezoidal correction is performed, after the original image is projected on the target plane, a trapezoidal projection screen is seen; when trapezoidal correction is performed, the original image is first pre-distorted (also called pre-distortion transformation) to obtain a pre-distorted image (also called target image), and then the pre-distorted image is projected on the target plane, and a rectangular projection screen is seen.

[0097] Figure 3 A schematic diagram of an implementation method for automatic keystone correction of a projection device, such as Figure 3As shown, the sensors may include: an inertial measurement unit (IMU), a time of flight sensor (ToF), and a camera. For example, a three-axis accelerometer can be used to detect the pitch and roll angles of the projection optical axis, two single-point ToFs can detect the yaw angle, and Cemera can detect the pitch and yaw angles. The microcontroller unit (MCU) / system on chip (SoC) is responsible for the front-end processing of the video stream and starts automatic trapezoidal correction. The spatial light modulator (SLM) driver chip such as the digital light processing controller (DLPC) is responsible for further display adaptation of the video. The SLM display chip such as the digital micromirror device (DMD) is responsible for modulating the spatial light to obtain the display image.

[0098] The process of single automatic keystone correction includes the following steps:

[0099] ① Read: MCU / SoC reads the sensor data once to obtain the projection device posture. If the sensor data changes slightly (for example, the change in sensor data read twice is less than the preset threshold), the projection device posture will not be recalculated.

[0100] ② Calculation: MCU / SoC establishes the projection coordinate relationship and calculates the projection rectangle and corresponding pre-distortion parameters.

[0101] ③ Allocation: MCU / SoC configures the pre-distortion correction parameters to the Graphics Processing Unit (GPU).

[0102] ④ Use: GPU takes effect of pre-distortion correction parameters, performs pre-distortion transformation on the original image, and obtains a pre-distorted image. The image data stream of the pre-distorted image is transmitted to SLM driver chips such as DLPC and SLM display chips such as DMD through a bus (e.g., a serial bus such as a serial peripheral interface (SPI)), thereby displaying a rectangular projection screen.

[0103] The "use" in the above step ④ can occur in the GPU in the front-end MCU / SoC to perform software trapezoidal correction, or it can occur in the SLM driver chip such as DLPC to perform hardware trapezoidal correction.

[0104] When dynamic trapezoidal correction is performed (i.e., trapezoidal correction is performed in real time during the posture change of the projection device so that the picture remains rectangular at all times), the projection device periodically performs the above steps during movement to ensure that the picture is always "square" (i.e., a rectangular picture) in the eyes of the user.

[0105] but Figure 3 In the method shown, the four steps in the process of a single automatic keystone correction are executed serially. Due to the following factors, this method is difficult to meet the requirements of high frame rate implementation, and the projected image may not always remain square, and even the user may still be able to perceive keystone distortion.

[0106] (1) The detection frame rate of the sensor;

[0107] (2) The time delay for the software to calculate the pre-distortion correction parameters;

[0108] (3) GPU load (software keystone correction) and the time required to transfer data over the serial bus (hardware keystone correction);

[0109] (4) GPU image processing delay or the delay in the effectiveness of pre-distortion correction parameters.

[0110] In order to solve the above technical problems, the present application provides a projection method and device, and the technical solution of the present application will be described below.

[0111] Figure 4 A schematic block diagram of a projection device 400 according to an embodiment of the present application is shown. Figure 4 , which shows a structural schematic diagram when the projection device 400 is a projector.

[0112] The projection device 400 may include a processor 410, an external memory interface 420, an internal memory 421, a universal serial bus (USB) interface 430, a charging management module 440, a power management module 441, a battery 442, an antenna 2, a wireless communication module 460, an audio module 470, a speaker 470A, a microphone 470C, an earphone interface 470D, a sensor module 480, a button 490, a motor 491, an indicator 492, and a display screen 494. The sensor module 480 may include a pressure sensor 480A, a gyroscope sensor 480B, an acceleration sensor 480E, a distance sensor 480F, a touch sensor 480K, an ambient light sensor 480L, and the like.

[0113] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the projection device 400. In other embodiments of the present application, the projection device 400 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0114] The processor 410 may include one or more processing units, for example, the processor 410 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), a controller, a video codec, a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0115] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0116] The processor 410 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 410 is a cache memory. The memory may store instructions or data that the processor 410 has just used or cyclically used. If the processor 410 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 410, and thus improves the efficiency of the system.

[0117] In some embodiments, the processor 410 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, and / or a universal serial bus (USB) interface, etc.

[0118] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 410 may include multiple groups of I2C buses. The processor 410 may be coupled to the touch sensor 480K, the charger, etc. through different I2C bus interfaces. For example, the processor 410 may be coupled to the touch sensor 480K through the I2C interface, so that the processor 410 communicates with the touch sensor 480K through the I2C bus interface, thereby realizing the touch function of the projection device 400.

[0119] The I2S interface can be used for audio communication. In some embodiments, the processor 410 can include multiple I2S buses. The processor 410 can be coupled to the audio module 470 via the I2S bus to achieve communication between the processor 410 and the audio module 470. In some embodiments, the audio module 470 can transmit an audio signal to the wireless communication module 460 via the I2S interface to achieve the function of playing sound through a Bluetooth speaker.

[0120] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 470 and the wireless communication module 460 can be coupled via a PCM bus interface. In some embodiments, the audio module 470 can also transmit audio signals to the wireless communication module 460 via the PCM interface to achieve the function of playing sound through a Bluetooth speaker. Both the I2S interface and the PCM interface can be used for audio communication.

[0121] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 410 and the wireless communication module 460. For example, the processor 410 communicates with the Bluetooth module in the wireless communication module 460 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 470 can transmit an audio signal to the wireless communication module 460 through the UART interface to implement the function of playing sound through a Bluetooth speaker.

[0122] The MIPI interface can be used to connect the processor 410 to peripheral devices such as the display screen 494. The MIPI interface includes a display serial interface (DSI), etc. In some embodiments, the processor 410 and the display screen 494 communicate via the DSI interface to implement the display function of the projection device 400.

[0123] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 410 with the display screen 494, the wireless communication module 460, the audio module 470, the sensor module 480, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0124] The USB interface 430 is an interface that complies with USB standard specifications, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 430 can be used to connect a charger to charge the projection device 400, and can also be used to transmit data between the projection device 400 and peripheral devices. It can also be used to connect headphones to play audio through the headphones.

[0125] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the projection device 400. In other embodiments of the present application, the projection device 400 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0126] The charging management module 440 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 440 may receive charging input from a wired charger through the USB interface 430. In some wireless charging embodiments, the charging management module 440 may receive wireless charging input through a wireless charging coil of the projection device 400. While the charging management module 440 is charging the battery 442, it may also power the projection device through the power management module 441.

[0127] The power management module 441 is used to connect the battery 442, the charging management module 440 and the processor 410. The power management module 441 receives input from the battery 442 and / or the charging management module 440, and supplies power to the processor 410, the internal memory 421, the display screen 494, the camera 493, and the wireless communication module 460. The power management module 441 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 441 can also be set in the processor 410. In other embodiments, the power management module 441 and the charging management module 440 can also be set in the same device.

[0128] The wireless communication function of the projection device 400 can be implemented through the antenna 2, the wireless communication module 460, the modem processor and the baseband processor.

[0129] Antenna 2 is used to transmit and receive electromagnetic wave signals. Each antenna in projection device 400 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 2 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0130] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 470A, etc.), or displays an image or video through a display screen 494. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 410 and be arranged in the same device with other functional modules.

[0131] The wireless communication module 460 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), near field communication (NFC), infrared (IR) and the like applied to the projection device 400. The wireless communication module 460 can be one or more devices integrating at least one communication processing module. The wireless communication module 460 receives electromagnetic waves via the antenna 2, modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 410. The wireless communication module 460 can also receive the signal to be sent from the processor 410, modulate the frequency, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0132] The projection device 400 implements the display function through a GPU, a display screen 494, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 494 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 410 may include one or more GPUs, which execute program instructions to generate or change display information.

[0133] The display screen 494 is used to display the control interface, etc. The display screen 494 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the projection device 400 may include 1 or N display screens 494, where N is a positive integer greater than 1.

[0134] Video codecs are used to compress or decompress digital videos. The projection device 400 may support one or more video codecs. Thus, the projection device 400 may play or record videos in a variety of coding formats, such as moving picture experts group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0135] NPU is a neural network (NN) computing processor, which can quickly process input information and continuously self-learn by drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain. NPU can realize applications such as intelligent cognition of the projection device 400, such as posture prediction, etc.

[0136] The external memory interface 420 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the projection device 400. The external memory card communicates with the processor 410 through the external memory interface 420 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0137] The internal memory 421 can be used to store computer executable program codes, which include instructions. The internal memory 421 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, at least one application required for a function, etc. The data storage area may store data created during the use of the projection device 400, etc. In addition, the internal memory 421 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 410 executes various functional applications and data processing of the projection device 400 by running instructions stored in the internal memory 421 and / or instructions stored in a memory provided in the processor.

[0138] The projection device 400 can implement audio functions, such as sound playback, through the audio module 470, the speaker 470A, the microphone 470C, the headphone interface 470D, and the application processor.

[0139] The audio module 470 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal. The audio module 470 can also be used to encode and decode audio signals. In some embodiments, the audio module 470 can be arranged in the processor 410, or some functional modules of the audio module 470 can be arranged in the processor 410.

[0140] The speaker 470A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The projection device 400 can listen to music or listen to a hands-free call through the speaker 470A.

[0141] Microphone 470C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can make a sound by approaching the microphone 470C with his mouth to input the sound signal into the microphone 470C. The projection device 400 can be provided with at least one microphone 470C. In other embodiments, the projection device 400 can be provided with two microphones 470C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the projection device 400 can also be provided with three, four or more microphones 470C to realize the collection of sound signals, noise reduction, identification of sound sources, realization of directional recording function, etc.

[0142] The earphone interface 470D is used to connect a wired earphone and can be a USB interface 430 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0143] The pressure sensor 480A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 480A can be disposed on the display screen 494. There are many types of pressure sensors 480A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. A capacitive pressure sensor can include at least two parallel plates having a conductive material.

[0144] The gyro sensor 480B may be used to determine the motion posture of the projection device 400 .

[0145] The acceleration sensor 480E can detect the magnitude of the acceleration of the projection device 400 in various directions (generally three axes).

[0146] Distance sensor 480F, used to measure distance.

[0147] The ambient light sensor 480L is used to sense the ambient light brightness.

[0148] The touch sensor 480K is also called a "touch device". The touch sensor 480K can be set on the display screen 494, and the touch sensor 480K and the display screen 494 form a touch screen, also called a "touch screen". The touch sensor 480K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 494. In other embodiments, the touch sensor 480K can also be set on the surface of the projection device 400, which is different from the position of the display screen 494.

[0149] The key 490 includes a power key, a volume key, etc. The key 490 may be a mechanical key or a touch key. The projection device 400 may receive key input and generate key signal input related to user settings and function control of the projection device 400.

[0150] Motor 491 can generate vibration prompts. Motor 491 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback.

[0151] Indicator 492 may be an indicator light, which may be used to indicate connection status, power changes, messages, notifications, etc.

[0152] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the projection device. In other embodiments of the present application, the projection device may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0153] Figure 5 Flow chart of process 500 of the projection method of the present application embodiment. Process 500 may be performed by the above-mentioned projection device. Process 500 is described as a series of steps or operations. It should be understood that process 500 may be performed in various orders and / or occur simultaneously, not limited to Figure 5 The execution order shown. Process 500 may include:

[0154] Step 501: Obtain pre-distortion correction parameters.

[0155] Distortion correction parameters may include***.

[0156] The pre-distortion correction parameters can be obtained by the following two methods:

[0157] One method adopts actual measurement method, whereby the projection device uses sensors to detect the deflection angles of the projection optical axis in various directions, such as pitch angle α, roll angle β, yaw angle θ, etc., to obtain the posture of the projection device, and obtain the degree of distortion of the picture according to the posture of the projection device, thereby calculating the pre-distortion correction parameters.

[0158] In this case, the pre-distortion correction parameters are obtained based on the actual measured data of the sensor. Figure 3 The embodiments shown will not be described in detail here.

[0159] Another method uses a prediction method. The projection device predicts the posture of the projection device based on historical data measured by the sensor or the historical posture of the projection device (the historical posture can be obtained based on historical data), obtains the degree of distortion of the picture based on the posture of the projection device, and thus calculates the pre-distortion correction parameters.

[0160] In this case, the pre-distortion correction parameters are predicted based on historical data / historical postures.

[0161] Optionally, the projection device can adopt a machine learning model, input historical data / historical posture, output the posture of the projection device, and then obtain the degree of distortion of the picture according to the posture of the projection device, so as to calculate the pre-distortion correction parameters.

[0162] The technology of the above machine learning model can be referred to the above description and will not be repeated here.

[0163] Optionally, the projection device may adopt a traditional solution, such as a particle filter, to obtain the posture of the projection device, and then obtain the degree of distortion of the picture according to the posture of the projection device, so as to calculate the pre-distortion correction parameters.

[0164] Particle filtering methods can include:

[0165] (1) Initialization phase: Read sensor data at time t and obtain the posture at time t. Based on this, initialize some randomly distributed particles;

[0166] (2) State prediction stage: The velocity and direction of movement calculated from the sensor data are assigned to the above particles, and the posture of the particles at the future time t+1 is predicted. Optionally, the average posture of all particles can be directly used as the posture at time t+1;

[0167] At this point, the predicted posture at the future time t+1 can be obtained. The next two steps are to update the particle weights to make subsequent predictions more accurate.

[0168] (3) Weight update stage: particle weights can be updated based on sensor data at time t+1;

[0169] (4) Resampling stage: New particles are extracted according to the updated particle weights. After resampling, particles with higher weights have a greater probability of being retained, and particles with smaller probabilities have a greater probability of being eliminated;

[0170] (5) Enter the next state prediction stage.

[0171] It should be noted that, in addition to the above two optional methods, the projection device may also use other methods to predict the pre-distortion correction parameters, which is not specifically limited in the embodiments of the present application.

[0172] In an embodiment of the present application, the projection device can execute two processing logics, wherein processing logic 1 periodically obtains pre-distortion correction parameters, and processing logic 2 performs pre-distortion correction on the image to be projected according to the pre-distortion correction parameters. The pre-distortion correction parameters required by processing logic 2 come from processing logic 1, and the closer the acquisition time of the pre-distortion correction parameters obtained by processing logic 2 is to the execution time of processing logic 2, the more the accuracy of the pre-distortion correction can be improved, that is, the shorter the effective delay (the time interval from its acquisition to its use) of the pre-distortion correction parameters obtained by processing logic 2, the more the accuracy of the pre-distortion correction can be improved. This is because the posture of the moving projection device is changing, and in order to ensure that the picture is always "square" (i.e., a rectangular picture) in the eyes of the user, the pre-distortion correction parameters obtained based on the posture are also changing. For example, the posture of the projection device changes from time t to time t+n, and the pre-distortion correction parameters are obtained once at these two times. When the projection device executes processing logic 2 at time t+m (m>n), the posture of the projection device is closer to the posture at time t+n. Therefore, using the pre-distortion correction parameters at time t+n is more accurate than using the pre-distortion correction parameters at time t.

[0173] Based on this, in the above processing logic 1, the projection device needs to record the latest pre-distortion correction parameters to ensure that when executing the processing logic 2, the pre-distortion correction parameters are closer to the current posture.

[0174] Optionally, in processing logic 1, each time the projection device obtains a pre-distortion correction parameter, it stores it in a fixed address of the cache and overwrites the pre-distortion correction parameter stored last time, so that in processing logic 2, the pre-distortion correction parameter extracted by the projection device from the fixed address of the cache is always the latest parameter at the current moment.

[0175] Optionally, in processing logic 1, each time the projection device obtains a pre-distortion correction parameter, it stores it in the latest address of the cache, so that in processing logic 2, the pre-distortion correction parameter extracted by the projection device from the latest address of the cache is always the latest parameter at the current moment.

[0176] It should be noted that, in addition to the above method, the projection device may also save and record the latest pre-distortion correction parameters in other ways, which is not specifically limited in the embodiments of the present application.

[0177] As mentioned above, in order to improve the accuracy of pre-distortion correction, the projection device needs to save and record the latest pre-distortion correction parameters. However, if only the above-mentioned actual measurement method is used, due to some factors such as detection frame rate, processing delay, and transmission time consumption, even if the pre-distortion correction parameters are the latest, their acquisition time is still far away from the execution time of processing logic 2, so that they do not fit the execution time of processing logic 2, thereby affecting the effect of pre-distortion correction, so that users can still feel the distortion.

[0178] Based on this, the embodiments of the present application may adopt the above-mentioned prediction method, or combine the actual measurement method with the prediction method. The goal is to make the acquisition time of the latest pre-distortion correction parameter and the execution time interval of the processing logic 2 shorter (for example, the time difference between the two is less than a preset threshold).

[0179] In a possible implementation, the projection device inserts one or more times of obtaining the predistortion correction parameters by prediction between two times of obtaining the predistortion correction parameters by actual measurement. This can increase the number of times the predistortion correction parameters are obtained, improve the frequency of obtaining the predistortion correction parameters, and make up for the defect of the sensor's low detection frame rate, so that when processing high frame rate video images, more and more suitable predistortion correction parameters can be used.

[0180] In a possible implementation, after obtaining the pre-distortion correction parameters by actual measurement, the projection device may obtain the pre-distortion correction parameters of the execution time of the corresponding processing logic 2 by prediction based on the parameters. In this way, the acquisition time of the pre-distortion correction parameters can be closer to the execution time of the processing logic 2, thereby shortening the effective delay of the pre-distortion correction parameters.

[0181] In a possible implementation, the projection device inserts one or more times of obtaining the predistortion correction parameters by prediction between two times of obtaining the predistortion correction parameters by actual measurement, and after obtaining the predistortion correction parameters by actual measurement, obtains the predistortion correction parameters of the execution time of the corresponding processing logic 2 by prediction based on the parameters. In this way, the number of times of obtaining the predistortion correction parameters can be increased, the frequency of obtaining can be improved, and the time of obtaining the predistortion correction parameters can be closer to the time of processing the corresponding image.

[0182] It should be noted that in order to improve the accuracy of the pre-distortion correction parameters, the embodiment of the present application may also adopt other methods, which are not specifically limited.

[0183] Step 502: Perform pre-distortion correction on the current image according to the pre-distortion correction parameters to obtain a target image.

[0184] Step 503: Project the target image onto the target plane.

[0185] Reference Figure 2 According to the pre-distortion correction parameters, the current image can be pre-distorted (also called pre-distortion transformation) to obtain a pre-distorted image (also called a target image). Relative to the rectangular current image, the pre-distorted image is its distorted appearance, but this distortion is the opposite of the trapezoidal distortion generated by the projection device. After the trapezoidal distortion of the projection, a rectangular image can be finally seen on the target plane. The principle is similar to "negative plus negative equals positive", that is, after the current image is pre-distorted, a pre-distorted image with reverse trapezoidal distortion is first obtained, and then after the projection of the projection device, the pre-distorted image is again trapezoidally distorted, so that a rectangular current image is obtained on the target plane.

[0186] In the embodiment of the present application, by adopting a prediction method to obtain pre-distortion correction parameters, the pre-distortion correction parameters can be made closer to the posture of the projection device during distortion correction, so as to meet the requirements of high frame rate implementation and keep the projected image square at all times.

[0187] In a possible implementation, if the difference between the sensor data read twice is small, the projection device may consider not calculating new pre-distortion correction parameters.

[0188] The projection device can read the sensor data multiple times and compare the data from two adjacent readings. If the difference between the two is small, for example, the difference between the two is less than a preset threshold, then it can be considered that the displacement itself is small, or no displacement may have occurred. In order to save computing power and improve efficiency, the old pre-distortion correction parameters can still be used for pre-distortion correction.

[0189] In a possible implementation, the projection device may be triggered to perform keystone correction based on the following situations:

[0190] (1) User-triggered keystone correction: For example, a keystone correction switch is set, and the user can actively turn on the switch, thereby triggering the projection device to automatically perform keystone correction.

[0191] (2) Triggering keystone correction when the device is turned on. After the projection device is turned on, the keystone correction function is automatically turned on.

[0192] (3) The projection device detects a change in posture and triggers trapezoidal correction. The projection device may be in a stationary or moving state, both of which can be detected by reading sensor data. If it is in a stationary state, then a trapezoidal correction can be triggered once; if it is in a moving state, the projection method provided in this application can be used to perform trapezoidal correction.

[0193] Figure 6 This is a schematic diagram of an embodiment of the present application implemented in a driver chip, such as Figure 6As shown, the sensors may include: IMU, ToF and Camera. For example, a three-axis accelerometer can be used to detect the pitch and roll angles of the projection optical axis, two single-point ToFs can detect the yaw angle, and Cemera can detect the pitch and yaw angles. The SLM driver chip starts automatic trapezoidal correction and further displays the video. SLM display chips such as Liquid Crystal On Silicon (LCOS) are responsible for modulating the spatial light to obtain the display image.

[0194] The process of single automatic keystone correction includes the following steps:

[0195] ①a Read: SoC reads the sensor data once to obtain the posture of the projection device. The read sensor data is transmitted to the SLM driver chip through the bus.

[0196] ①b Prediction: The SLM driver chip predicts the posture of the projection device.

[0197] ② Calculation: The SLM driver chip establishes the projection coordinate relationship and calculates the projection rectangle and the corresponding pre-distortion parameters.

[0198] ③ Allocation: The SLM driver chip configures the pre-distortion correction parameters to the GPU.

[0199] ④ Use: Application Specific Integrated Circuit (ASIC) hardware to validate pre-distortion correction parameters, perform pre-distortion transformation on the original image, and obtain a pre-distorted image. The image data stream of the pre-distorted image is transmitted to an SLM display chip such as LCOS through a bus (for example, a serial bus such as a serial peripheral interface (SPI)), thereby displaying a rectangular projection screen.

[0200] In the above steps ①b and ②, pre-distortion correction parameters are obtained through hardware prediction, and the "use" of step ④ occurs in the SLM driver chip to perform hardware trapezoidal correction, which has high real-time performance and can improve the efficiency of trapezoidal correction. In addition, hardware processing can save GPU resources compared with software processing.

[0201] In addition, compared to Figure 3 The illustrated embodiment eliminates the bus communication between the MCU / SOC and the SLM driver chip, thereby reducing transmission time.

[0202] Several specific embodiments are used below to explain the technical solutions of the embodiments of the present application in detail.

[0203] Figure 7 Schematic diagram of interpolation calculation in an embodiment of the present application, such as Figure 7As shown, when the sensor frame rate is low (for example, some ToF frame rates <30Hz) or multiple sampling and filtering processes are required to obtain a usable posture of the projection device, the realization frame rate of automatic trapezoidal correction becomes lower, resulting in the inability to accurately correct the projection device in real time when the user moves the projection device. In this embodiment, one or more device postures predicted based on historical data / historical postures can be inserted between two trapezoidal corrections using measured sensor data, so that more trapezoidal corrections can effectively improve the realization frame rate of automatic trapezoidal correction.

[0204] For example, the projection device predicts the posture once between reading the sensor data for the Nth time and the N+1th time, and predicts the posture once between reading the sensor data for the N+1th time and the N+2th time. In this way, the postures of the projection device increase from 3 to 5. More postures can match more images, thereby effectively improving the frame rate of automatic keystone correction.

[0205] Figure 8 Schematic diagram of posture pre-compensation in an embodiment of the present application, such as Figure 8 As shown, Figure 3 In the prior art solution shown, the sensor data read at time t1 is not effective until time t4, which is a delay. When the data is actually used at time t4, the posture of the projection device may have changed. This embodiment obtains the posture at time t4 in advance by means of posture prediction, and then performs trapezoidal correction based on it, completes posture pre-compensation, and ensures that the picture is always square.

[0206] Fig. 9 A schematic diagram of parallel computing of an embodiment of the present application is shown in FIG. Fig. 9 As shown, the prior art solution adopts a serial processing flow of reading, calculating, matching and using. The final parameter validation stage in the trapezoidal correction processing flow is implemented by an image processing unit hardened by a GPU or ASIC. The software can synchronously start calculation / data reading when the image processing unit performs pre-distortion transformation. The pipeline operation saves software processing time and may achieve a higher processing frame rate.

[0207] Fig.10 Schematic diagram of the structure of the projection device 1000 according to the embodiment of the present application. Fig.10 As shown, the projection device 1000 of this embodiment can be applied to the above-mentioned projection equipment, such as a projector, etc. The projection device 1000 may include: an acquisition module 1001, a correction module 1002, a delivery module 1003 and a prediction module 1004.

[0208] The acquisition module 1001 is used to acquire a first pre-distortion correction parameter, which is obtained in advance based on a first predicted posture of the projection device; the correction module 1002 is used to perform a pre-distortion correction on the current image according to the first pre-distortion correction parameter to obtain a first target image; the projection module 1003 is used to project the first target image onto a target plane.

[0209] In a possible implementation, the prediction module 1004 is used to obtain a historical posture, where the historical posture is obtained based on previously collected sensor data; predict based on the historical posture to obtain the first predicted posture; and obtain the first pre-distortion correction parameter based on the first predicted posture.

[0210] In a possible implementation, the prediction module 1004 is specifically configured to predict based on the historical posture to obtain at least one predicted posture corresponding to at least one moment; and determine the predicted posture corresponding to the at least one moment closest to the current moment as the first predicted posture.

[0211] In a possible implementation, the prediction module 1004 is specifically configured to input the historical posture into a machine learning model to obtain the at least one predicted posture.

[0212] In a possible implementation, the prediction module 1004 is specifically configured to perform particle filtering processing according to the historical posture to obtain the at least one predicted posture.

[0213] In a possible implementation, the acquisition module 1001 is further used to acquire sensor data collected twice adjacently; when the similarity of the sensor data collected twice adjacently is greater than a preset threshold, a second pre-distortion correction parameter is acquired, and the second pre-distortion correction parameter is the pre-distortion correction parameter used last time; the correction module 1002 is further used to perform pre-distortion correction on the current image according to the second pre-distortion correction parameter to obtain a second target image; the delivery module 1003 is further used to project the second target image onto a target plane.

[0214] In a possible implementation, the acquisition module 1001 is further used to acquire a third pre-distortion correction parameter, where the third pre-distortion correction parameter is obtained based on sensor data collected in real time; the correction module 1002 is further used to perform pre-distortion correction on the current image based on the third pre-distortion correction parameter to obtain a third target image; and the delivery module 1003 is further used to project the third target image onto a target plane.

[0215] The device of this embodiment can be used to perform Figure 5The technical solution of the method embodiment shown has similar implementation principles and technical effects, which will not be repeated here.

[0216] In the implementation process, each step of the above method embodiment can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software. The processor can be a general-purpose processor, a digital signal processor (digital signal processor, DSP), an application-specific integrated circuit (application-specific integrated circuit, ASIC), a field programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware coding processor to be executed, or the hardware and software modules in the coding processor are combined to be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0217] The memory mentioned in the above embodiments may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0218] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0219] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0220] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0221] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0222] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0223] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0224] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A projection method, characterized in that: include: Acquire a first pre-distortion correction parameter, where the first pre-distortion correction parameter is obtained in advance based on a first predicted posture of the projection device; Performing pre-distortion correction on the current image according to the first pre-distortion correction parameter to obtain a first target image; The first target image is projected onto a target plane.

2. The method according to claim 1, characterized in that Also includes: Acquire a historical posture, where the historical posture is obtained based on previously collected sensor data; Performing prediction based on the historical posture to obtain the first predicted posture; The first pre-distortion correction parameter is acquired according to the first predicted posture.

3. The method according to claim 2, characterized in that The step of performing prediction according to the historical posture to obtain the first predicted posture includes: Predicting according to the historical posture to obtain at least one predicted posture corresponding to at least one moment; The predicted posture corresponding to the at least one time point closest to the current time point is determined as the first predicted posture.

4. The method according to claim 3, characterized in that: The step of predicting according to the historical posture to obtain at least one predicted posture corresponding to at least one moment includes: The historical gestures are input into a machine learning model to obtain the at least one predicted gesture.

5. The method according to claim 3, characterized in that: The step of predicting according to the historical posture to obtain at least one predicted posture corresponding to at least one moment includes: Particle filtering is performed according to the historical posture to obtain the at least one predicted posture.

6. The method according to any one of claims 1 to 5, characterized in that Also includes: Get sensor data collected twice consecutively; When the similarity of the sensor data collected twice adjacently is greater than a preset threshold, obtaining a second pre-distortion correction parameter, where the second pre-distortion correction parameter is the pre-distortion correction parameter used last time; Performing pre-distortion correction on the current image according to the second pre-distortion correction parameter to obtain a second target image; The second target image is projected onto a target plane.

7. The method according to any one of claims 1 to 6, characterized in that Also includes: Acquiring a third pre-distortion correction parameter, wherein the third pre-distortion correction parameter is obtained according to sensor data collected in real time; Performing pre-distortion correction on the current image according to the third pre-distortion correction parameter to obtain a third target image; The third target image is projected onto a target plane.

8. A projection device, characterized in that: include: An acquisition module, used to acquire a first pre-distortion correction parameter, where the first pre-distortion correction parameter is obtained in advance based on a first predicted posture of the projection device; A correction module, configured to perform pre-distortion correction on the current image according to the first pre-distortion correction parameter to obtain a first target image; A projection module is used to project the first target image onto a target plane.

9. The device according to claim 8, characterized in that Also includes: A prediction module is used to obtain a historical posture, where the historical posture is obtained based on previously collected sensor data; predict according to the historical posture to obtain the first predicted posture; and obtain the first pre-distortion correction parameter according to the first predicted posture.

10. The device according to claim 9, characterized in that The prediction module is specifically configured to perform prediction based on the historical posture to obtain at least one predicted posture corresponding to at least one moment; and determine the predicted posture corresponding to the at least one moment closest to the current moment as the first predicted posture.

11. The device according to claim 10, characterized in that The prediction module is specifically used to input the historical posture into a machine learning model to obtain the at least one predicted posture.

12. The device according to claim 10, characterized in that The prediction module is specifically configured to perform particle filtering processing according to the historical posture to obtain the at least one predicted posture.

13. The device according to any one of claims 8 to 12, characterized in that The acquisition module is further used to acquire sensor data collected twice adjacently; when the similarity of the sensor data collected twice adjacently is greater than a preset threshold, acquire a second pre-distortion correction parameter, where the second pre-distortion correction parameter is the pre-distortion correction parameter used last time; The correction module is further used to perform pre-distortion correction on the current image according to the second pre-distortion correction parameter to obtain a second target image; The projection module is further used to project the second target image onto a target plane.

14. The device according to any one of claims 8 to 13, characterized in that The acquisition module is further used to acquire a third pre-distortion correction parameter, wherein the third pre-distortion correction parameter is obtained based on sensor data collected in real time; The correction module is further used to perform pre-distortion correction on the current image according to the third pre-distortion correction parameter to obtain a third target image; The projection module is further used to project the third target image onto a target plane.

15. A projection device, characterized in that: include: A sensor and a spatial light modulator SLM driver chip; the sensor and the SLM driver chip are connected via a bus; The sensor is used to obtain corresponding sensor data according to the posture of the projection device; The SLM driver chip is used to implement the method according to any one of claims 1-7.

16. A computer-readable storage medium, characterized in that: The invention comprises a computer program, which, when executed on a computer, causes the computer to execute the method according to any one of claims 1 to 7.

17. A computer program product, characterized in that The computer program product comprises a computer program code, and when the computer program code is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 7.

Images