Projection control method and projection equipment

Abstract

The invention relates to a projection control method and projection equipment, and the method comprises the steps: obtaining a first environment feature of a current position of the projection equipment when a recovery instruction for a memory position of the projection equipment is received; based on matching between the first environment feature and second environment features of the memory position, determining a target environment feature matched with the first environment feature in the second environment features; wherein the first environment feature and the second environment feature are generated based on multi-sensor data collected from corresponding projection environments; determining adjustment information of the projection equipment based on the first environment feature and the target environment feature; and driving the projection equipment to rotate in the space according to the adjustment information, so that a projection picture of the projection equipment is projected to a memory position corresponding to the target environment feature. By using the method and the device, the projection picture of the projection equipment can be accurately recovered after the position of the projection equipment is changed.

Description

Technical Field

[0001] The present application relates to the field of projection technology, and in particular, to a projection control method and a projection device. Background Art

[0002] With the rapid development of display technology, projection equipment is increasingly widely used. Projection equipment is a device that can project images or videos onto a projection surface (such as a screen or wall), and is currently widely used in homes, offices, schools, and entertainment venues.

[0003] When the position of the projection device changes, the user is often required to manually adjust the focal length, angle and position of the projection device in order to map the projection image to the original projection position. This manual adjustment method is not only difficult to accurately control the projection position and angle, but also time-consuming and costly, greatly affecting the user experience. Summary of the invention

[0004] The present application discloses a projection control method and a projection device, which can solve the technical problem that it is difficult to accurately restore the projection image of the projection device.

[0005] On the one hand, the present application provides a projection control method, which includes: when receiving a recovery instruction for a memory position of a projection device, obtaining a first environmental feature of the current position of the projection device, and determining a target environmental feature in the second environmental feature that matches the first environmental feature based on a match between the first environmental feature and a second environmental feature of the memory position; wherein the first environmental feature and the second environmental feature are generated based on multi-sensor data collected for a corresponding projection environment, adjustment information of the projection device is determined based on the first environmental feature and the target environmental feature, and the projection device is driven to rotate in space according to the adjustment information, so that a projection picture of the projection device is projected to the memory position corresponding to the target environmental feature.

[0006] In some embodiments of the present application, the memory location includes corresponding playback information, and the playback information includes one or more of playback content, playback type, and playback settings.

[0007] In some embodiments of the present application, the method for generating the first environmental feature includes: fusing multi-sensor data corresponding to the current position to obtain a target image, determining feature points in the target image and descriptors of the feature points based on detection of the target image, and determining the first environmental feature based on the multi-sensor data and the descriptors of the feature points.

[0008] In some embodiments of the present application, the multi-sensor data corresponding to the current position includes image information, depth information and illumination information of the projection environment, and the fusing of the multi-sensor data corresponding to the current position to obtain the target image includes: aligning the image information with the depth information, determining the depth information corresponding to each pixel in the image information, generating an initial image based on the texture information corresponding to each pixel in the image information and the depth information corresponding to each pixel, adjusting the visual parameters of the initial image based on the illumination information, and obtaining the target image.

[0009] In some embodiments of the present application, determining the feature points of the target image and the descriptors of the feature points based on the detection of the target image includes: performing edge detection on the target image to obtain an edge image, determining the feature points from the pixels based on the pixel values ​​of the pixels in the edge image, dividing the neighborhood corresponding to the feature points in the edge image into multiple grids, generating a gradient histogram corresponding to each grid based on the gradient information of each grid, determining the gradient amplitude vector corresponding to each grid based on the target gradient amplitude of each gradient direction in the gradient histogram, and aggregating the gradient amplitude vectors corresponding to the multiple grids to obtain the descriptors of the feature points.

[0010] In some embodiments of the present application, determining the adjustment information of the projection device based on the first environmental feature and the target environmental feature includes: determining multiple groups of initial feature point pairs by matching the descriptors in the first environmental feature and the target environmental feature, mapping the depth information of each group of initial feature point pairs to a first preset coordinate system, obtaining updated depth information of each group of initial feature point pairs, selecting target feature point pairs from the multiple groups of initial feature point pairs based on the updated depth information, the target feature point pairs including a first target feature point and a second target feature point, aligning the first target feature point with the second target feature point, obtaining a perspective transformation matrix, and determining the adjustment information based on the perspective transformation matrix.

[0011] In some embodiments of the present application, the adjustment information includes posture information, and driving the projection device to rotate in space according to the adjustment information includes: controlling the projection device to rotate in space using a pan / tilt platform of the projection device according to the posture information.

[0012] In some embodiments of the present application, the adjustment information includes correction information, and the method further includes: correcting the projection image according to the correction information, wherein the correction of the projection image includes image scaling and trapezoidal correction.

[0013] In some embodiments of the present application, the method further includes: if it is detected that the current projection environment has changed, adjusting the projection parameters of the projection device.

[0014] On the other hand, the present disclosure also provides a projection device, comprising: a processor; a memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the projection control method of the first aspect above.

[0015] In the projection control method provided in the embodiment of the present application, since the multi-sensor data can fully reflect the characteristics of the projection environment, by matching the first environmental characteristics with the environmental characteristics of the memory position, the target environmental characteristics of the memory position that matches the first environmental characteristics can be accurately determined. Through the first environmental characteristics and the target environmental characteristics, the adjustment information for the projection device can be accurately determined, and the projection device can be driven to rotate in space through the accurate adjustment information, so that the projection screen can be accurately projected to the matching memory position, so that the projection screen of the projection device can be accurately restored after the position of the projection device changes. Since the restoration of the projection screen does not require the user to perform complex manual adjustments, the restoration of the projection screen is intelligent and simple, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 It is an application scenario diagram of the projection control method provided in an embodiment of the present application.

[0018] Figure 2 It is a flowchart of a method for generating a target image provided by an embodiment of the present application.

[0019] Figure 3 It is a flowchart of a method for generating a descriptor provided in another embodiment of the present application.

[0020] Figure 4 It is a flowchart of a method for determining adjustment information provided by an embodiment of the present application.

[0021] Figure 5 It is a schematic diagram of the structure of the projection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0023] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0024] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0025] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0026] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0027] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0028] In the related art, when the position of the projection device changes (for example, moves to a new position), if you want to restore the projection image at the original projection position, it is often achieved by the following methods:

[0029] The first method relies on the user to perform a series of complex adjustments on the projection device to restore the projection image, such as adjusting the position and angle of the projection device, in order to restore the original projection image. This adjustment process is not only cumbersome and time-consuming, but also prone to errors, affecting the user experience.

[0030] The second method relies on sensors (gyroscopes and accelerometers, etc.) installed in a fixed position or in the projection device to restore the projection image. However, this method cannot cope with complex environmental changes and it is difficult to accurately restore the projection image to the original projection position.

[0031] In order to solve the above problems, an embodiment of the present application provides a projection control method, which can accurately restore the projection image of the projection device after the position of the projection device changes.

[0032] In this embodiment, the projection control method can be applied to one or more projection devices. For projection devices that require projection control, the function for projection control provided by the method of the present application can be directly integrated on the projection device, or run on the projection device in the form of a software development kit (SDK).

[0033] like Figure 1 , which is a flowchart of a projection control method provided by an embodiment of the present application. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted. The projection control method is applied to a projection device. For example, Figure 5 Projection equipment in.

[0034] S11, when receiving a recovery instruction for the memory position of the projection device, obtaining a first environmental feature of the current position of the projection device.

[0035] In some embodiments of the present application, the projection device can be integrated with the pan-tilt platform. For example, the projection device can be located on the pan-tilt platform. The pan-tilt platform can have multiple degrees of freedom, so that the pan-tilt platform can be adjusted in multiple directions.

[0036] In some embodiments of the present application, since the projection position of the projection image is determined by the posture information of the projection device, the posture information of the projection device can be used to represent the memory position.

[0037] In some embodiments of the present application, the memory position can be obtained through multiple adjustments. For example, the initial posture information of the projection device is recorded as A, and the projection device can receive the user's adjustment operation on the initial posture information A, so that the projection picture is projected onto the projection surface desired by the user (for example, a wall or a curtain, etc.), and the posture information obtained by adjusting the initial posture information A is recorded as B. In order to correct the projection picture, the projection device can adjust the posture information B, and the posture information obtained by adjusting the posture information B is recorded as C. In response to the user's save operation, the projection device can use the posture information C as a memory position.

[0038] In other embodiments of the present application, the memory position may also correspond to the playback information, for example, a corresponding relationship between the memory position and the playback information is set, and the playback information includes one or more of the playback content, the playback type, and the playback setting. The playback type may be an image, video, audio, playback channel, etc., the playback content may be the content corresponding to the playback type, and the playback setting may be a playback rule such as the playback time and playback order.

[0039] In some embodiments of the present application, there may be multiple memory locations, and they are set according to different usage scenarios and needs of users. For example, in some home usage scenarios of the projection device, the living room, bedroom, and study may each have a memory location.

[0040] When there are multiple memory positions, each memory position has a corresponding second environmental feature. The second environmental feature corresponding to each memory position can be generated by the multi-sensor data corresponding to each memory position, and the multi-sensor data corresponding to each memory position can be obtained by the projection device collecting the projection environment through multiple sensors at the memory position. For the description of the multi-sensor data corresponding to each memory position, please refer to the multi-sensor data corresponding to the current position below. The method for generating the second environmental feature corresponding to each memory position according to the multi-sensor data of each memory position can refer to the description of the method for generating the first environmental feature below, and this application will not repeat the description.

[0041] In some embodiments of the present application, the recovery instruction can be triggered in a variety of ways. For example, the user can send the recovery instruction to the projection device through a remote control or other device, or directly input the corresponding recovery instruction through any physical button or virtual control set in the projection device.

[0042] In some embodiments of the present application, the first environmental feature may be generated based on multi-sensor data collected from the projection environment of the current location. The multi-sensor data may include data collected by the sensor of the projection device from the projection environment and / or data processed from the data collected by the sensor.

[0043] The sensors include but are not limited to: visual sensors, depth sensors, optical sensors, inertial measurement units (IMU), etc. For example, the visual sensors may be cameras and the depth sensors may be time of flight (ToF) sensors and structured light sensors.

[0044] The multi-sensor data may include, but is not limited to, image information of the projection surface (e.g., a curtain or a wall) in the projection environment, depth information of the projection environment, illumination information, position information of the projection device, and position information of the ladder calibration points on the projection screen. The image information of the projection surface may be obtained by photographing the projection surface through a visual sensor. The texture information of the projection surface may include, but is not limited to, color, pattern shape, and roughness. The depth information of the projection environment may constitute a three-dimensional model of the projection environment. The illumination information may be obtained through a light sensor, and the illumination information may include brightness and light intensity, etc. The position information of the projection device may be obtained through an inertial measurement unit, and the position information of the projection device may include position information, attitude information, acceleration information, and angular velocity information, etc. The position information may be the height of the projection device and the distance between the projection device and the projection surface, etc., and the attitude information may be the yaw angle (Yaw), pitch angle (Pitch), and roll angle (Roll) of the projection device, etc. The position information of the ladder calibration points on the projection screen may be obtained by recognizing the image information of the projection surface collected by the visual sensor through an image recognition algorithm, wherein the image recognition algorithm may be a convolutional neural network and a target detection algorithm, etc., and the present application does not limit the image recognition method. The ladder calibration points on the projection screen may be multiple corner points of the projection screen, the corner points of the projection screen may be vertices, and the position information of the ladder calibration points may be the coordinates of the corner points. The position information of the ladder calibration points may be in the optomechanical coordinate system of the projection device. The optomechanical coordinate system may be a coordinate system corresponding to the optical system of the projection device, and the optical system may include one or more optical components, such as a lens, a reflector, etc. Exemplarily, the optomechanical coordinate system may take the center point of the lens as the origin, the width direction of the projection screen as the X-axis, the height direction of the projection screen as the Y-axis, and the direction perpendicular to the projection screen as the Z-axis.

[0045] In some embodiments of the present application, before generating the first environmental feature, the multi-sensor data may be preprocessed. Exemplarily, considering that the image information collected by the visual sensor may have problems such as noise or distortion, the image information collected by the visual sensor may be filtered, distorted, etc., so as to reduce the noise and distortion of the processed image information. Considering that the depth information collected by the depth sensor may be inaccurate due to factors such as reflection and occlusion, the depth information collected by the depth sensor may be filtered and the depth information after filtering may be corrected, so that abnormal values ​​can be eliminated from the forehead to ensure the accuracy of the depth information. Considering that the posture information collected by the inertial measurement unit may correspond to different coordinate systems, the projection device can filter and standardize the posture information collected by the inertial measurement unit, so that the posture data collected at different times can be compared in the same coordinate system. Considering that the illumination information collected by the optical sensor may have dimensional differences and order of magnitude differences, the projection device can normalize the illumination information so that the normalized illumination information can reflect the relative changes in illumination.

[0046] In some embodiments of the present application, the method for generating the first environmental feature includes: the projection device fuses multi-sensor data corresponding to the current position to obtain a target image, determines feature points and descriptors of the feature points in the target image based on the detection of the target image, and determines the first environmental feature based on the multi-sensor data and the descriptors of the feature points.

[0047] Among them, the descriptor is also called a descriptor, which is used to describe the feature point. The descriptor can be in the form of a vector, including the geometry, texture or gradient information of the pixels around the feature point. The projection device can obtain the target image by fusing all or part of the data in the multi-sensor data. Exemplarily, the projection device can obtain the target image by fusing the texture information, depth information and illumination information of the image information in the multi-sensor data.

[0048] Exemplarily, the projection device may determine the descriptor of the feature point and all multi-sensor data corresponding to the current position as the first environmental feature. Alternatively, the projection device may determine the descriptor of the feature point and part of the multi-sensor data corresponding to the current position as the first environmental feature.

[0049] In some embodiments of the present application, the projection device may detect feature points in the target image by a variety of methods.

[0050] Exemplarily, the projection device may perform feature detection on the target image using a feature detection algorithm to determine feature points. The feature points may be corner points, which may be pixels in the target image with drastic brightness changes and significant features, such as pixel points where edges meet. The feature detection algorithm may be a corner detection algorithm, which may include but is not limited to Harris corner detection and FAST (Features from Accelerated Segment Test) algorithms.

[0051] In some embodiments of the present application, after detecting the feature points, the projection device may generate descriptors of the feature points using algorithms such as Scale-Invariant Feature Transform (SIFT) and Speeded-Up Robust Features (SURF).

[0052] In other embodiments of the present application, the projection device may use algorithms such as SIFT and SURF to detect feature points in the target image and generate descriptors of the feature points.

[0053] Exemplarily, a method for generating a descriptor of a feature point includes: for each detected feature point, the projection device can determine a feature area corresponding to each feature point in a target image or in a feature image after feature detection of the target image, input the feature area into a preset neural network model, obtain feature maps output by multiple network layers in the neural network model, perform a dimensionality reduction operation on each feature map to obtain a feature vector, aggregate the feature vectors corresponding to multiple feature maps, and obtain a descriptor corresponding to each feature point.

[0054] Among them, the feature area corresponding to each feature point in the target image or in the feature image after feature detection of the target image can be an area of ​​a preset size centered on each feature point in the target image or the feature image, wherein the preset size can be customized and the present application does not impose any restrictions on this.

[0055] Neural network models include but are not limited to: ResNet, VGG or MobileNet. The neural network model may include multiple convolutional layers and pooling layers, which can extract features at different levels to obtain multiple feature maps, and the multiple feature maps may include information such as shape, texture, color and depth information. By performing pooling processing on each of the feature maps, dimensionality reduction of each of the feature maps can be achieved. The pooling processing can be maximum pooling or average pooling, and this application does not limit the pooling type.

[0056] The projection device can use techniques such as average pooling, maximum pooling or attention mechanism to aggregate the feature vectors corresponding to multiple feature maps, so as to obtain the descriptor corresponding to each feature point.

[0057] Considering that a single sensor data cannot cope with complex environmental changes and it is difficult to accurately restore the projection image to the memory position, in this embodiment, since multi-sensor data can comprehensively reflect the characteristics of the projection environment, the first environmental feature and the second environmental feature are generated by multi-sensor data, which can make the first environmental feature and the second environmental feature accurate, distinguishable and stable, and can adapt to complex projection environments and cope with changes in projection environments. Since the neural network model can generate multi-level and multi-scale descriptors, the descriptors can be made robust.

[0058] In some embodiments, the projection device may optimize the generated descriptor to obtain an optimized descriptor.

[0059] Exemplarily, the projection device can calculate the loss of the neural network model based on the descriptors of the generated feature points, adjust the network parameters of the neural network model according to the loss until convergence, and determine the descriptor output when the neural network model converges as the optimized descriptor.

[0060] The loss may be a triplet loss, a contrastive loss, etc. The network parameters may be a learning rate, a weight, a bias, etc. The conditions for determining the convergence of the neural network model may be customized. For example, the convergence condition may be that the gap between multiple consecutive loss values ​​is within a preset range.

[0061] In this embodiment, the network parameters of the neural network model are adjusted through loss, so that in the descriptors generated by the neural network model, the distances between similar descriptors are closer and the distances between dissimilar descriptors are farther, thereby being able to distinguish between similar descriptors and dissimilar descriptors.

[0062] S12, based on the match between the first environmental feature and the second environmental feature of the memory location, determine a target environmental feature in the second environmental feature that matches the first environmental feature.

[0063] In some embodiments of the present application, the first environmental feature and the second environmental feature corresponding to each memory position both include descriptors of feature points.

[0064] In some embodiments of the present application, the projection device can use a variety of methods to match the first environmental feature with the second environmental feature corresponding to each memory location, so that the target environmental feature that matches the first environmental feature can be determined in the second environmental feature. The present application does not limit the matching method.

[0065] Exemplarily, the projection device may calculate the similarity between the first environmental feature and the second environmental feature corresponding to each memory position, and determine the second environmental feature corresponding to the greatest similarity as the target environmental feature. The projection device may determine the descriptors that match between the descriptor of the first environmental feature and the descriptor of each second environmental feature, and the more the number of matching descriptors in the second environmental feature is, the more similar the second environmental feature is to the first environmental feature.

[0066] The above-mentioned example of the method of matching the first environmental feature with the second environmental feature corresponding to each memory position is only an example and is not limited to this in actual application.

[0067] In this embodiment, since the first environmental feature and the second environmental feature are accurate, distinguishable and stable, the target environmental feature can be accurately determined by calculating the similarity between the first environmental feature and the second environmental feature of each memory location, and determining the second environmental feature corresponding to the maximum similarity as the target environmental feature.

[0068] S13: Determine adjustment information of the projection device based on the first environment feature and the target environment feature.

[0069] In some embodiments of the present application, the projection device determines adjustment information of the projection device based on the first environmental feature and the target environmental feature, including: determining multiple groups of initial feature point pairs by matching descriptors in the first environmental feature and the target environmental feature; mapping the depth information of each group of initial feature point pairs to a first preset coordinate system, obtaining updated depth information of each group of initial feature point pairs, selecting target feature point pairs from the multiple groups of initial feature point pairs based on the updated depth information, the target feature point pairs including a first target feature point and a second target feature point, aligning the first target feature point with the second target feature point, obtaining a perspective transformation matrix, and determining adjustment information based on the perspective transformation matrix.

[0070] Among them, the projection device can match the descriptors in the first environmental feature and the target environmental feature through a variety of methods, so as to determine a plurality of groups of initial feature point pairs. The present application does not limit the matching method.

[0071] Exemplarily, the projection device may calculate the distance (e.g., Euclidean distance and Hamming distance) between the descriptor of the feature point in the first environmental feature and the descriptor of the feature point in the target environmental feature, and determine the feature points corresponding to the descriptors whose distances are in a preset interval as the initial feature point pair. The preset interval may be customized, and in order to ensure the matching degree of the initial feature point pair, the preset interval may be set to a smaller value, such as [0, 1].

[0072] In this embodiment, by determining the feature points corresponding to the descriptors whose distances are within a preset interval as the initial feature point pairs, the initial feature point pairs can be quickly determined.

[0073] Exemplarily, the projection device may determine a matching feature point pair from the feature points of the first environmental feature and the feature points of the second environmental feature using a nearest neighbor search algorithm (NNS) based on the descriptors of the feature points in the first environmental feature and the descriptors of the feature points in the target environmental feature, and use the matching feature point pair as the initial feature point pair. The nearest neighbor search algorithm may be a Fast Library for Approximate Nearest Neighbors (FLANN) or a K-dimensional tree (KD-Tree).

[0074] In this embodiment, since the nearest neighbor search algorithm is highly efficient, using the nearest neighbor search algorithm to determine the initial feature point pairs can improve the efficiency of determining the initial feature point pairs.

[0075] For example, the projection device can utilize bidirectional matching (Bidirectional Matching g ) method or random sampling consensus algorithm (RANSAC) to determine the matching feature point pairs from the feature points of the first environmental feature and the feature points of the second environmental feature, and use the matching feature point pairs as the initial feature point pairs. Among them, two feature points that meet or satisfy the two-way matching rule can be successfully matched in both the forward and reverse directions. By using RANSAC for optimization, the feature point pairs with incorrect matching in the initial feature point pairs can be reduced, and the accuracy of the initial feature point pairs can be improved.

[0076] In this embodiment, since two feature points that meet or satisfy the bidirectional matching rule can be successfully matched in both the forward and reverse directions, the reliability of the initial feature point pair can be improved by determining the initial feature point pair using the bidirectional matching method.

[0077] In other embodiments of the present application, the projection device can use a deep neural network to match the descriptors in the first environmental feature and the target environmental feature, so as to determine the matched feature points in the feature points of the first environmental feature and the second environmental feature, and use the matched feature point pairs as the initial feature point pairs. Among them, the deep neural network can include but is not limited to: convolutional neural networks such as ResNet, VGG and Inception, Siamese Network, Attention Mechanism, and neural networks trained based on contrastive learning.

[0078] In other embodiments of the present application, the projection device can match the image information in the first environmental feature with the image information in the second environmental feature, and match the descriptor in the first environmental feature with the descriptor in the second environmental feature, so as to determine that the feature points of the first environmental feature and the feature points of the second environmental feature that match each other are the initial feature point pairs.

[0079] Exemplarily, the projection device can match the first environmental feature with the target environmental feature through a global feature matching algorithm, thereby determining a matching feature point pair among the feature points of the first environmental feature and the feature points of the second environmental feature, and using the matching feature point pair as an initial feature point pair. The global feature matching algorithm includes, but is not limited to: Image Pyramid Matching, Optical Flow Matching, and Global Descriptor Matching.

[0080] In this embodiment, by matching the image information and descriptors in the first environmental feature with those in the second environmental feature, the stability and accuracy of the matching can be ensured.

[0081] In some embodiments of the present application, the first preset coordinate system may be a coordinate system corresponding to a visual sensor or a coordinate system corresponding to a depth sensor. An initial feature point pair whose gap between updated depth information in a plurality of sets of initial feature point pairs is within a preset range may be determined as a target feature point pair, wherein the preset range may be customizable and the present application does not impose any restrictions thereon. The dimension of the perspective transformation matrix may be 4*4, including a 3*3 first rotation matrix and a 3*1 translation vector.

[0082] Since the first target feature point and the second target feature point have corresponding depth information, the first target feature point and the second target feature point can be regarded as a point cloud in space.

[0083] Exemplarily, the projection device may align the first target feature point with the second target feature point using iterative closest point (ICP) to obtain a perspective transformation matrix. During the alignment process, the relationship between the depth information of the first target feature point and the second target feature point may be converted into a least squares problem. The goal of the least squares problem may be to minimize the distance or error between the first target feature point and the second target feature point. By solving the least squares problem using singular value decomposition (SVD), the perspective transformation matrix that aligns the first target feature point to the second target feature point may be obtained.

[0084] The above-mentioned example of the method of aligning the first target feature point with the second target feature point is only an example and is not limited thereto in practical applications.

[0085] In this embodiment, the perspective transformation matrix is ​​calculated by using the nearest neighbor iterative algorithm and the singular value decomposition algorithm, so that the accuracy of the perspective transformation matrix can be ensured.

[0086] In some embodiments of the present application, the projection device may determine the adjustment information according to the perspective transformation matrix, the first environment feature and the second environment feature. The method for determining the adjustment information will be described in detail below.

[0087] The adjustment information may include posture information corresponding to the projection device and / or correction information corresponding to the projection screen. The posture information may be a target Euler angle, and the target Euler angle includes one or more of a yaw angle, a pitch angle, and a roll angle. The posture of the projection device may be adjusted by the posture information. Exemplarily, the pitch angle in the posture information may be used to control the up and down tilt angle of the projection device, and / or, the yaw angle in the posture information may be used to control the left and right rotation angle of the projection device, and / or, the roll angle in the posture information may be used to control the left and right flip angle of the projection device.

[0088] The correction information may include target gradient calibration coordinates of multiple corner points in the projection picture, and the correction information may be used to calibrate the projection picture of the projection device.

[0089] In this embodiment, since the perspective transformation matrix is ​​accurate, the first environmental features and the second environmental features are accurate, distinguishable and stable, therefore, based on the perspective transformation matrix, the first environmental features and the second environmental features, the accuracy of the posture information of the projection device and the correction information of the projection image can be ensured.

[0090] S14, driving the projection device to rotate in space according to the adjustment information, so that the projection image of the projection device is projected to the memory position corresponding to the target environment feature.

[0091] In some embodiments of the present application, taking controlling the gimbal according to posture information as an example, the projection device can convert the posture information into control instructions for the gimbal, and control the motor of the gimbal according to the control instructions, thereby driving the projection device to rotate in space, so that the projection image is projected to the memory position corresponding to the target environment feature.

[0092] The control instructions may include parameters such as the rotation direction, angle and speed of the gimbal.

[0093] In this embodiment, the pan / tilt head is controlled to rotate by accurate posture information, thereby driving the projection device to rotate in space, and the projection image can be accurately restored to the memory position corresponding to the target environment feature.

[0094] In some embodiments of the present application, the projection device may correct the projection image according to the correction information, wherein the correction of the projection image includes image scaling and trapezoidal correction.

[0095] The coordinates of the corner points in the projection picture can be converted into corresponding target gradient coordinates in the correction information, thereby realizing trapezoidal correction and picture scaling of the projection picture.

[0096] In this embodiment, by performing image scaling and trapezoidal correction on the projection image, trapezoidal distortion can be eliminated to provide a better visual effect.

[0097] In other embodiments of the present application, if a change in the current projection environment is detected, such as a change in illumination, a change in wall texture, etc., the projection parameters of the projection device may be adjusted. The projection parameters may include, but are not limited to, posture information, calibration information, and the like.

[0098] In this embodiment, the projection parameters are adjusted in real time according to the change of the projection environment, so as to ensure the stability and clarity of the projection picture.

[0099] In other embodiments of the present application, if the memory position includes corresponding playback content, the projection device may determine unplayed content in the playback content, and play or display the unplayed content in the projection screen.

[0100] In the projection control method provided in the embodiment of the present application, since the multi-sensor data can fully reflect the characteristics of the projection environment, by matching the first environmental characteristics with the environmental characteristics of the memory position, the target environmental characteristics of the memory position that matches the first environmental characteristics can be accurately determined. Through the first environmental characteristics and the target environmental characteristics, the adjustment information for the projection device can be accurately determined, and the projection device can be driven to rotate in space through the accurate adjustment information, so that the projection screen can be accurately projected to the matching memory position, so that the projection screen of the projection device can be accurately restored after the position of the projection device changes. Since the restoration of the projection screen does not require the user to perform complex manual adjustments, the restoration of the projection screen is intelligent and simple, thereby improving the user experience.

[0101] like Figure 2 , which is a flow chart of a method for generating a target image provided in one embodiment of the present application.

[0102] S111, registering the image information and the depth information to determine the depth information corresponding to each pixel in the image information.

[0103] In some embodiments of the present application, the projection device can use a nearest neighbor matching method such as the Fast Library for Approximate Nearest Neighbors (FLANN) or a k-dimensional tree (KD-Tree) to align image information and depth information, thereby being able to quickly determine the depth information corresponding to each pixel.

[0104] S112, generating an initial image according to the texture information corresponding to each pixel point in the image information and the depth information corresponding to each pixel point.

[0105] In some embodiments of the present application, the projection device may associate the texture information of each pixel with the corresponding depth information, thereby generating a three-dimensional initial image.

[0106] The texture information may be a pixel value of a pixel point. For example, if the image information is in color, the pixel value may be an RGB value. By fusing the RGB information and the depth information, an initial image in RGB-D format may be obtained.

[0107] In this embodiment, by fusing texture information and depth information, the amount of information in the initial image can be increased, providing a basis for generating comprehensive, stable and accurate first environmental features and second environmental features.

[0108] S113, adjusting the visual parameters of the initial image according to the illumination information to obtain a target image.

[0109] In some embodiments of the present application, the visual parameters may be parameters such as brightness, contrast, and color saturation.

[0110] In this embodiment, the visual parameters of the initial image are adjusted according to the illumination information, so that a clear and natural target image can be obtained.

[0111] like Figure 3 FIG. 1 is a flowchart of a method for generating a descriptor provided in another embodiment of the present application, comprising the following steps:

[0112] S114, performing edge detection on the target image to obtain an edge image.

[0113] In some embodiments of the present application, the projection device may perform edge detection on the target image using an edge detection algorithm, wherein the edge detection algorithm includes but is not limited to: Canny edge detection, Sobel operator, and the like.

[0114] In this embodiment, by performing edge detection on the target image, a binary edge image can be obtained. In the edge image, a pixel point whose pixel value is a preset value (for example, 255) represents an edge.

[0115] S115, determining feature points from the pixels according to the pixel values ​​of the pixels in the edge image.

[0116] In some embodiments of the present application, the projection device may determine feature points from the pixels using a feature detection algorithm based on pixel values ​​of the pixels in the edge image.

[0117] The feature point may be a corner point, which may be a pixel point with a sharp change in brightness and a significant feature in the target image, such as a pixel point where an edge intersects. The feature detection algorithm may be a corner point detection algorithm, which includes but is not limited to: Harris corner point detection, FAST (Features from Accelerated Segment Test) algorithm.

[0118] The above examples of feature point detection methods are only examples and are not limited to them in practical applications.

[0119] S116, dividing the neighborhood corresponding to the feature point in the edge image into a plurality of grids, and generating a gradient histogram corresponding to each grid according to the gradient information of each grid.

[0120] In some embodiments of the present application, each feature point in the edge image has a corresponding neighborhood, and the size of the neighborhood corresponding to each feature point can be customized. For example, the neighborhood corresponding to each feature point can be a 16*16 area centered on each feature point.

[0121] The present application does not limit the number of grids. For example, following the above embodiment, a 16*16 neighborhood can be divided into 16 grids.

[0122] The gradient histogram corresponding to each feature point can reflect the gradient intensity distribution of each feature point in multiple gradient directions, and the multiple gradient directions can be obtained by dividing a preset angle. For example, the preset angle can be 360°, and each gradient direction has a corresponding angle interval.

[0123] The number of gradient directions obtained by dividing the preset angle can be customized, and this application does not limit this. The gradient information of each pixel point may include the gradient direction and gradient magnitude of each pixel point.

[0124] Determine the angular interval corresponding to the gradient direction of each pixel point, accumulate the gradients of all pixel points corresponding to each angular interval, and obtain the target gradient amplitude of the gradient direction corresponding to each angular interval. The target gradient amplitudes corresponding to the multiple gradient directions can constitute the gradient histogram corresponding to each feature point.

[0125] S117, determining the gradient magnitude vector corresponding to each grid according to the target gradient magnitude of each gradient direction in the gradient histogram.

[0126] In some embodiments of the present application, the target gradient magnitudes of all gradient directions in the gradient histogram corresponding to each grid may be used as elements in the gradient magnitude vector corresponding to each grid, thereby obtaining the gradient magnitude vector corresponding to each grid.

[0127] S118, aggregating the gradient magnitude vectors corresponding to the multiple grids to obtain a descriptor of the feature point.

[0128] In some embodiments of the present application, the projection device may use methods such as average pooling, maximum pooling or attention mechanism to aggregate the gradient amplitude vectors corresponding to the multiple grids, so as to obtain a descriptor corresponding to each feature point.

[0129] In some embodiments of the present application, the position information of the gradient calibration points of the projection screen in the first environmental feature may include multiple first gradient calibration coordinates, and the position information of the gradient calibration points of the projection screen in the second environmental feature may include multiple second gradient calibration coordinates. For example, the multiple first gradient calibration coordinates and the multiple second gradient calibration coordinates may be the three-dimensional space coordinates of four gradient calibration points on the projection screen. Figure 4 FIG. 1 is a flowchart of a method for determining adjustment information provided by an embodiment of the present application, comprising the following steps:

[0130] S131, using the perspective transformation matrix to transform each first-calibration coordinate to obtain the third-calibration coordinate corresponding to each first-calibration coordinate, mapping the third-calibration coordinate to the second preset coordinate system, and obtaining the fourth-calibration coordinate corresponding to each first-calibration coordinate.

[0131] In some embodiments of the present application, the method for generating the third calibration coordinates corresponding to each first calibration coordinate may refer to formula (1):

[0132] p'=R*p+t;(1)

[0133] Among them, p′ represents the third-level calibration coordinates, R represents the rotation matrix in the perspective transformation matrix, p represents the first-level calibration coordinates, and t represents the translation vector in the perspective transformation matrix.

[0134] The second preset coordinate system may be an optomechanical coordinate system.

[0135] In this embodiment, by transforming each first ladder calibration coordinate using a perspective transformation matrix, each first ladder calibration coordinate can be mapped from the second preset coordinate system to the first preset coordinate system to obtain a third ladder calibration coordinate corresponding to each first ladder calibration coordinate. Each third ladder calibration coordinate can be mapped from the first preset coordinate system to the second preset coordinate system using an inverse matrix of the perspective transformation matrix to obtain a fourth ladder calibration coordinate corresponding to each first ladder calibration coordinate.

[0136] S132, transform each fourth gradient-calibrated coordinate according to any second gradient-calibrated coordinate to obtain a first transformation matrix between each fourth gradient-calibrated coordinate and any second gradient-calibrated coordinate.

[0137] In some embodiments of the present application, the method for obtaining the first transformation matrix according to any second gradient coordinate can refer to formula (1) in step S131, and the present application will not repeat the description. The dimension of the first transformation matrix is ​​the same as the dimension of the perspective transformation matrix.

[0138] S133, calculating a second transformation matrix according to the posture information in the first environmental feature and the second environmental feature.

[0139] In some embodiments of the present application, the projection device can determine the pitch-axis gravity change GA1, the yaw-axis change GA2, and the calibration parameters of the gravity sensor (G-sensor) based on the posture information of the first environmental feature and the second environmental feature, and use the inverse kinematics solution method to calculate the pitch-axis gravity change GA1, the yaw-axis change GA2, and the calibration parameters of the gravity sensor (G-sensor) to obtain a second transformation matrix.

[0140] When the pan / tilt of the projection device has multiple degrees of freedom, the second transformation matrix can be multiple. The method for generating the second transformation matrix can refer to the method for solving the inverse kinematics, and this application will not repeat the description. The dimension of each second transformation matrix is ​​the same as the dimension of the perspective transformation matrix.

[0141] S134, determining a third transformation matrix according to each first transformation matrix and the second transformation matrix corresponding to each first calibration coordinate.

[0142] In some embodiments of the present application, a third transformation matrix may be obtained by multiplying each first transformation matrix corresponding to each first gradient coordinate with the second transformation matrix. The dimension of the third transformation matrix is ​​the same as the dimension of the perspective transformation matrix.

[0143] S135, calculating the initial Euler angle corresponding to each of the first calibrated coordinates according to the rotation matrix in each third transformation matrix, and determining the polygon corresponding to each of the first calibrated coordinates in the two-dimensional parameter space according to the multiple initial Euler angles corresponding to each of the first calibrated coordinates.

[0144] In some embodiments of the present application, the projection device can extract the initial Euler angle from the rotation matrix in the third transformation matrix, and the initial Euler angle includes one or more of the yaw angle, the pitch angle, and the roll angle. When the first-step calibration coordinates and the second-step calibration coordinates are both 4, each first-step calibration coordinate can correspond to 4 initial Euler angles, and 4 first-step calibration coordinates correspond to 16 initial Euler angles. The polygon corresponding to the 4 Euler angles corresponding to each first-step calibration coordinate in the two-dimensional plane can be a rectangle.

[0145] S136, determining a two-dimensional coordinate system corresponding to the two-dimensional parameter space, and determining overlapping areas between polygons corresponding to the plurality of first calibration coordinates.

[0146] In some embodiments of the present application, the horizontal axis and the vertical axis of the two-dimensional coordinate system may respectively represent a component of the initial Euler angle. For example, if the initial Euler angle includes a yaw angle and a pitch angle, the horizontal axis of the two-dimensional coordinate system may represent the yaw angle, and the vertical axis may represent the pitch angle.

[0147] S137, determining the target Euler angle according to the coordinates of the target point in the overlapped area in the two-dimensional coordinate system, performing inverse solution on the target Euler angle, and obtaining a fourth transformation matrix.

[0148] In some embodiments of the present application, a point can be randomly selected from the overlapped area as the target point, and the coordinates of the target point can be determined as the target Euler angle. For example, if the horizontal axis of the two-dimensional coordinate system represents the yaw angle and the vertical axis represents the pitch angle, the target point can be the centroid of the overlapped area of ​​multiple polygons, and the coordinates of the centroid can correspond to a yaw angle and a pitch angle, so the yaw angle and the pitch angle corresponding to the centroid can be determined as the target Euler angle.

[0149] The above examples of target point selection methods are only examples and are not limited to these in practical applications.

[0150] According to the target Euler angle, the projection device can determine the rotation matrix in the fourth transformation matrix, determine the preset vector as the translation vector in the fourth rotation matrix, thereby obtaining the fourth rotation matrix, wherein the preset vector can be customized, or the translation vector in the first transformation matrix corresponding to the initial Euler angle that is the same as or similar to the target Euler angle can be used as the translation vector in the fourth transformation matrix, or the projection device can directly use the rotation matrix determined by the target Euler angle as the fourth transformation matrix. The dimension of the fourth transformation matrix is ​​the same as that of the perspective transformation matrix.

[0151] S138, transform the plurality of second gradient calibration coordinates according to a fourth transformation matrix to obtain target gradient calibration coordinates.

[0152] In some embodiments of the present application, the multiple second calibration coordinates are transformed according to the fourth transformation matrix, and the method for obtaining the target calibration coordinates can refer to the method for generating the third calibration coordinates corresponding to each first calibration coordinate, and the present application will not repeat the description.

[0153] S139, determining the target Euler angle and the target ladder calibration coordinate as adjustment information.

[0154] In some embodiments of the present application, the target Euler angles may be used to adjust the posture of the projection device, and the target ladder calibration coordinates may be used to calibrate the projection image of the projection device.

[0155] like Figure 5 , which is a schematic diagram of the structure of the projection device provided in an embodiment of the present application.

[0156] The projection device 200 includes a projection unit 210 and a driving unit 220 driving the projection unit 210. The projection unit 210 may form an optical image and project the optical image onto an imaging medium SC.

[0157] The projection unit 210 includes a light source unit 211 , a light modulator 212 , and an optical system 213 . The driving unit 220 includes a light source driving unit 221 and a light modulator driving unit 222 .

[0158] The light source unit 211 may include a solid light source such as a light emitting diode (LED), a laser, a pump lamp, etc. The light source unit 211 may include optical elements such as a lens and a polarizing plate for improving the optical characteristics of the projected light, and a dimming element for adjusting the light flux.

[0159] The light source driving unit 221 can control the light source in the light source unit 211 to operate, including lighting and extinguishing, according to the instruction of the control unit 250 .

[0160] The light modulator 212 includes a display panel 215 , which may be a transmissive liquid crystal display (LCD) panel, a reflective liquid crystal on silicon (LCOS) panel, or a digital micromirror device (DMD).

[0161] The light modulator 212 is driven by a light modulator driving unit 222 , and the light modulator driving unit 222 is connected to the image processing unit 245 .

[0162] The image processing unit 245 inputs image data to the light modulator driving unit 222. The light modulator driving unit 222 converts the input image data into a data signal suitable for the operation of the display panel 215. The light modulator driving unit 222 applies a voltage to each pixel of each display panel 215 according to the converted data signal, and draws an image on the display panel 215.

[0163] The optical system 213 includes a lens or a reflector for forming an image of the incident image light PLA on the imaging medium SC. The optical system 213 may also include a zoom mechanism for magnifying or reducing the image projected on the imaging medium SC and a focus adjustment mechanism for adjusting the focus.

[0164] The projection device 200 further includes an operation unit 231, a signal receiving unit 233, an input interface 235, a storage unit 237, a data interface 241, an interface unit 242, a frame memory 243, an image processing unit 245, and a control unit 250. The input interface 235, the storage unit 237, the data interface 241, the interface unit 242, the image processing unit 245, and the control unit 250 can communicate data with each other via the internal bus 207.

[0165] The operation unit 231 can generate corresponding operation signals according to the operations of various buttons and switches on the housing surface of the projection device 200, and output them to the input interface 235. The input interface 235 includes a circuit that outputs the operation signals input from the operation unit 231 to the control unit 250.

[0166] After receiving a signal (such as an infrared signal or a Bluetooth signal) sent from the control device 5 (such as a remote controller), the signal receiving unit 233 can decode the received signal to generate a corresponding operation signal. The signal receiving unit 233 outputs the generated operation signal to the input interface 235. The input interface 235 outputs the received operation signal to the control unit 250.

[0167] The storage unit 237 may be a magnetic recording device such as a hard disk drive (HDD) or a storage device using a semiconductor storage element such as a flash memory. The storage unit 237 stores programs executed by the control unit 250, data processed by the control unit 250, image data, and the like.

[0168] The data interface 241 includes a connector and an interface circuit, and can be wired to other electronic devices 100. The data interface 241 can be a communication interface for performing communication with other electronic devices 100. The data interface 241 receives image data, sound data, etc. from other electronic devices 100. In this embodiment, the image data can be a content image.

[0169] The interface unit 242 is a communication interface for communicating with other electronic devices 100 according to the Ethernet standard. The interface unit 242 includes a connector and an interface circuit for processing a signal sent by the connector. The interface unit 242 is an interface substrate including a connector and an interface circuit and is connected to the main substrate of the control unit 250, which is a substrate on which a processor 253 and other components are mounted. The connector and the interface circuit constituting the interface unit 242 are mounted on the main substrate of the control unit 250. The interface unit 242 can receive setting information or instruction information sent by other electronic devices 100.

[0170] The control unit 250 includes a memory 251 and a processor 253 .

[0171] The memory 251 is a storage device that stores programs and data executed by the processor 253 in a non-volatile manner. The memory 251 is composed of a semiconductor storage element such as a magnetic storage device, a flash read-only memory (ROM), or other types of non-volatile storage devices. The memory 251 may also include a random access memory (RAM) that constitutes a work area of ​​the processor 253. The memory 251 stores data processed by the control unit 250 and a control program executed by the processor 253.

[0172] The processor 253 may be composed of a single processor or a combination of multiple processing groups. The processor 253 executes a control program to control various parts of the projection device 200. For example, the processor 253 performs corresponding interactive projection control according to the operation signal generated by the operation unit 231, and outputs the parameters used in the interactive projection control (such as parameters for performing trapezoidal correction on the image) to the image processing unit 245. In addition, the processor 253 can control the light source in the light source unit 211 to light up, turn off or adjust the brightness by controlling the light source driving unit 221.

[0173] The image processing unit 245 and the frame memory 243 may be composed of integrated circuits. Integrated circuits include large-scale integrated circuits (LSI), application-specific integrated circuits (ASIC), and programmable logic devices (PLD), wherein the PLD may include a field-programmable gate array (FPGA). Among them, the integrated circuit may also include a portion of an analog circuit, or a combination of a processor and an integrated circuit. The combination of a processor and an integrated circuit is called a microcontroller unit (MCU), a system on chip (SoC), a system LSI, a chipset, etc.

[0174] The image processing unit 245 may store the image data received from the data interface 241 in the frame memory 243. The frame memory 243 includes a plurality of memory banks, each of which includes a storage capacity that can write image data of one frame. The frame memory 243 may be composed of a synchronous dynamic random access memory (SDRAM) or a dynamic random access memory (DRAM).

[0175] The image processing unit 245 can perform interactive projection control on the image data stored in the frame memory 243, including resolution conversion, size adjustment, distortion correction, shape correction, digital zoom, image tone adjustment, image brightness adjustment, etc.

[0176] The image processing unit 245 can also convert the input frame frequency of the vertical synchronization signal into a drawing frequency and generate a vertical synchronization signal having the drawing frequency. The generated vertical synchronization signal is called an output synchronization signal. The image processing unit 245 then outputs the output synchronization signal to the optical modulator driving unit 222.

[0177] In some embodiments of the present application, a computer-readable storage medium is provided, which stores a computer program, and the computer program is loaded by a processor, so that the processor executes the steps of the above-mentioned projection control method. The steps of the projection control method here can be the steps of the projection control method in each of the above-mentioned embodiments.

[0178] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0179] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0180] Although the subject matter has been described in the language of specific method logic actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims. Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

Claims

1. A projection control method, characterized in that: The method comprises: When receiving a restore instruction for the memory position of the projection device, obtaining a first environmental feature of the current position of the projection device; Based on the match between the first environmental feature and the second environmental feature of the memory location, determining a target environmental feature in the second environmental feature that matches the first environmental feature; wherein the first environmental feature and the second environmental feature are generated based on multi-sensor data collected from the corresponding projection environment; Determining adjustment information of the projection device based on the first environmental feature and the target environmental feature; The projection device is driven to rotate in space according to the adjustment information, so that the projection picture of the projection device is projected to the memory position corresponding to the target environment feature.

2. The projection control method according to claim 1, characterized in that: The memory location includes corresponding playback information, and the playback information includes one or more of playback content, playback type, and playback settings.

3. The projection control method according to claim 1, characterized in that: The method for generating the first environmental feature includes: Fusing the multi-sensor data corresponding to the current position to obtain a target image; Determining feature points in the target image and descriptors of the feature points based on the detection of the target image; The first environmental feature is determined according to the multi-sensor data and the descriptors of the feature points.

4. The projection control method according to claim 3, characterized in that: The multi-sensor data corresponding to the current position includes image information, depth information and illumination information of the projection environment, and fusing the multi-sensor data corresponding to the current position to obtain the target image includes: Registering the image information with the depth information to determine the depth information corresponding to each pixel in the image information; Generate an initial image according to the texture information corresponding to each pixel in the image information and the depth information corresponding to each pixel; The visual parameters of the initial image are adjusted according to the illumination information to obtain the target image.

5. The projection control method according to claim 3, characterized in that: The determining, based on the detection of the target image, the feature points of the target image and the descriptors of the feature points comprises: Performing edge detection on the target image to obtain an edge image; Determining the feature points from the pixels according to the pixel values ​​of the pixels in the edge image; Dividing the neighborhood corresponding to the feature point in the edge image into a plurality of grids, and generating a gradient histogram corresponding to each grid according to the gradient information of each grid; Determine the gradient magnitude vector corresponding to each grid according to the target gradient magnitude of each gradient direction in the gradient histogram; Aggregate the gradient magnitude vectors corresponding to the multiple grids to obtain a descriptor of the feature point.

6. The projection control method according to claim 3, characterized in that: The determining the adjustment information of the projection device based on the first environmental feature and the target environmental feature includes: Determining a plurality of groups of initial feature point pairs by matching the first environmental feature with the descriptors in the target environmental feature; Mapping the depth information of each group of initial feature point pairs to a first preset coordinate system to obtain updated depth information of each group of initial feature point pairs; Selecting a target feature point pair from the multiple groups of initial feature point pairs according to the updated depth information, the target feature point pair comprising a first target feature point and a second target feature point; Aligning the first target feature point with the second target feature point to obtain a perspective transformation matrix; Based on the perspective transformation matrix, the adjustment information is determined.

7. The projection control method according to claim 1, characterized in that: The adjustment information includes posture information, and driving the projection device to rotate in space according to the adjustment information includes: According to the posture information, the projection device is controlled to rotate in space using the pan / tilt platform of the projection device.

8. The projection control method according to claim 1, characterized in that: The adjustment information includes correction information, and the method further includes: The projection image is corrected according to the correction information, wherein the correction of the projection image includes image scaling and trapezoidal correction.

9. The projection control method according to claim 1, characterized in that: The method further comprises: If it is detected that the current projection environment changes, the projection parameters of the projection device are adjusted.

10. A projection device, characterized in that: The projection device comprises: processor; A memory; and an application program, wherein the application program is stored in the memory and is configured to be executed by the processor to implement the projection control method according to any one of claims 1 to 9.

Images