A projector and a projection control method

Abstract

Embodiments of the present application provide a projector and a projection control method. The projector comprises a projection component, a camera and a pose sensor group. The projector is configured to: acquire user picture content selected by a user, emit a projection structured light to a target projection surface through the projection component, and display the user picture content on the target projection surface in real time by using the projection structured light; collect a pose transformation signal of the projector in real time by using the pose sensor group, and use the pose transformation signal to indicate a pose change state of the projector; continuously capture the target projection surface by using the camera to obtain a real-time updated captured image when the pose of the projector changes; and adjust an emission parameter of the projection component multiple times according to the currently collected pose transformation signal and the currently updated captured image, so that an imaging range of the user picture content remains rectangular or square. The embodiments can realize high-frequency and high-precision adjustment of the parameters of the projector at a low hardware cost.

Description

Technical Field

[0001] This application relates to the field of image intelligent projector technology, and in particular to a projector and projection control method. Background Technology

[0002] A projector consists of a projection component. Based on the user's selected screen content, the projector emits projection light through the projection component, displaying the user's screen content on a target projection surface for viewing. However, in actual use, the projector's posture is critical. Ideally, the projector should be kept as stationary as possible. Otherwise, changes in the projector's position due to movement will degrade the quality of the image projected onto the target surface, affecting the user's viewing experience. For example, changes in the relative position between the projector and the target projection surface can cause the projection to lose focus, resulting in a blurry image; or changes in the relative angle between the projector and the target projection surface can alter the shape of the image border, causing aspect ratio distortion. This effect is particularly pronounced in application scenarios where the projector's posture changes frequently.

[0003] One improved method involves installing a time-of-flight (TOF) sensor on the projector. The TOF sensor rapidly acquires depth information between the projector and the target projection surface. Based on this depth information, the projector's projection parameters can be dynamically adjusted in real time as the projector frequently changes its pose, ensuring the projected user image remains in a high-quality imaging state. However, depth detection using TOF sensors relies on increasing the number of TOF sensors and detection points to improve accuracy, and TOF sensors are very expensive. Summary of the Invention

[0004] This application provides a projector and a projection control method that achieves high-precision, high-frequency adjustment of projector parameters while reducing costs.

[0005] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0006] In a first aspect, a projector is provided, comprising a projection component, a camera, and a pose sensor array. The projector is used to: acquire user-selected screen content; emit structured light from the projection component onto a target projection surface, the structured light being used to display the user screen content on the target projection surface in real time; acquire pose change signals of the projector in real time via the pose sensor array, the pose change signals indicating the pose change state of the projector; continuously capture images of the target projection surface using the camera to obtain real-time updated images when the projector's pose changes; and adjust the emission parameters of the projection component multiple times based on the currently acquired pose change signals and the currently updated images to maintain the imaging range of the user screen content in a rectangular or square shape, the emission parameters including the emission focal length of the structured light and the projection imaging angle.

[0007] In this embodiment, the projector can perform multiple depth information calculations based on a single captured image. Although the camera's shooting rate is relatively slow, the pose change signal acquisition rate of the pose sensor array is very fast, typically reaching 100Hz-1000Hz. Therefore, if the pose change signal is not merely used as an auxiliary parameter to assist in depth information parameter adjustment, but rather its high acquisition rate is utilized to fuse the pose change signal with the captured image, then it is possible to: perform real-time data fusion based on the currently acquired pose change signal and the currently updated captured image, allowing for multiple adjustments to the projection component's emission parameters. By repeatedly adjusting the emission focal length, the clarity of the projected display can be adjusted even as the projector's pose changes. By adjusting the projection imaging angle, the border shape of the image area of ​​the user's screen content can be adjusted, ensuring that the image area always remains rectangular or square. Since almost all low-priced, mid-priced, and high-priced projectors on the market now come with cameras and pose sensor groups, this solution can achieve high-frequency adjustment of transmission parameters by using a low-cost hardware architecture with a camera and pose sensor components, while decoupling the limitation of the camera's shooting frequency on the frequency of transmission parameter adjustment.

[0008] In one possible implementation, the signal acquisition frequency of the pose sensor group is higher than the camera's shooting frequency. During the time interval between two consecutive camera shooting operations, the pose sensor group acquires and updates the pose transformation signal multiple times. The aforementioned multiple adjustments to the projection component's emission parameters based on the currently acquired pose transformation signal and the currently updated captured image include: before the next updated captured image, adjusting the emission parameters multiple times based on the multiple acquired and updated pose transformation signals and the currently updated captured image. In this embodiment, the pose transformation signal acquired by the pose sensor group can be acquired and updated multiple times during the time interval between two consecutive image acquisitions. Therefore, multiple adjustments to the emission parameters can be achieved based on the multiple acquired and updated pose transformation signals, combined with a single updated captured image. In this implementation, the adjustment frequency of the emission parameters is no longer limited by the camera's shooting frequency and can reach a higher adjustment frequency, specifically related to the signal acquisition frequency of the pose sensor group. Since the signal acquisition frequency of the pose sensor group can reach 100Hz-1000Hz, the adjustment frequency of this solution can also reach a very high level.

[0009] In one possible implementation, the pose sensor group includes multiple pose sensors, each used to acquire different types of pose transformation signals, and the signal acquisition frequencies of the different pose sensors are different. In this embodiment, based on multiple different pose sensors, different pose transformation signals can be acquired, and using multiple types of pose transformation signals can improve the accuracy of projector pose detection.

[0010] In some possible implementations, each time the projector adjusts its emission parameters, it needs to rely on a new calculation of the depth information after data fusion. The following are several different frequency design methods for data fusion:

[0011] In some examples, the above-mentioned adjustment of the transmission parameters based on multiple acquired and updated pose transformation signals and the currently updated captured image includes setting multiple fixed time periods. Within each time period, the transmission parameters are adjusted once based on the currently acquired and updated pose transformation signals and the currently updated captured image, thus completing multiple adjustments of the transmission parameters within multiple fixed time periods. In the embodiments of this application, the adjustment operation can be set based on a certain time period interval. After reaching a fixed time interval, the projector can perform an adjustment operation based on the currently acquired and updated pose transformation signals and the currently acquired and updated captured image.

[0012] In some examples, the multiple pose sensors include a first pose sensor, which is the pose sensor with the highest signal acquisition frequency among the multiple pose sensors. During the time interval between two consecutive camera shooting operations, the first pose sensor acquires and updates the corresponding first pose transformation signal multiple times. The aforementioned multiple adjustments to the transmission parameters based on the multiple acquired and updated pose transformation signals and the currently updated captured image include: adjusting the transmission parameters once each time the first pose transformation signal is acquired and updated, based on the currently acquired and updated multiple pose transformation signals and the currently updated captured image, so that the transmission parameters are adjusted multiple times during multiple acquisitions and updates of the first pose transformation signal. In the embodiments of this application, data fusion can be achieved based on the frequency of the pose sensor with the highest signal acquisition frequency. In this way, each data fusion can ensure that newly acquired and updated data is incorporated, and the adjustment frequency is relatively fast.

[0013] In some examples, the multiple pose sensors include a second pose sensor, which is the pose sensor with the lowest signal acquisition frequency among the multiple pose sensors. During the time interval between two consecutive camera shooting operations, the second pose sensor repeatedly acquires and updates the corresponding second pose transformation signal. The aforementioned multiple adjustments to the transmission parameters based on the repeatedly acquired and updated pose transformation signals and the currently updated captured image include: adjusting the transmission parameters once each time the second pose transformation signal is acquired and updated, based on the currently acquired and updated multiple pose transformation signals and the currently updated captured image, so that the transmission parameters are adjusted multiple times during multiple acquisitions and updates of the second pose transformation signal. In the embodiments of this application, data fusion can be achieved based on the frequency of the pose sensor with the lowest signal acquisition frequency. In this way, it can be ensured that the multiple pose transformation signals used in each data fusion operation are updated pose transformation signals, thus ensuring the accuracy of the adjustment in each fusion operation.

[0014] In one possible implementation, the multiple pose sensors include a gravity acceleration sensor and a gyroscope sensor, and the pose transformation signal includes a gravity acceleration signal acquired by the gravity acceleration sensor and a rotation rate signal acquired by the gyroscope. In this embodiment, when the projector is in a dynamic state, the pitch angle, roll angle, and yaw angle of the projector can be calculated based on the combination of the gravity acceleration sensor and the gyroscope sensor, and the pose of the projector and the target projection surface can be determined based on the pitch angle, roll angle, and yaw angle.

[0015] In one possible implementation, the above-mentioned adjustment of the emission parameters of the projection component based on the currently acquired pose transformation signal and the currently updated captured image includes: obtaining depth information about the target projection surface based on the currently updated captured image; and adjusting the emission parameters of the projection component multiple times based on the currently acquired pose transformation signal and the depth information of the target projection surface. In this embodiment, during each adjustment of the emission parameters under data fusion, it is necessary to obtain depth information from the captured image and perform data fusion based on the depth information and the currently acquired and updated pose transformation signal. Over a continuous period of time, the currently acquired pose transformation signal is continuously updated. Data fusion can be performed based on the updated pose transformation signal and the currently updated depth information, according to the frequency of data fusion described in the above embodiment. The emission parameters are then adjusted based on the result of the data fusion. The result of multiple data fusions can be used for multiple adjustments of the emission parameters.

[0016] In one possible implementation, when the projector's pose changes, the projected structured light is also used to display a marker pattern on the target projection surface in real time, and the captured image includes the captured content of the marker pattern. Obtaining depth information about the target projection surface based on the currently updated captured image includes: obtaining depth information about the target projection surface based on the captured content of the marker pattern in the currently updated captured image and the marker pattern itself, using binocular vision matching technology. In this embodiment, the projector emits projected structured light onto the target projection surface. This projected structured light serves as the projection light for displaying the user's screen content, and the projected image can be used as a structured light pattern to form a calibration pattern. A camera captures an image of the target projection surface to obtain a captured pattern. Since the data information of the calibration pattern projected by the projector is known, the projector can obtain depth information based on the captured image of the known calibration pattern on the target projection surface, combined with the known data information of the projected content. In this implementation, while adjusting the emission parameters, the user can maintain real-time viewing of the user's screen content.

[0017] In some examples, the logo pattern is displayed outside the imaging area of ​​the user's screen content. In this embodiment, the logo pattern is an independent pattern outside the user's screen content. In this case, although the user is aware of the projection of the logo pattern, it does not affect the user's viewing of the screen content. In this approach, the data processing load of the projector's internal controller is relatively small, and the requirements for the projector's hardware performance are lower. This solution can be well applied to projectors with very low market prices, achieving adjustment frequencies of tens to over thirty Hz. Furthermore, in low-end projectors with relatively higher hardware costs, the adjustment frequency can be even higher. In mid-range projectors, adjustment frequencies of over 30 Hz can be achieved. In higher-end projectors, with improvements in the performance of internal control components, even higher adjustment frequencies can be achieved.

[0018] In some examples, the marker pattern is part of the image content within the user's screen. In this implementation, the controller inside the projector can use the marker features within the user's screen content as marker points, and visually match the information about the marker points in the user's screen content data with the information about the feature points in the captured image to calculate depth information.

[0019] In one possible implementation, the above-mentioned adjustment of the emission parameters of the projection component based on the currently acquired pose transformation signal and the depth information of the target projection surface includes: determining the relative pose relationship between the projector and the target projection surface based on the depth information of the target projection surface; dynamically determining the pose information of the projector in the ground coordinate system based on the currently acquired pose transformation signal; dynamically determining the pose information of the target projection surface in the ground coordinate system based on the relative position relationship and the pose information of the projector in the ground coordinate system; and adjusting the emission parameters of the projection component multiple times based on the pose information of the target projection surface in the ground coordinate system. In this embodiment, the depth information records the distance information between the projector and the marker points on the target projection surface, etc. Based on this depth information, the relative pose relationship of the target projection surface in the projector coordinate system can be determined. Based on the pose transformation signal, the coordinates of the projector in the ground coordinate system can be determined. By setting the transformation relationship between the ground coordinate system and the projector coordinate system, the coordinates of the target projection surface in the ground coordinate system can be obtained, thereby enabling the adjustment of the emission parameters.

[0020] For example, a transformation relation matrix can be generated based on matrix transformation, recording the transformation relationship between any angular position in the ground coordinate system and the projector coordinate system. This transformation relation matrix can express the transformation relationship between the two coordinate systems based on sine and cosine functions, and the function angles recorded in the sine and cosine functions are the pitch, roll, and yaw angles determined based on the pose transformation signals from the pose sensor array. In practical applications, after obtaining the features expressing the relative positional relationship based on depth information, the pitch, roll, and yaw angles are determined based on the pose transformation function. The actual values ​​of the pitch, roll, and yaw angles are then written into the transformation relation matrix, thus converting the target projection surface to its coordinate position in the ground coordinate system.

[0021] For example, based on depth information, the coordinates of the target projection surface's marker-fitted plane in the projector coordinate system can be determined. Based on the plane equation of this fitted plane, the normal vector of the fitted plane in the projector coordinate system can be obtained. Regardless of the coordinate system in which the fitted plane is located, the dot product of the normal vector and the gravitational component of the fitted plane remains unchanged. Therefore, the dot product of the normal vector and the gravitational component of the fitted plane in the projector coordinate system is equal to the dot product of the normal vector and the gravitational component of the fitted plane in the ground coordinate system, and also equal to the Z-axis component of the fitted plane in the ground coordinate system. After determining the Z-axis component of the fitted plane in the ground coordinate system, based on the transformation relationship between its Y-axis and X-axis components in the ground coordinate system, and the transformation matrix between the ground coordinate system and the projector coordinate system, the coordinate information of the fitted plane in the ground coordinate system, i.e., the coordinate information of the target projection surface T in the ground coordinate system, can be obtained.

[0022] In this embodiment, the pose transformation signal acquired by the pose sensor array can adapt to coordinate transformations in any scene and at any angle, without relying on lookup table mappings with precision limitations. Furthermore, the pose transformation signal itself is used to acquire various angles of the projector. In the aforementioned coordinate system transformation, the transformation relationship matrix is ​​based on sine and cosine functions including angle information. As long as the information acquired by the pose sensor array is accurate, there will be no fusion errors in the pose transformation signal, thus ensuring processing accuracy.

[0023] Secondly, this application also provides a projection control method applied to a projector, which includes a projection component, a camera, and a pose sensor array. The projection control method includes: acquiring user-selected screen content; controlling the projection component to emit projection structured light onto a target projection surface, the projection structured light being used to display the user screen content on the target projection surface in real time; acquiring pose change signals of the projector in real time through the pose sensor array, the pose change signals indicating the pose change state of the projector; controlling the camera to continuously capture images of the target projection surface to obtain real-time updated images when the projector's pose changes; and adjusting the emission parameters of the projection component multiple times based on the currently acquired pose change signals and the currently updated images to maintain the imaging range of the user screen content in a rectangular or square shape, the emission parameters including the emission focal length of the projection structured light and the projection imaging angle.

[0024] In one possible implementation, the signal acquisition frequency of the pose sensor group is higher than the camera's shooting frequency. During the time interval between two consecutive camera shooting operations, the pose sensor group acquires and updates the pose transformation signal multiple times. The aforementioned multiple adjustments to the projection component's emission parameters based on the currently acquired pose transformation signal and the currently updated captured image include: before the next updated captured image, adjusting the emission parameters multiple times based on the multiple acquisitions and updates of the pose transformation signal and the currently updated captured image.

[0025] In one possible implementation, the above-mentioned adjustment of the emission parameters of the projection component multiple times based on the currently acquired pose transformation signal and the currently updated captured image includes: obtaining depth information about the target projection surface based on the currently updated captured image; and adjusting the emission parameters of the projection component multiple times based on the currently acquired pose transformation signal and the depth information of the target projection surface.

[0026] In one possible implementation, when the projector's pose changes, the structured light projection is also used to display a marker pattern on the target projection surface in real time, and the captured image includes the captured content of the marker pattern. The aforementioned method of obtaining depth information about the target projection surface based on the currently updated captured image includes: obtaining depth information about the target projection surface based on the captured content of the marker pattern in the currently updated captured image and the marker pattern itself, using binocular vision matching technology.

[0027] In one possible implementation, the logo pattern is displayed outside the imaging area of ​​the user screen content.

[0028] In one possible implementation, the logo pattern is the image content within the user's screen content.

[0029] In one possible implementation, the above-mentioned adjustment of the emission parameters of the projection component based on the currently acquired pose transformation signal and the depth information of the target projection surface includes: determining the relative pose relationship between the projector and the target projection surface based on the depth information of the target projection surface; dynamically determining the pose information of the projector in the ground coordinate system based on the currently acquired pose transformation signal; dynamically determining the pose information of the target projection surface in the ground coordinate system based on the relative position relationship and the pose information of the projector in the ground coordinate system; and adjusting the emission parameters of the projection component multiple times based on the pose information of the target projection surface in the ground coordinate system.

[0030] Thirdly, embodiments of this application also provide a computer-readable storage medium including instructions that, when executed on a controller, cause the controller to perform the projection control method as described in the second aspect above.

[0031] The technical principles and beneficial effects of the second and third aspects mentioned above can be referred to the relevant description of the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the structure of a projector provided in an embodiment of this application;

[0033] Figure 2 This is a schematic diagram of the structure of a first projector provided in an embodiment of this application;

[0034] Figure 3 This is a schematic diagram of the structure of a second projector provided in an embodiment of this application;

[0035] Figure 4 This is a schematic diagram of the structure of a third projector provided in an embodiment of this application;

[0036] Figure 5 This is a schematic diagram of another third projector provided in an embodiment of this application;

[0037] Figure 6 A waveform diagram of a pose transformation signal acquired by a gravity acceleration sensor, provided in an embodiment of this application;

[0038] Figure 7 A schematic flowchart of a projection control method provided in an embodiment of this application;

[0039] Figure 8 A schematic diagram illustrating the display of a logo pattern outside the imaging area of ​​the user screen content, provided as an embodiment of this application;

[0040] Figure 9A frequency diagram illustrating a pose transformation signal, a third captured image, and an adjustment operation, provided in an embodiment of this application;

[0041] Figure 10 A schematic diagram illustrating the frequency of another pose transformation signal, a third captured image, and an adjustment operation provided in an embodiment of this application;

[0042] Figure 11 A schematic flowchart illustrating another projection control method provided in an embodiment of this application;

[0043] Figure 12 A schematic diagram illustrating a projection of user screen content as a logo pattern, provided for an embodiment of this application;

[0044] Figure 13 This is a schematic diagram illustrating feature matching of marker points based on user screen content as a marker pattern and a third captured image, as provided in an embodiment of this application.

[0045] Figure 14 A schematic flowchart illustrating another projection control method provided in an embodiment of this application;

[0046] Figure 15 This is a schematic diagram of the normal vector of the fitting plane obtained by calculating the marker points on the target projection surface in the projector coordinate system, provided as an embodiment of this application. Detailed Implementation

[0047] It should be noted that the terms "first" and "second" used in the embodiments of this application are only used to distinguish features of the same type and should not be construed as indicating relative importance, quantity, order, etc.

[0048] The terms "exemplary" or "for example" used in the embodiments of this application are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0049] The terms "coupling" and "connection" used in the embodiments of this application should be interpreted broadly. For example, they can refer to a physical direct connection or an indirect connection achieved through electronic devices, such as a connection achieved through resistors, inductors, capacitors or other electronic devices.

[0050] With the development of imaging technology, projectors are being used in more and more life and work scenarios.

[0051] Figure 1An exemplary projector 1000 is provided, comprising a projection component 100. The projector 1000 is used to: acquire data information of user-selected screen content; and generate and emit projection light based on the user screen content data. The projection light is emitted onto a target projection surface T to display the user screen content on the target projection surface T. In practical applications, the emission parameters of the projection light affect the image quality of the user screen content. For example, depending on the relative distance between the projector 1000 and the target projection surface T, the projection focal length of the emitted projection light needs to be adjusted to adjust the sharpness of the projected user screen content. Depending on the relative distance and relative angle between the projector 1000 and the target projection surface T, the projection imaging angle of the emitted projection light needs to be adjusted to adjust the shape of the image border of the projected user screen content, for example, from a distorted trapezoidal border to a regular rectangular or square border. Therefore, changes in the pose of the projector 1000 affect the image quality of the projection. To ensure the quality of the projected image, the projector should typically be kept as still as possible.

[0052] When the projector 1000 changes its position due to movement, the quality of the user image projected onto the target projection surface T decreases, affecting the user's viewing experience. For example, a change in the relative position between the projector 1000 and the target projection surface can cause the projection to lose focus, resulting in a blurry image; or a change in the relative angle between the projector 1000 and the target projection surface T can alter the shape of the projection image's borders, leading to image distortion. This effect is particularly pronounced in application scenarios where the projector's position changes frequently.

[0053] In related technologies, a pose sensor array and a depth detection device are typically installed within the projector. The depth detection device detects the depth information between the projector and the target projection surface T, while the pose sensor array acquires the projector's pose angle. Based on the projector's pose angle and depth information, the projector's emission parameters are adjusted to improve the projection display quality. For example, sharpness is adjusted by adjusting the projector's focus, the shape of the image frame is adjusted by adjusting the projection imaging angle, and the coverage of the image area between the projected image and the target projection surface is adjusted by adjusting the size of the projection imaging frame.

[0054] The common operation for acquiring depth information is as follows: A depth detection device sets multiple marker points on the target projection surface T. The detection device then detects the distance information between the projector and the target projection surface T with respect to these marker points. This distance information is the acquired depth information. Based on this depth information, the adjustment range of the aforementioned transmission parameters is determined, and the transmission parameters are adjusted based on the determined adjustment range. Depending on the type of detection device selected, different methods can be used to acquire depth information.

[0055] Currently, in the application of projectors, common methods for depth information detection and acquisition include two types: time-of-flight (TOF) sensors and binocular camera detection.

[0056] In TOF sensor detection, a time-of-flight (TOF) sensor needs to be installed on the projector to acquire depth information between the projector and the target projection surface at a high rate. Due to the extremely high detection rate of TOF sensors, the projection parameters of the projector can be dynamically adjusted in real time even when the projector frequently changes its pose. This ensures that the projected image content remains in a high-quality state in real time. For example, a TOF sensor can achieve a detection rate of 30-60Hz, thus enabling an adjustment rate of 30Hz per second, which is essentially perceptible to the user's naked eye. However, depth detection based on TOF sensors relies on increasing the number of TOF sensors and detection points to improve detection accuracy, but TOF sensors are very expensive. Since mid-to-low-end projectors dominate the market, this solution is not applicable to most projector products. In addition, the detection method based on TOF sensor has low resolution and cannot acquire image information. To improve this shortcoming, it is necessary to combine it with a camera to achieve detection. However, the combination of camera, pose sensor group and TOF sensor will further increase the cost of the device and the processing complexity.

[0057] In binocular vision-based binocular camera detection schemes, a calibration image is projected onto the target projection surface. Then, the binocular camera captures a binocular vision image of the target projection surface. Binocular vision matching is performed based on the calibration points of the calibration image in the binocular vision image to obtain depth information related to these calibration points. Adjustments can then be made based on this depth information. While the cost of binocular cameras in projectors is lower than that of TOF sensors, their image capture rate is relatively slow, making it difficult to achieve a capture rate of 30-60Hz. Furthermore, the acquisition of depth information in binocular vision heavily relies on the projector's processing devices performing data processing and computation at the software algorithm level. This image data processing takes time, so the depth information calculation frequency of low-cost binocular cameras is typically only a few Hz. Under this scheme, the adjustment frequency of projection parameters is very low. This makes it impossible to achieve a real-time dynamic adjustment effect that is imperceptible to the user's naked eye, given the projector's optical-engine projection playback rate of tens of Hz per second. In addition, when the projected calibration image occupies the target projection surface, it will also affect the user's viewing of the content on the screen and affect the user's viewing experience.

[0058] In some improved schemes, depth information can be acquired using a monocular camera. The following two examples illustrate this:

[0059] Example 1, such as Figure 2 As shown, the first projector 1000A includes a first projection component 100A, a first camera 200A, a gravity acceleration sensor 300A, and a controller 400A. The controller 400A controls the first projection component 100A to project a first calibration image T1 onto a target projection surface T. The first calibration image T1 has multiple three-dimensional points D. The first camera 200A captures the first calibration image T1 to obtain a first captured image. The controller 400A of the first projector 1000A determines a planar coordinate system parallel to the ground based on the gravity acceleration sensor 300A, and searches for the largest rectangle in the planar coordinate system based on the multiple three-dimensional points D in the first captured image. The obtained largest rectangle is then projected back onto the target projection surface T as a reverse-mapping image TF. The controller 400A calculates the correction offset based on the projected first calibration image T1 and the reverse-mapping image TF, and adjusts parameters according to the calculation results, such as adjusting the shape of the image frame.

[0060] But Figure 2In the illustrated embodiment, the following problems exist: Problem 1: The need to project a visible first calibration image T1 and an inverse-mapped image TF affects the user's viewing experience. Problem 2: This adjustment operation is also limited by the shooting rate of the first camera 200A. The first projection component 100A can only project the calculated inverse-mapped image TF once for each image captured by the first camera 200A. Furthermore, after projecting the inverse-mapped image TF, the first camera 200A may need to capture multiple images to adjust the borders of the projected image. Therefore, this solution is limited by the shooting rate of the first camera 200A and the image processing rate within the first projector 1000A, and can only be applied to parameter adjustments when the first projector 1000A is in a static state. It is not suitable for continuous, high-frequency, and rapid adjustments in scenarios where the projector moves frequently and continuously.

[0061] Example 2, such as Figure 3 As shown, the second projector 1000B includes a second projection component 100B, a second camera 200B, an inertial measurement unit (IMU) sensor 300B, and a controller 400B. The controller 400B controls the second projection component 100B to project a second calibration image T2 onto a target projection surface T. The data signal of the second calibration image T2 carries a known calibration image, such as a rectangular pattern with multiple calibration points. However, due to possible angular offsets between the target projection surface T and the second projector 1000B, the actual projected display of the second calibration pattern T2 may be a non-rectangular pattern. In this case, the second camera 200B captures an image of the second calibration image T2, obtaining a second captured image. The controller 400B calculates depth information based on the calibration point information in the second captured image, and obtains the coordinate information of the multiple calibration points in the camera coordinate system of the second projector 1000B based on the depth information. Based on this coordinate information, it fits the normal vector of the fitting plane of the multiple calibration points. The controller 400B determines the current pose of the second projector 1000B based on the inertial measurement unit sensor 300B. The mapping relationship between the depth information of multiple feature points and the camera coordinate system is pre-defined in the second projector 1000B. Based on the current pose, the determined normal vector, and the pre-defined mapping relationship between the depth information and the camera coordinate system, the controller 400B adjusts the projection parameters.

[0062] But Figure 3The illustrated embodiment has the following problems: Problem 1: A second calibration image T2 visible to the user is still required for depth information detection. This second calibration image T2 will still interrupt the user's viewing. Problem 2: This adjustment scheme requires a pre-set mapping relationship between depth information and the camera coordinate system. This mapping relationship has limitations in accuracy and cannot cover all application scenarios. Problem 3: If the pose information provided by the inertial measurement unit sensor 300B is used incorrectly, it will lead to errors and failures in the scheme implementation when looking up the mapping relationship. Problem 4: This scheme still performs image data processing based on the second captured image after each shooting operation to obtain depth information. Based on the current depth information, a lookup mapping fusion is performed with the pose information provided by the inertial measurement unit sensor 300B. Therefore, this scheme is still limited by the shooting rate of the second camera 200B and can only be applied to parameter adjustment when the second projector 1000B is in a static state. It cannot be applied to scenarios where the projector moves frequently and continuously for multiple rapid adjustments.

[0063] In summary, current technical solutions for achieving rapid, real-time adjustments to projection parameters in scenarios involving frequent projector movement primarily rely on a combination of Time-of-Flight (TOF) sensors and IMU (Integrated Mutual Memory) sensors, or a combination of TOF and IMU sensors with a camera. However, these solutions are extremely costly. While the cost of a camera is far less than half that of a TOF sensor, and a camera-based IMU sensor solution can adjust projection parameters at a lower cost, the adjustment frequency in these technologies is still limited by the camera's shooting rate, preventing the achievement of high-speed adjustments.

[0064] To achieve high-speed adjustment of projection parameters in scenarios involving frequent projector movement while significantly reducing projector costs, embodiments of this application also provide a third projector, such as... Figure 4As shown, the third projector 1000C includes a third projection component 100C, a third camera 200C, and a pose sensor group 300C. The third projector 1000C is used to: acquire the user-selected screen content; emit structured light onto a target projection surface via the third projection component 100C; and display the user screen content on the target projection surface in real time using the structured light. The pose sensor group 300C acquires the pose transformation signal of the third projector 1000C in real time, indicating the pose change state of the third projector 1000C. When the pose of the third projector 1000C changes, the third camera 200C continuously captures images of the target projection surface to obtain a real-time updated third image. Based on the currently acquired pose transformation signal and the currently updated third image, the emission parameters of the third projection component 100C are adjusted multiple times to maintain the imaging range of the user screen content as rectangular or square. The emission parameters include the emission focal length of the structured light and the projection imaging angle.

[0065] In this application, Figure 4 In the illustrated embodiment, the third projector 1000C can perform multiple depth information calculations based on a third captured image obtained from a single capture. Although the capture rate of the third camera 200C is relatively slow, the acquisition rate of the pose transformation signal of the pose sensor group 300C is very fast, typically reaching 100Hz-1000Hz. Therefore, if the pose transformation signal is not merely used as an auxiliary parameter to assist in depth information parameter adjustment, but rather its high acquisition rate is utilized to fuse the pose transformation signal with the third captured image acquired by the third camera 200C, then it is possible to: fuse the pose transformation signal with the third captured image, and based on the fused data, and according to the currently acquired pose transformation signal and the currently updated third captured image, adjust the emission parameters of the third projection component 100C multiple times. By adjusting the emission focal length, the clarity of the projected display can be adjusted when the pose of the third projector 1000C changes. By adjusting the projection angle, the shape of the border of the image area of ​​the user's screen content can be adjusted so that the image area always remains rectangular or square.

[0066] exist Figure 4 The illustrated implementation method has the following advantages:

[0067] Advantage 1: Based on the hardware architecture of the third camera 200C and the pose sensor group 300C, the device cost is far lower than that of a TOF sensor combined with an IMU sensor. With this low-cost hardware architecture, the adjustment frequency of projection parameters is not limited by the shooting frequency of the third camera 200C, enabling rapid adjustment of projection parameters.

[0068] Advantage 2: The third projection component 100C emits structured light onto the target projection surface T. This structured light serves as the projection light for displaying the user's screen content, and the projected image can also be used as a structured light pattern. The third camera 200C captures an image of the target projection surface T, obtaining a third captured pattern. Since the data information of the content projected by the third projector 1000C is known, the third projector 1000C, based on the known projection content on the target projection surface T captured in the third captured pattern, combined with the known data information of this projection content, can acquire depth information. In this implementation, while adjusting the emission parameters, the user can maintain real-time viewing of the user's screen content.

[0069] In some possible implementations, since the signal acquisition frequency of the pose sensor group 300C is higher than the shooting frequency of the third camera 200C, the pose sensor group 300C acquires and updates the pose transformation signal multiple times during the time interval between two adjacent shooting operations of the third camera 200C. In this embodiment, the pose transformation signal acquired by the pose sensor group 300C can be acquired and updated multiple times during the time interval between two adjacent acquisitions and updates of the third captured image. Therefore, multiple adjustments to the transmission parameters can be achieved based on the multiple acquisitions and updates of the pose transformation signal, combined with a single updated third captured image. In this implementation, the adjustment frequency of the transmission parameters is no longer limited by the shooting frequency of the third camera 200C, and a higher adjustment frequency can be achieved. The specific adjustment frequency is related to the signal acquisition frequency of the pose sensor group 300C. Since the signal acquisition frequency of the pose sensor group 300C can reach 100Hz-1000Hz, theoretically, the adjustment frequency of this solution can also reach a very high level.

[0070] In some examples, the pose sensor group 300C includes multiple pose sensors, each used to acquire different types of pose transformation signals, and the signal acquisition frequencies of these different pose sensors are different. In some examples, such as... Figure 5 As shown, the pose sensor with the highest signal acquisition frequency among various pose sensors is the first pose sensor 311C. In some examples, the pose sensor with the lowest signal acquisition frequency among various pose sensors is the second pose sensor 312C.

[0071] For example, the pose sensor group 300C may include multiple pose sensors such as an accelerometer and a gyroscope. For example, the pose sensor group 300C may be a six-axis sensor including multiple pose sensors, i.e., an inertial measurement unit (IMU) sensor. The accelerometer and gyroscope are integrated into the IMU sensor. For example, the pose sensor group 300C may be a nine-axis sensor including multiple pose sensors, integrating an accelerometer, gyroscope, and magnetometer. In this embodiment, when the projector is in a static state, the roll angle and pitch angle of the projector can be detected based on the accelerometer. When the projector is in a dynamic state, a combination of a gravity accelerometer and a gyroscope is needed to detect the projector's roll angle (r0ll), pitch angle, and yaw angle. To maintain accurate detection of these three angles over a sustained period and prevent yaw deviations, a magnetometer based on a nine-axis sensor is required to improve detection accuracy. However, in scenarios involving projector adjustments, a six-axis sensor is sufficient.

[0072] For example, such as Figure 6 The diagram shows the pose transformation signal of the gravity acceleration sensor. With the direction of gravity towards the ground as the Z-axis of the ground coordinate system and the direction towards the target projection plane as the X-axis, line ① represents the change of gravitational acceleration of the third projector 1000C with respect to the X-axis of the ground coordinate system over time; line ② represents the change of gravitational acceleration of the third projector 1000C with respect to the Y-axis of the ground coordinate system over time; and line ③ represents the change of gravitational acceleration of the third projector 1000C with respect to the Z-axis of the ground coordinate system over time. The diagram for the pose transformation signal acquired by the gyroscope is similar to that of the gravity acceleration sensor and will not be repeated here. Figure 6 As can be seen from the diagram, the pose changes of the third projector 1000C can be represented as continuous changes in the pose transformation signal based on the data collected by sensors such as the gyroscope and gravity acceleration sensor. Figure 6The horizontal axis in the figure illustrates the pose transformation with a time unit of seconds. Although not shown in the attached figure, because the signal acquisition frequency of the 300C pose sensor group is very high, if the window time accuracy in the attached figure is set more finely, the waveform of the pose transformation signal can be seen to show more detailed changes in the sampled data.

[0073] In some possible implementations, such as Figure 4 and Figure 5 As shown, the third projector 1000C also includes a controller 400C. The controller 400C is electrically connected to the third projection component 100C, the third camera 200C, and the pose sensor group 300C. Based on user operations, the controller 400C acquires data information of the user's selected screen content and controls the third projection component 100C to emit projection structured light, so that the emitted projection structured light can display the user's screen content on the target projection surface T. Simultaneously, the controller 400C can control the third camera 200C to take pictures when the emission parameters need to be adjusted, and acquire the third captured image. The controller 400C can also acquire pose transformation signals collected by the pose sensor group 300C. The controller 400C detects depth information based on the data information of the third captured image and the user screen content. It also adjusts the emission parameters of the third projection component 100C multiple times based on the currently acquired pose transformation signals and the currently updated third captured image.

[0074] For example, the third projection component 100C may also include a memory. The memory is used to cache or store pose transformation signals and third captured images acquired in real time by the third projection component 100C. Alternatively, the memory can store data information of relevant calibration patterns of the third projector 1000C.

[0075] Based on the above, including Figure 4 , Figure 5 and Figure 6 The third projector 1000C of the illustrated embodiment can perform the following: Figure 7 The projection control method shown includes steps S100-S400:

[0076] S100: Obtain the content of the user's selected screen.

[0077] For example, a user can transmit data information of the selected user screen content to the third projector 1000C through the data interface on the third projector 1000C.

[0078] For example, a user can connect to a storage device that stores data information of the user's screen content through the data interface on the third projector 1000C, and select the relevant user screen content based on the user's operation on the third projector 1000C.

[0079] For example, the third projector 1000C can be connected to the network via wired or wireless means, and the user can select relevant user screen content and obtain relevant data information from the network based on user operations on the third projector 1000C.

[0080] S200, control the third projection component 100C to emit projection structured light toward the target projection surface T.

[0081] In this embodiment, the controller 400C within the third projector 1000C controls the third projection component 100C to emit projection structured light onto the target projection surface T based on the acquired data information about the user screen content. The projection structured light is used to display the user screen content on the target projection surface T in real time.

[0082] In some possible implementations, the projected structured light is also used to display a logo pattern on the target projection surface T.

[0083] In some examples, the marker pattern is the image content within the user's screen. For example, multiple marker points can be set within the marker pattern, and these marker points can be image features within the user's screen content. In this embodiment, when the user's screen content is used as the marker pattern, the user is completely unaware of the depth information detection operation of the third projector 1000C while viewing the user's screen content.

[0084] In some examples, such as Figure 8 As shown, the marker pattern is displayed outside the imaging area of ​​the user screen content. Exemplarily, the marker pattern can be an edge line surrounding the outer edge of the imaging area of ​​the user screen content, random speckle, or a checkerboard pattern. In this embodiment, when the marker pattern is located outside the imaging area of ​​the user screen content, it ensures that although the marker pattern is visible, the user screen content can still be viewed during the depth information detection operation.

[0085] The S300A acquires the pose transformation signals of the third projector 1000C in real time through the pose sensor group 300C.

[0086] In this embodiment, the pose change signal is used to indicate the pose change state of the third projector 1000C.

[0087] S300B controls the third camera 200C to continuously capture images of the target projection surface T to obtain real-time updated third images.

[0088] In this embodiment, when the pose of the third projector 1000C changes, the third camera 200C is controlled to continuously capture images of the target projection surface T to obtain a real-time updated third captured image. The third captured image includes the captured content of the logo pattern.

[0089] S400: The emission parameters of the third projection component 100C are adjusted multiple times based on the currently acquired pose transformation signal and the currently updated third captured image.

[0090] In this embodiment, the emission parameters of the third projection component 100C are adjusted multiple times based on the currently acquired pose transformation signal and the currently updated third captured image, so that the imaging range of the user's screen content remains rectangular or square. Specifically, during the adjustment, since the third camera 200C acquires the third captured image at a relatively low frequency, the pose transformation signal acquired by the pose sensor group 300C, which has a faster signal acquisition frequency, can be fused with the third captured image. By combining the high-speed characteristics of the pose transformation signal, the limitation of the third camera 200C's capture frequency on the adjustment frequency is decoupled.

[0091] In some possible implementations, the signal acquisition frequency of the pose sensor group 300C is greater than the shooting frequency of the third camera 200C. During the time interval between two consecutive shooting operations of the third camera 200C, the pose sensor group 300C acquires and updates the pose transformation signal multiple times. Step S400 includes: adjusting the transmission parameters multiple times based on the repeatedly acquired and updated pose transformation signals and the currently updated third image before the next update of the third captured image.

[0092] exist Figure 2 and Figure 3 In the illustrated embodiment, the projector performs an adjustment operation based on the currently acquired image each time it captures a corresponding image. However, in the embodiments of this application, as... Figure 9 As shown, during a continuous adjustment period, the third projector continuously captures the third image according to the shooting frequency of the third camera. The third projector continuously acquires pose transformation signals based on the signal acquisition frequency of the pose sensor group. Because the pose transformation signal acquisition frequency is very high, data fusion is performed between the real-time updated pose transformation signal and the current third image to achieve multiple adjustments to the third projector's transmission parameters based on the high frequency of the pose transformation signal. In this implementation, the limitation of the adjustment frequency imposed by the third camera is decoupled, and a single third image combined with the pose transformation signal can be used to perform multiple adjustments to the transmission parameters.

[0093] In some possible implementations, each adjustment operation requires performing data fusion of the pose transformation signal and the third captured image. The pose transformation signal and the third captured image can be fused based on different data fusion frequencies. The following are examples of several data fusion frequency settings:

[0094] Example 1, such as Figure 10 As shown, in step S400, multiple fixed time periods are set. Within each time period, the transmission parameters are adjusted once based on the currently acquired and updated pose transformation signals and the currently updated third captured image, thus completing multiple adjustments to the transmission parameters within multiple fixed time periods. In this embodiment, the adjustment operation can be set based on a certain time period interval. After a fixed time interval is reached, the controller within the third projector can perform an adjustment operation based on the currently acquired and updated pose transformation signals and the currently acquired and updated third captured image.

[0095] For example, the length of the fixed time period is less than the time interval between two adjacent shooting operations.

[0096] For example, the length of the fixed time period can be greater than the sampling interval of the first pose sensor, or it can be less than or equal to the sampling interval of the first pose sensor. In this embodiment, the first pose sensor is the sensor with the highest signal acquisition frequency.

[0097] Example 2: Multiple pose sensors include a first pose sensor, which has the highest signal acquisition frequency among the multiple pose sensors. During the time interval between two consecutive shooting operations of the third camera 200C, the first pose sensor repeatedly acquires and updates the corresponding first pose transformation signal. In step S400, each time the first pose transformation signal is acquired and updated, the transmission parameters are adjusted once based on the currently acquired and updated multiple pose transformation signals and the currently updated third-shot image, so that the transmission parameters are adjusted multiple times when the first pose transformation signal is acquired and updated multiple times. In this embodiment, data fusion can be achieved based on the frequency of the pose sensor with the highest signal acquisition frequency. In this way, each data fusion can ensure that newly acquired and updated data is combined, and the adjustment frequency is relatively fast.

[0098] Example 3: Multiple pose sensors include a second pose sensor, which is the pose sensor with the lowest signal acquisition frequency among the multiple pose sensors. During the time interval between two consecutive shooting operations of the third camera 200C, the second pose sensor repeatedly acquires and updates the corresponding second pose transformation signal. In step S400, each time the second pose transformation signal is acquired and updated, the transmission parameters are adjusted once based on the currently acquired and updated multiple pose transformation signals and the currently updated third captured image, so that the transmission parameters are adjusted multiple times during multiple acquisitions and updates of the second pose transformation signal.

[0099] In some possible implementations, during each data fusion operation, an adjustment of the transmission parameters is performed based on the data fusion. In this case, step S400 may include the following: Figure 11 The sub-operations shown in steps S410-S420 are as follows:

[0100] S410. Obtain depth information about the target projection plane T based on the currently updated third captured image.

[0101] In this embodiment of the application, the depth information about the target projection surface T can be calculated based on the structured light emitted by the third projection component 100C and the third captured image obtained by the third camera 200C.

[0102] In some possible implementations, when the pose of the third projector 1000C changes, the projected structured light is also used to display a marker pattern on the target projection surface T in real time, and the third captured image includes the captured content of the marker pattern. Step S410 includes: based on binocular vision matching technology, obtaining depth information about the target projection surface T based on the captured content of the marker pattern in the currently updated third captured image and the marker pattern. In this embodiment, the third projection component 100C emits projected structured light to the target projection surface T. This projected structured light serves as the projection light for displaying user screen content, and the projected image can also be used as a structured light pattern to form a calibration pattern. The third camera 200C captures an image of the target projection surface T to obtain a third captured pattern. Since the data information of the calibration pattern projected by the third projector 1000C is known, the third projector 1000C can obtain depth information based on the captured image of the known calibration pattern on the target projection surface T in the third captured pattern, combined with the known data information of the projected content. In this implementation, users can keep watching the content on their screen in real time while adjusting the transmission parameters.

[0103] For example, such as Figure 8As shown, the logo pattern is displayed outside the imaging area of ​​the user's screen content. In this embodiment, the logo pattern is an independent pattern outside the user's screen content. At this time, although the user is aware of the projection of the logo pattern, it does not affect the user's viewing of the screen content. In this way, the data processing volume of the controller 400C inside the third projector 1000C is relatively small, and the hardware performance requirements of the third projector 1000C are low. This solution can be well applied to projectors with very low market prices, achieving adjustment frequencies of tens to thirty Hz or higher. Furthermore, in low-end projectors with relatively high hardware costs, the adjustment frequency can be even higher. In mid-range projectors, an adjustment frequency of over 30 Hz can be achieved. In higher-end projectors, with improvements in the performance of internal control components, even higher adjustment frequencies can be achieved.

[0104] For example, such as Figure 12 As shown, the logo pattern represents the image content within the user's screen. In this implementation, as... Figure 13 As shown, the controller 400C inside the third projector 1000C can use the marker features within the user's screen content as marker points, and visually match the information about the marker points in the data information of the user's screen content with the information about the feature points in the third captured image to calculate depth information. In this embodiment, the computational load of image processing is greater than... Figure 8 In this embodiment, the adjustment frequency is relatively lower than Figure 8 Examples of implementations.

[0105] S420. The emission parameters of the third projection component 100C are adjusted multiple times based on the currently acquired pose transformation signal and the depth information of the target projection surface T.

[0106] In some possible implementations, step S420 may include the following: Figure 14 The diagram shows the sub-operations including steps S421-S424:

[0107] S421. Determine the relative pose relationship between the third projector 1000C and the target projection surface T based on the depth information of the target projection surface T.

[0108] In this embodiment, the depth information records the distance between the third projector 1000C and the marker points on the target projection surface T. Based on this depth information, the relative pose of the target projection surface T in the projector coordinate system of the third projector 1000C can be determined.

[0109] In some examples, the calibration pattern can have more than three marker points. When there are more than three marker points, the planar coordinates of the target projection surface T in the projector coordinate system can be directly obtained based on the coordinate information of the three marker points.

[0110] In some examples, the number of marker points in the calibration pattern can be less than three, for example, one or two. When there are fewer than three marker points, depth information can be obtained based on the movement of the third projector 1000C. In this embodiment, when there are fewer than three marker points, a pose sensor group 300C is provided within the third projector 1000C. When there are two marker points, based on the pose change information of the third projector 1000C obtained by the pose sensor group 300C and the depth information of the two marker points, the planar coordinates of the target projection surface T in the projector coordinate system can be calculated. When there is only one marker point, based on the pose change information of the third projector 1000C obtained by the pose sensor group 300C and the depth information of the marker point before and after movement, the planar coordinates of the target projection surface T in the projector coordinate system can be calculated.

[0111] S422. Dynamically determine the pose information of the third projector 1000C in the ground coordinate system based on the currently acquired pose transformation signal.

[0112] In this embodiment, the coordinate changes of the third projector 1000C in the ground coordinate system can be determined based on the pose transformation signal of the pose sensor group 300C.

[0113] S423. Dynamically determine the pose information of the target projection surface T in the ground coordinate system based on the relative position relationship and the pose information of the third projector 1000C in the ground coordinate system.

[0114] In this embodiment, a transformation relationship matrix can be generated based on matrix transformation. This matrix records the transformation relationship between any angular position in the ground coordinate system and the projector coordinate system. The transformation relationship matrix can express the transformation relationship between the two coordinate systems based on sine and cosine functions. The angles recorded in the sine and cosine functions are the pitch, roll, and yaw angles determined based on the pose transformation signal from the pose sensor group 300C. In practical applications, after obtaining the features expressing the relative positional relationship based on depth information, the pitch, roll, and yaw angles are determined based on the pose transformation function. The actual values ​​of the pitch, roll, and yaw angles are then written into the transformation relationship matrix, thus converting the target projection surface T to its coordinate position in the ground coordinate system.

[0115] In some examples, such as Figure 15As shown, based on depth information, the coordinates of the target projection surface's marker-fitted plane in the projector coordinate system can be determined. Based on the plane equation of this fitted plane, its normal vector in the projector coordinate system can be obtained. Regardless of the coordinate system in which the fitted plane is located, the dot product of its normal vector and its gravitational component remains unchanged. Therefore, the dot product of the fitted plane's normal vector and its gravitational component in the projector coordinate system is equal to the dot product of the fitted plane's normal vector and its gravitational component in the ground coordinate system, and also equal to the Z-axis component of the fitted plane in the ground coordinate system. After determining the Z-axis component of the fitted plane in the ground coordinate system, based on the transformation relationship between its Y-axis and X-axis components in the ground coordinate system, and the transformation matrix between the ground coordinate system and the projector coordinate system, the coordinate information of the fitted plane in the ground coordinate system, i.e., the coordinate information of the target projection surface T in the ground coordinate system, can be obtained.

[0116] In this embodiment, the pose transformation signal acquired by the pose sensor group 300C can adapt to coordinate transformations in any scene and at any angle, without relying on lookup table mappings with accuracy limitations. Furthermore, the pose transformation signal itself is used to acquire various angles of the third projector 1000C. In the aforementioned coordinate system transformation, the transformation relationship matrix is ​​based on sine and cosine functions including angle information. As long as the information acquired by the pose sensor group 300C is accurate, there will be no fusion errors in the pose transformation signal, thus ensuring processing accuracy.

[0117] For example, when the target projection plane T is not perfectly parallel to the ground (i.e., it is not a projection plane parallel to the XOY plane of the ground coordinate system), only a six-axis sensor, including a gravity acceleration sensor and a gyroscope, is needed to determine the pitch, roll, and yaw angles of the third projector 1000C. However, when the target projection plane T is perfectly parallel to the XOY plane of the ground coordinate system, a nine-axis sensor, including a gravity acceleration sensor, a gyroscope, and a magnetometer, is required to determine the pitch, roll, and yaw angles of the third projector 1000C. In practical applications, however, there is usually only a very low probability, or even a probability close to zero, that the user-set target projection plane T is located in the direction of the ceiling and perfectly parallel to the XOY plane of the ground coordinate system.

[0118] For example, when acquiring pose transformation signals based on a six-axis or nine-axis sensor, the fusion of different pose transformation signals can be achieved based on the Kalman filtering algorithm.

[0119] S424. The emission parameters of the third projection component 100C are adjusted multiple times based on the pose information of the target projection surface T in the ground coordinate system.

[0120] In this embodiment, based on each data fusion operation, the coordinate information of the target projection plane T in the ground coordinate system can be obtained based on the transformation processing in step S423, so as to realize the adjustment of the launch parameters. During a continuous adjustment period, based on the continuous acquisition and updating of pose transformation signals and third captured images, multiple adjustments of the launch parameters can be realized.

[0121] For example, the emission parameters include at least the emission focal length and projection imaging angle of the structured light. By adjusting the emission focal length, the sharpness of the projected display can be adjusted when the pose of the third projector 1000C changes. By adjusting the projection imaging angle, the border shape of the imaging range of the user's screen content can be adjusted so that the imaging range always remains rectangular or square.

[0122] This application also provides a computer-readable storage medium including instructions. When the instructions are executed on a controller, the controller causes the controller to perform the projection control method described in the above embodiments, for example... Figure 7 , Figure 8 , Figure 9 , Figure 10 , Figure 11 , Figure 12 , Figure 13 , Figure 14 and Figure 15 The projection control method described in the relevant embodiments.

[0123] The controller involved in the embodiments of this application can be a chip. For example, it can be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0124] The memory involved in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0125] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0126] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0127] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0128] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or modules may be electrical, mechanical, or other forms.

[0129] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located on one device or distributed across multiple devices. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0130] In addition, the functional modules in the various embodiments of this application can be integrated into one device, or each module can exist physically separately, or two or more modules can be integrated into one device.

[0131] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0132] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A projector, characterized in that, The projector includes a projection assembly, a camera, and a pose sensor group, and is used for: The system acquires the user-selected screen content and emits structured light onto the target projection surface via the projection component. The structured light is used to display the user-selected screen content on the target projection surface in real time. The pose sensor group collects the pose transformation signal of the projector in real time, and the pose transformation signal is used to indicate the pose change state of the projector. When the pose of the projector changes, the camera continuously captures images of the target projection surface to obtain real-time updated images. The emission parameters of the projection component are adjusted multiple times based on the currently acquired pose transformation signal and the currently updated captured image, so that the imaging range of the user screen content remains rectangular or square. The emission parameters include the emission focal length of the projection structured light and the projection imaging angle.

2. The projector according to claim 1, characterized in that, The signal acquisition frequency of the pose sensor group is greater than the shooting frequency of the camera; during the time period between two consecutive shooting operations of the camera, the pose sensor group acquires and updates the pose transformation signal multiple times. The step of adjusting the emission parameters of the projection component multiple times based on the currently acquired pose transformation signal and the currently updated captured image includes: Before the next update of the captured image, the transmission parameters are adjusted multiple times based on the pose transformation signals acquired and updated multiple times and the currently updated captured image.

3. The projector according to claim 2, characterized in that, The pose sensor group includes multiple pose sensors, and different types of pose sensors are used to collect different types of pose transformation signals, and the signal acquisition frequencies of different types of pose sensors are different.

4. The projector according to claim 3, characterized in that, The step of adjusting the transmission parameters multiple times based on the pose transformation signal acquired and updated multiple times and the currently updated captured image includes: Multiple fixed time periods are set; within each time period, the transmission parameters are adjusted once based on the currently acquired and updated pose transformation signals and the currently updated captured image, so as to complete the multiple adjustments of the transmission parameters within multiple fixed time periods.

5. The projector according to claim 3, characterized in that, The multiple pose sensors include a first pose sensor, which is the pose sensor with the highest signal acquisition frequency among the multiple pose sensors; during the time period between two adjacent shooting operations of the camera, the first pose sensor acquires and updates the corresponding first pose transformation signal multiple times. The step of adjusting the transmission parameters multiple times based on the pose transformation signal acquired and updated multiple times and the currently updated captured image includes: Each time the first pose transformation signal is acquired and updated, the transmission parameters are adjusted once based on the currently acquired and updated multiple pose transformation signals and the currently updated captured image, so that the transmission parameters are adjusted multiple times when the first pose transformation signal is acquired and updated multiple times.

6. The projector according to claim 3, characterized in that, The plurality of pose sensors includes a second pose sensor, which is the pose sensor with the lowest signal acquisition frequency among the plurality of pose sensors; during the time period between two adjacent shooting operations of the camera, the second pose sensor acquires and updates the corresponding second pose transformation signal multiple times. The step of adjusting the transmission parameters multiple times based on the pose transformation signal acquired and updated multiple times and the currently updated captured image includes: Each time the second pose transformation signal is acquired and updated, the transmission parameters are adjusted once based on the currently acquired and updated multiple pose transformation signals and the currently updated captured image, so that the transmission parameters are adjusted multiple times when the second pose transformation signal is acquired and updated multiple times.

7. The projector according to any one of claims 3-6, characterized in that, The various posture sensors include a gravity acceleration sensor and a gyroscope sensor, and the posture transformation signal includes the gravity acceleration signal collected by the gravity acceleration sensor and the rotation rate signal collected by the gyroscope.

8. The projector according to any one of claims 1-7, characterized in that, The step of adjusting the emission parameters of the projection component multiple times based on the currently acquired pose transformation signal and the currently updated captured image includes: Depth information about the target projection surface is obtained based on the currently updated captured image; The emission parameters of the projection component are adjusted multiple times based on the currently acquired pose transformation signal and the depth information of the target projection surface.

9. The projector according to claim 8, characterized in that, When the pose of the projector changes, the structured light is also used to display the logo pattern on the target projection surface in real time, and the captured image includes the captured content of the logo pattern. Obtaining depth information about the target projection surface based on the currently updated captured image includes: Based on binocular vision matching technology, depth information about the target projection surface is obtained by comparing the captured content of the sign pattern in the currently updated captured image with the sign pattern.

10. The projector according to claim 9, characterized in that, The logo pattern is displayed outside the imaging area of ​​the user screen content.

11. The projector according to claim 9, characterized in that, The logo pattern is the image content within the user's screen.

12. The projector according to any one of claims 8-11, characterized in that, The step of adjusting the emission parameters of the projection component multiple times based on the currently acquired pose transformation signal and the depth information of the target projection surface includes: The relative pose relationship between the projector and the target projection surface is determined based on the depth information of the target projection surface; The pose information of the projector in the ground coordinate system is dynamically determined based on the currently acquired pose transformation signal; The pose information of the target projection surface in the ground coordinate system is dynamically determined based on the relative positional relationship and the pose information of the projector in the ground coordinate system. The emission parameters of the projection component are adjusted multiple times based on the pose information of the target projection surface in the ground coordinate system.

13. A projection control method, characterized in that, Applied to a projector, the projector including a projection component, a camera, and a pose sensor group, the projection control method includes: The system acquires the user-selected screen content and controls the projection component to emit projection structured light toward the target projection surface. The projection structured light is used to display the user screen content on the target projection surface in real time. The pose sensor group collects the pose transformation signal of the projector in real time, and the pose transformation signal is used to indicate the pose change state of the projector. When the pose of the projector changes, the camera is controlled to continuously capture images of the target projection surface to obtain real-time updated images. The emission parameters of the projection component are adjusted multiple times based on the currently acquired pose transformation signal and the currently updated captured image, so that the imaging range of the user screen content remains rectangular or square. The emission parameters include the emission focal length of the projection structured light and the projection imaging angle.

14. The projection control method according to claim 13, characterized in that, The signal acquisition frequency of the pose sensor group is greater than the shooting frequency of the camera; during the time period between two consecutive shooting operations of the camera, the pose sensor group acquires and updates the pose transformation signal multiple times. The step of adjusting the emission parameters of the projection component multiple times based on the currently acquired pose transformation signal and the currently updated captured image includes: Before the next update of the captured image, the transmission parameters are adjusted multiple times based on the pose transformation signals acquired and updated multiple times and the currently updated captured image.

15. The projection control method according to claim 13 or 14, characterized in that, The step of adjusting the emission parameters of the projection component multiple times based on the currently acquired pose transformation signal and the currently updated captured image includes: Depth information about the target projection surface is obtained based on the currently updated captured image; The emission parameters of the projection component are adjusted multiple times based on the currently acquired pose transformation signal and the depth information of the target projection surface.

16. The projection control method according to claim 15, characterized in that, When the pose of the projector changes, the structured light is also used to display the logo pattern on the target projection surface in real time, and the captured image includes the captured content of the logo pattern. Obtaining depth information about the target projection surface based on the currently updated captured image includes: Based on binocular vision matching technology, depth information about the target projection surface is obtained by comparing the captured content of the sign pattern in the currently updated captured image with the sign pattern.

17. The projection control method according to claim 16, characterized in that, The logo pattern is displayed outside the imaging area of ​​the user screen content.

18. The projection control method according to claim 16, characterized in that, The logo pattern is the image content within the user's screen.

19. The projection control method according to any one of claims 15-18, characterized in that, The step of adjusting the emission parameters of the projection component multiple times based on the currently acquired pose transformation signal and the depth information of the target projection surface includes: The relative pose relationship between the projector and the target projection surface is determined based on the depth information of the target projection surface; The pose information of the projector in the ground coordinate system is dynamically determined based on the currently acquired pose transformation signal; The pose information of the target projection surface in the ground coordinate system is dynamically determined based on the relative positional relationship and the pose information of the projector in the ground coordinate system. The emission parameters of the projection component are adjusted multiple times based on the pose information of the target projection surface in the ground coordinate system.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on a controller, cause the controller to perform the projection control method as described in any one of claims 13-19.

Images