Projection control method of projection system and related device

Abstract

The invention discloses a projection control method of a projection system and a related device, and relates to the technical field of projection system calibration, and the method comprises the steps: obtaining factory calibration data of the projection system and a calibration pattern projected by a projection device at a user side of the projection system, and resolving a sensing device-projection device external parameter matrix of the projection system, and obtaining projection control parameters corresponding to the target projection object. According to the method, system calibration is carried out based on the geometric parameters of the projection system, calibration of the projection system can be carried out in different user side scenes, the projection content can be projected to the target projection object more accurately in different user side scenes, and the method is suitable for projection calibration in a non-fixed user side scene.

Description

A projection control method and related apparatus for a projection system Technical Field

[0001] This application relates to the field of precision projection control technology for projection systems, and in particular to a projection control method and related apparatus for a projection system. Background Technology

[0002] In order to eliminate image distortion caused by physical defects and manufacturing errors in equipment (such as cameras and projectors) and thus establish a precise and quantifiable mathematical relationship between digital images and the real physical world, camera or projector lenses need to be calibrated.

[0003] The goal of camera calibration is to determine the camera's intrinsic and extrinsic parameters. Intrinsic parameters include focal length, principal point, and distortion coefficients, while extrinsic parameters include its position and orientation in the world coordinate system. Zhang Zhengyou's checkerboard calibration method is the most classic and accurate method currently available.

[0004] A projector can be viewed as a "reverse camera." By modeling it as a camera and attempting to obtain the correspondence between pixels in its "image" and world points, the projector can be calibrated. In existing technologies, the projector and camera are typically calibrated together using a structured light system. The principle is that structured light is a light source with a known spatial coding pattern, projecting a series of carefully designed two-dimensional light patterns (such as Gray code or phase-shifted fringes) with completely known coding rules onto the scene (usually a calibration board with feature points). The camera captures the deformed patterns modulated by the calibration board. By decoding these images, the unique pixel coordinates corresponding to each pixel in the camera image can be found in the projector's digital pattern, thus establishing a precise pixel-level correspondence between the camera and the projector. With the correspondence data between projector pixels and world points, the same optimization algorithms used for camera calibration (such as bundled adjustment) can be applied to solve for the projector's intrinsic parameters (focal length, principal point, distortion coefficient) and extrinsic parameters relative to the world coordinate system, thereby completing the projector calibration.

[0005] Existing calibration methods (Zhang Zhengyou method, structural beam calibration) are all used to calibrate equipment parameters in a fixed, laboratory environment, which cannot solve the problem of achieving high-precision spatial pointing in non-fixed (such as different projector installation angles and distances from the projection surface) user-side scenarios. Summary of the Invention

[0006] The purpose of this application is to provide a projection control method and related device for a projection system, which can project the content more accurately to the target object in a user-side scenario, and solve the problem of not being able to achieve high-precision spatial pointing in a non-fixed, user-side scenario.

[0007] To achieve the above objectives, this application provides the following solution: In a first aspect, this application provides a projection control method for a projection system, comprising: acquiring the factory calibration data of the projection system and the calibration pattern projected by the projection device on the user side of the projection system, and calculating the sensor-projection device extrinsic parameter matrix of the projection system; and generating corresponding projection control parameters based on the sensor-projection device extrinsic parameter matrix and the projection characteristics of the target projection object.

[0008] Secondly, this application provides a projection system, including: a control unit and a sensing device and a projection device communicatively connected to the control unit; the sensing device is used to acquire an image of a target projection object; the control unit is used to execute the projection control method of the projection system described above and generate projection control parameters for the target projection object; the projection device is used to execute a projection action according to the projection control parameters.

[0009] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor executes the computer program to implement the projection control method of the projection system described above.

[0010] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the projection control method of the projection system described above.

[0011] According to the specific embodiments provided in this application, this application has the following technical effects: This application provides a projection control method and related apparatus for a projection system. When the projection system is installed in different user-side scenarios, it is necessary to recalibrate the projection system in each user-side scenario. This application uses the factory calibration data of the projection system and the calibration pattern projected by the projection device to calculate the extrinsic parameter matrix of the sensing device-projection device. Finally, this application generates projection control parameters corresponding to the target projection object based on the extrinsic parameter matrix of the sensing device-projection device. This application performs system calibration based on the geometric parameters of the projection system, which can be adapted to the calibration of the projection system in different user-side scenarios. It can project the projection content more accurately to the target projection object in different user-side scenarios, solving the problem of not being able to achieve high-precision spatial pointing in non-fixed, user-side scenarios. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 is a schematic flowchart of a projection control method for a projection system according to an embodiment of this application; Figure 2 is a schematic structural diagram of a computer device according to an embodiment of this application. Detailed Implementation

[0014] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0015] Camera calibration using Zhang Zhengyou's method: The fundamental purpose is to obtain the camera's imaging parameters in order to achieve geometric correction of the acquired images; the calibration object is a single camera; the calibration process is usually carried out in a controlled environment, by observing the physical calibration board through the camera and acquiring calibration images under multiple spatial poses; the camera's intrinsic parameter matrix and distortion coefficients are solved; the calibration accuracy is usually evaluated by the reprojection error on the image plane (pixel-level error of image coordinates).

[0016] Traditional projector calibration using structured light aims to obtain the projector's imaging parameters for geometric correction of the projected image. The calibration object is a single projector, which is geometrically equivalent to a reverse camera. The calibration process is typically performed in a controlled, fixed laboratory environment, observing the structured light pattern projected onto a known calibration plane using a calibrated camera. The projector's intrinsic parameter matrix and distortion coefficients are then solved. Calibration accuracy is usually evaluated using indirect reprojection error (pixel-level error in image coordinates).

[0017] Existing calibration methods cannot solve the problem of achieving high-precision spatial pointing in non-fixed, user-side scenarios. To address this, this application proposes a projection control method for a projection system, which provides a calibration scheme for the projection system: the fundamental purpose is to obtain the projection control parameters of the entire projection pointing system to calibrate the accuracy of its pointing actions. The calibration object is a set of actively cooperating sensing devices and projection devices. The sensing devices acquire visual or spatial information of the target projection object, and the projection devices execute the projection pointing action according to the projection control parameters. The calibration environment includes at least a dynamic, non-fixed scene consistent with the actual usage environment (such as a user's wall). The projection system actively modulates the spatial distribution and / or emission direction of the projected light to achieve alignment with the target projection object. Through the above calibration process, a mapping relationship is established between the sensing space and the projection pointing space. This mapping relationship is used to convert the sensed target information into the projection control parameters of the projection system. The verification standard is the pointing accuracy of the projection system in physical space. Its evaluation is based on the actual spatial information deviation between the projected light field and the target object. This is different from the error evaluation method based on pixel coordinates inside the camera or projector. By applying the calibration method proposed in this application, the system can be flexibly adapted to the calibration of different user-side scenarios, thereby enabling more accurate projection onto the target object.

[0018] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0019] In an exemplary embodiment, as shown in FIG1, a projection control method for a projection system is provided, comprising: S1: acquiring the factory calibration data of the projection system and the calibration pattern projected by the projection device on the user side of the projection system, and calculating the external parameter matrix of the sensing device-projection device of the projection system.

[0020] As an example, sensing devices include, but are not limited to, devices for acquiring two-dimensional image information, such as RGB cameras; and / or three-dimensional measurement sensors for acquiring three-dimensional information, such as depth cameras (ToF / structured light), binocular vision systems, and LiDAR. Projection devices may include actuators for modulating the direction and / or spatial distribution of projected light rays, including but not limited to DLP / LCD / LCoS projectors, laser scanning mechanisms, MEMS mechanisms, and phase modulation mechanisms.

[0021] S2: Generate corresponding projection control parameters based on the extrinsic parameter matrix of the sensing device-projection device and the projection characteristics of the target projection object.

[0022] Projection control parameters are input to the projection device to perform projection operations and project the desired content (such as a specific image, a specific light spot, etc.) onto the target projection object.

[0023] In this application, when the projection system is installed in different user-side scenarios, it is necessary to recalibrate the projection system in each user-side scenario. This application calibrates the user-side system based on the geometric parameters (factory calibration data) of the projection system, which can calibrate the projection system in different user-side scenarios. In this way, the projection content can be projected more accurately onto the target projection object in different user-side scenarios. It is suitable for projection calibration in non-fixed user-side scenarios and solves the problem that high-precision spatial pointing cannot be achieved in non-fixed user-side scenarios.

[0024] In another exemplary embodiment of this application, in step S2, when the sensing device cannot acquire spatial three-dimensional information, a calibration scheme based on the geometric constraints provided by monocular vision and reference objects is adopted. The intrinsic parameter matrix and distortion coefficient of the sensing device and the projection device are calibrated at the factory, and subsequent deployment calibration of the projection system is only required in the actual application environment.

[0025] Phase 1: Factory calibration (1) Factory calibration - Sensing device calibration, taking a camera as an example.

[0026] Calibration objective: Obtain high-precision intrinsic parameter matrix of the camera. And distortion coefficient.

[0027] Method: Standard Zhang Zhengyou method. Multiple images of a physical calibration board are acquired at different locations and angles in a laboratory environment. The calibration board can be a checkerboard, grid, or dot array, etc. The positions of feature points (such as corner points) are extracted from each image. Algorithms (such as the `calibrateCamera` function in OpenCV) are used to analyze the feature points in these images, and the camera's intrinsic parameter matrix is ​​calculated backwards based on the known geometry and structure of the physical calibration board. And distortion coefficients. All subsequent camera images are processed using these parameters to correct distortion, resulting in the distorted pixel coordinates. .

[0028] (2) Factory calibration - Projection device calibration, taking a projector as an example.

[0029] Calibration objective: Obtain the intrinsic parameter matrix of the projector. And distortion coefficient.

[0030] Methods: Use the standard Zhang Zhengyou method (treating the projector as an inverse camera) or the structured light joint calibration method, etc.

[0031] Preparation and Acquisition: Use a high-precision, solid calibration board with known feature points (such as a checkerboard, grid, or dot array) and place it on a stable plane (such as a wall) at the typical working distance of the projection system. Project a known, high-contrast calibration pattern (such as a checkerboard, grid, or dot array) onto this plane. The projector records the pixel coordinates of each feature point (corner point) of the calibration pattern on the projector's digital chip (DMD / LCD / LCOS). The image of the "physical calibration board covered by the calibration pattern" is captured using a calibrated camera, and the pixel coordinates of each feature point (corner point) are identified. .

[0032] Parameter solution: Solving for the camera extrinsic parameter matrix Using the camera intrinsic parameter matrix Based on the corner points of the physical calibration board detected in at least one image, the camera extrinsic parameter matrix relative to the coordinate system of the calibration board is calculated. (i.e., the camera's position and orientation), and then, the distortion-free pixel coordinates By inversely calculating to a planar coordinate system, its three-dimensional coordinates in the calibration plate coordinate system can be obtained. Commonly used solution methods include: spatial transformation models, PnP (Perspective-n-Point) algorithm, stereo vision and machine learning optimization, etc.

[0033] Solving the intrinsic parameter matrix of the projector And distortion coefficients: obtain multiple sets of pixel coordinates With three-dimensional coordinates The corresponding set is obtained, and this data is input into a standard camera calibration algorithm (such as OpenCV's calibrateCamera function) to solve for the projector's intrinsic parameter matrix. And distortion coefficient.

[0034] The second stage: Deployment and calibration in the actual application environment of the projection system. Specifically, when the sensing device cannot acquire spatial three-dimensional information, in step S2, the factory calibration data of the projection system and the calibration pattern projected by the projection device on the user side are acquired, and the external parameter matrix of the sensing device-projection device of the projection system is calculated. Specifically, this includes: (a1) acquiring the internal parameter matrix of the sensing device in the projection system and the image containing the calibration pattern collected by the sensing device after the projection device projects the calibration pattern onto a reference object with known or definite geometric constraints on the user side; the reference object with known or definite geometric constraints on the user side refers to the reference object that exists or can be acquired in the actual use environment of the projection system. The reference object with known or definite geometric constraints can be a solid calibration plate (including checkerboard, grid or dot array, etc.), an attached calibration pattern, AR marker, natural plane, known structural wall, known structural object, etc. Geometric constraints can be planar constraints, known size constraints, spatial point set constraints, known structural model constraints, or other constraints that can establish spatial scale relationships, among which planar constraints are a preferred constraint method.

[0035] A reference object with known or definable geometric constraints is attached to the actual working surface (such as a wall), and the projection system controls the projector to project the calibration pattern, while the camera takes pictures.

[0036] (a2) Based on the intrinsic parameter matrix of the sensing device and the feature points of the reference object with known or definite geometric constraints on the user side in the image containing the calibration pattern, calculate the extrinsic parameter matrix of the sensing device relative to the coordinate system of the reference object (corresponding to...). The reference object coordinate system is a spatial coordinate system established based on the geometric features of the reference object. Its origin and coordinate axis directions can be predefined according to the structural features of the reference object or determined through calculation.

[0037] Using the camera's intrinsic parameter matrix Based on the corner points of a reference object with known or definable geometric constraints detected on the user side in at least one image, the camera extrinsic parameter matrix relative to the current reference object coordinate system is calculated. (i.e., the camera's position and orientation), and then, the distortion-free pixel coordinates By inversely calculating to a planar coordinate system, its three-dimensional coordinates in the current reference object's coordinate system can be obtained. .

[0038] (a3) Based on the external parameter matrix of the sensing device (corresponding to) The spatial information of the calibration pattern projected by the projection device and the coordinate system of the reference object are used to calculate the external parameter matrix of the projection device relative to the coordinate system of the reference object. That is, the position and orientation of the projector relative to the coordinate system of the reference object.

[0039] The computational methods include, but are not limited to, spatial transformation models, PnP (Perspective-n-Point) algorithms, stereo vision, and machine learning optimization.

[0040] (a4) Based on the extrinsic parameter matrix of the sensing device and projection device extrinsic matrix Calculate the first extrinsic parameter matrix R|t of the sensing device-projection device, which is used to realize the spatial coordinate transformation from the sensing device coordinate system to the projection device coordinate system.

[0041] Specifically, by analyzing the extrinsic parameter matrix of the sensing device Perform matrix inversion and compare it with the extrinsic matrix of the projection device. The first extrinsic parameter matrix R|t of the sensing device-projection device is obtained by performing matrix multiplication.

[0042] In another exemplary embodiment of this application, when the sensing device cannot acquire spatial three-dimensional information, in step S2, the corresponding projection control parameters are generated based on the extrinsic parameter matrix of the sensing device and the projection characteristics of the target projection object. Specifically, this includes: (b1) based on the geometric constraints provided by the reference object, transforming the projection characteristics of the target projection object into the sensing device coordinate system according to the intrinsic parameter matrix of the sensing device and the extrinsic parameter matrix of the sensing device relative to the coordinate system of the reference object, thereby obtaining the three-dimensional coordinates of the projection characteristics of the target projection object in the sensing device coordinate system (corresponding to...). ).

[0043] The camera captures an image of the target projection object, identifies the outline of the target projection object (such as a picture frame), and obtains the distortion-free pixel coordinates of its feature points (such as corner points). Recognition method: The system can identify the outline of the target object and obtain the distortion-free pixel coordinates of key points (corners) through image processing. Methods include, but are not limited to, frame difference method, traditional image processing algorithm or deep learning segmentation model (such as YOLOV8-SEG).

[0044] Coordinates of each distortion-reduced pixel Combined with camera intrinsic parameter matrix Based on the geometric constraints provided by the reference object, the three-dimensional coordinates of the corner points in the coordinate system of the sensing device are calculated in reverse. .

[0045] (b2) According to the first extrinsic parameter matrix of the sensing device-projection device in the sensing device-projection device extrinsic parameter matrix The three-dimensional coordinates of the projection features of the target object in the coordinate system of the sensing device. Transform to the projection device coordinate system to obtain the three-dimensional coordinates of the projection features of the target object in the projection device coordinate system. .

[0046] (b3) Using the intrinsic parameter matrix of the projection device The distortion coefficient is used to correct the projection characteristics of the target object in the three-dimensional coordinates of the projection device coordinate system.

[0047] Using the projector's intrinsic parameter matrix Three-dimensional coordinates in the projector coordinate system The projection is used as the image coordinates, and the distortion coefficient of the projector is applied for reverse pre-correction to generate the final projected image.

[0048] (b4) Generate projection control parameters based on the feature point coordinates of the projection correction.

[0049] In another exemplary embodiment of this application, in step S2, when the sensing device can acquire spatial three-dimensional information, the sensing device has both image acquisition function and spatial three-dimensional information acquisition function. The sensing device includes an image acquisition component (such as a camera) and a three-dimensional information acquisition component (such as a three-dimensional measurement sensor). As an example, when the sensing device has the function of acquiring spatial three-dimensional information, a calibration scheme integrating a three-dimensional measurement sensor (such as a depth camera, binocular vision system, or LiDAR) is adopted. Similarly, the sensing device and projection device have already undergone intrinsic parameter matrix calibration before leaving the factory, and deployment calibration is performed in the actual application environment.

[0050] Phase 1: Factory calibration (1) Camera calibration: Obtaining the camera intrinsic parameter matrix And distortion coefficients; refer to the previous content for details. Camera intrinsic parameter matrix. The distortion coefficients are used to calculate the camera's distorted pixel coordinates, and participate in the calculation of projector intrinsic parameters and distortion coefficients (as described above). They also participate in the 3D measurement sensor-camera extrinsic parameter matrix during the joint calibration of the 3D measurement sensor-camera system. The calculation.

[0051] (2) Projector calibration: Obtain the projector intrinsic parameter matrix And distortion coefficients; refer to the aforementioned content for details. Projector intrinsic parameter matrix. The distortion coefficients are mainly used for solving the projector's extrinsic parameters and projection correction during the subsequent deployment and calibration phase.

[0052] (3) Joint calibration of three-dimensional measurement sensor and camera: 1) Internal calibration: Calibrate the internal parameters of the three-dimensional measurement sensor to ensure the accuracy of its three-dimensional information measurement values.

[0053] 2) Camera-3D Measurement Sensor Joint Calibration: By jointly observing a specific calibration scene, the extrinsic parameter matrix of the 3D measurement sensor and camera is accurately solved. (Relative pose parameters between the camera and the 3D measurement sensor). 3D measurement sensor-camera extrinsic parameter matrix. This is used to accurately map 3D point clouds measured by 3D measurement sensors to the camera coordinate system. Common solution methods include spatial transformation models, PnP (Perspective-n-Point) algorithms, stereo vision, estimation based on 3D information, and machine learning optimization.

[0054] As an example, the 3D measurement sensor can be a ToF sensor, a binocular vision sensor, a structured light sensor, a LiDAR, or a combination thereof, and the corresponding joint calibration process can be implemented based on active projection patterns, passive feature matching, or depth-image fusion.

[0055] Phase 2: Deployment and Calibration: Using 3D measurement sensors, the projection system is calibrated automatically in the end-use environment, typically without the need for a reference object with known or definable geometric constraints.

[0056] Specifically, in step S2, the factory calibration data of the projection system and the calibration pattern projected by the projection device on the user side are obtained, and the external parameter matrix of the sensing device-projection device of the projection system is calculated. Specifically, it includes: (c1) After the calibration pattern (two-dimensional or three-dimensional calibration pattern, such as checkerboard, grid, dot array, cube array or other three-dimensional structure array) is projected by the projection device on the user side, it is observed by the camera and the three-dimensional measurement sensor in the sensing device. That is, the camera collects the image containing the calibration pattern and the three-dimensional information of the calibration pattern is scanned by the three-dimensional measurement sensor.

[0057] When a sensing device can acquire three-dimensional spatial information, it generally does not require a reference object with known or determinable geometric constraints. The system controls the projector to project the calibration pattern onto the target projection object (such as a wall or other projection surface / spatial area), based on the camera intrinsic parameter matrix. And distortion coefficients, 3D measurement sensor-camera extrinsic matrix The system combines images captured by the camera with 3D information provided by a 3D measurement sensor and maps them onto the coordinate system of the sensing device to form a 3D point cloud. These 3D point clouds are then filtered or fitted to determine the primary workspace, thereby selecting the effective set of points close to the target projection area for subsequent extrinsic parameter matrix calculations.

[0058] The point cloud obtained by the 3D measurement sensor is distributed across the entire field of view and may not be strictly located within the target projection area. For example, walls may have uneven surfaces or decorative elements; the space occupied by desktops, floors, or other objects may also appear within the scanning range. Directly using the entire point cloud for extrinsic parameter calculations will introduce noise and geometric errors. Filtering or fitting the main working space is a spatial filtering / noise reduction operation that ensures the 3D point set used to calculate the extrinsic parameter matrix of the sensing device-projection device matches the actual position of the projected pattern, thereby improving projection accuracy.

[0059] (c2) Based on the image containing the calibration pattern collected by the sensing device, the scanned 3D information, and the factory calibration data of the projection system (including the intrinsic parameter matrix and distortion coefficients of the projection device), the second extrinsic parameter matrix of the sensing device-projection device is derived. .

[0060] According to the intrinsic parameter matrix of the projection device The distortion coefficients are used to establish a correspondence between the 3D point cloud mapped to the sensing device coordinate system and the calibration pattern points in the projector coordinate system. The extrinsic parameter matrix of the sensing device and the projector is then calculated using rigid transformation or optimization algorithms (including spatial transformation models based on minimizing reprojection errors or machine learning methods). .

[0061] By analyzing the data jointly observed by the camera and the 3D measurement sensor, the camera-projector extrinsic parameter matrix (corresponding to) is automatically calibrated in the deployment environment. This is used to transform 3D points in the camera coordinate system to the projector coordinate system.

[0062] Camera-projector extrinsic matrix The solution process can be jointly calculated using optimization algorithms, including but not limited to spatial transformation models based on geometric constraints, optimization methods based on minimizing reprojection errors, or introducing machine learning models to adaptively fine-tune the calibration parameters.

[0063] In another exemplary embodiment of this application, when the sensing device can acquire spatial three-dimensional information, in step S2, the corresponding projection control parameters are generated according to the external parameter matrix of the sensing device-projection device and the projection characteristics of the target projection object. Specifically, this includes: (d1) acquiring the projection characteristics of the target projection object and the three-dimensional information of the projection characteristics of the target projection object collected by the sensing device.

[0064] (d2) Based on the projection features and three-dimensional information of the target projection object collected by the sensing device, the projection features of the target projection object are transformed into the sensing device coordinate system (camera coordinate system) by applying the factory calibration data of the projection system (external parameter matrix of three-dimensional measurement sensor-camera) to obtain the three-dimensional coordinates of the projection features of the target projection object in the sensing device coordinate system.

[0065] The camera captures the target projection object, identifies key points of the target projection object (such as corner points of the frame outline), and combines this with 3D information measured by a 3D measurement sensor (specifically, if the 3D measurement sensor is a depth sensor, then the depth value). ) and 3D measurement sensor-camera extrinsic matrix Mapping key points to 3D coordinates in the camera coordinate system , defined relative to the camera optical center and coordinate axes.

[0066] 3D coordinates The acquisition process can be performed point-by-point for a single key point, or in parallel batch calculation for multiple key points, and the calculation process can be executed in real-time or non-real-time mode.

[0067] (d3) According to the second extrinsic parameter matrix of the sensing device-projection device in the sensing device-projection device extrinsic parameter matrix The projection control parameters are derived from the intrinsic parameter matrix of the projection device and the three-dimensional coordinates of the projection characteristics of the target object in the coordinate system of the sensing device.

[0068] Using the camera-projector extrinsic parameter matrix obtained from deployment calibration (corresponding to (This refers to the three-dimensional coordinates of the corner points of the target object in the camera coordinate system.) Transform to 3D coordinates in the projector coordinate system .

[0069] Using the projector's intrinsic parameter matrix And distortion coefficients, for three-dimensional coordinates in the projector coordinate system Perform projection correction, generate the final projected image, and execute the projection.

[0070] In another exemplary embodiment of this application, in step (d3), the second extrinsic parameter matrix of the sensing device-projection device is determined according to the extrinsic parameter matrix of the sensing device-projection device. The projection control parameters are derived from the intrinsic parameter matrix of the projection device and the three-dimensional coordinates of the projection characteristics of the target object in the coordinate system of the sensing device. Specifically, this includes: (d31) deriving the corresponding projection control parameters based on the second extrinsic parameter matrix of the sensing device-projection device. The three-dimensional coordinates of the projection features of the target object in the sensing device coordinate system are transformed to the projection device coordinate system, thus obtaining the three-dimensional coordinates of the projection features of the target object in the projection device coordinate system.

[0071] (d32) Using the intrinsic parameter matrix and distortion coefficient of the projection device, the projection characteristics of the target object are projected in the three-dimensional coordinates of the projection device coordinate system for projection correction.

[0072] (d33) Generate projection control parameters based on the feature point coordinates of projection correction.

[0073] In another exemplary embodiment of this application, after the projection operation is performed according to the projection control parameters, the projection control method of the projection system further includes: (e1) using a sensing device to acquire a projected image containing the target projection object and the current projection content.

[0074] (e2) Determine the deviation between the feature points of the target projected object in the projected image and the aligned feature points in the current projected content; the aligned feature points are feature points determined in the projected content based on the feature points of the target projected object.

[0075] (e3) Determine whether the current iteration meets the preset iteration convergence condition. As an example, the preset iteration convergence condition can be that the deviation is less than a preset value, or that the maximum number of iterations has been reached.

[0076] (e4) If not, adjust the current projection control parameters according to the deviation, and use the adjusted projection control parameters as the current projection control parameters.

[0077] (e5) Control the projection device to perform projection operation according to the current projection control parameters; and return to the step "Use the sensing device to collect the projected image containing the target projection object and the current projection content" until the current iteration meets the preset iteration convergence condition.

[0078] In this embodiment, the calibration results (sensing device-projection device extrinsic parameter matrix) on the deployment side are obtained by calibration methods based on the geometric parameters (factory calibration data) of the projection system. After determining the projection control parameters and performing the projection operation based on the sensing device-projection device extrinsic parameter matrix, in order to ensure that the actual projection effect is more accurate, the projection alignment closed-loop feedback method can be further applied for adjustment.

[0079] In another exemplary embodiment of this application, a projection system is provided, including: a control unit and a sensing device and a projection device communicatively connected to the control unit.

[0080] A sensing device is used to acquire images of a projected target object. The sensing device has image acquisition capabilities and can also acquire three-dimensional information (such as depth information) from the image.

[0081] The control unit is used to execute the projection control method of the projection system and generate projection control parameters for the target projection object.

[0082] A projection device used to perform projection actions according to projection control parameters.

[0083] In an exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram is shown in Figure 2. The computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is connected to the system bus via the I / O interfaces. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer device stores relevant data required for the projection control process. The I / O interfaces of the computer device are used for exchanging information between the processor and external devices. The communication interface of the computer device is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a projection control method for a projection system.

[0084] Those skilled in the art will understand that the structure shown in Figure 2 is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. A specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0085] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0086] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0087] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of the relevant data are carried out in compliance with the relevant data protection laws and policies of the country where the location is located, and with the authorization granted by the owner of the corresponding device.

[0088] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0089] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0090] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0091] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A projection control method for a projection system, characterized in that, include: The system acquires the factory calibration data of the projection system and the calibration pattern projected by the projection device on the user side, and calculates the external parameter matrix of the sensing device-projection device of the projection system; and generates the corresponding projection control parameters based on the external parameter matrix of the sensing device-projection device and the projection characteristics of the target object.

2. The projection control method of the projection system according to claim 1, characterized in that, When the sensing device cannot acquire spatial three-dimensional information, the system acquires the factory calibration data of the projection system and the calibration pattern projected by the projection device on the user side, and calculates the sensing device-projection device extrinsic parameter matrix of the projection system. Specifically, this includes: acquiring the intrinsic parameter matrix of the sensing device in the projection system and the image containing the calibration pattern acquired by the sensing device after the projection device projects the calibration pattern onto a reference object with known or definite geometric constraints on the user side; the reference object with known or definite geometric constraints on the user side refers to a reference object that exists or can be acquired in the actual use environment of the projection system; based on the intrinsic parameter matrix of the sensing device and the feature points of the reference object with known or definite geometric constraints on the user side in the image containing the calibration pattern, calculating the sensing device extrinsic parameter matrix of the sensing device relative to the reference object coordinate system; calculating the projection device extrinsic parameter matrix of the projection device relative to the reference object coordinate system based on the sensing device extrinsic parameter matrix and the spatial information of the calibration pattern projected by the projection device; calculating the first extrinsic parameter matrix of the sensing device-projection device based on the sensing device extrinsic parameter matrix and the projection device extrinsic parameter matrix; the sensing device-projection device extrinsic parameter matrix includes the first extrinsic parameter matrix of the sensing device-projection device.

3. The projection control method for the projection system according to claim 2, characterized in that, The corresponding projection control parameters are generated based on the extrinsic parameter matrix of the sensing device-projection device and the projection characteristics of the target projection object. Specifically, this includes: based on the geometric constraints provided by the reference object, and according to the intrinsic parameter matrix of the sensing device and the extrinsic parameter matrix of the sensing device relative to the reference object coordinate system, transforming the projection characteristics of the target projection object into the sensing device coordinate system to obtain the three-dimensional coordinates of the projection characteristics of the target projection object in the sensing device coordinate system; based on the first extrinsic parameter matrix of the sensing device-projection device in the extrinsic parameter matrix, transforming the three-dimensional coordinates of the projection characteristics of the target projection object in the sensing device coordinate system into the projection device coordinate system to obtain the three-dimensional coordinates of the projection characteristics of the target projection object in the projection device coordinate system; using the intrinsic parameter matrix and distortion coefficients of the projection device, performing projection correction on the three-dimensional coordinates of the projection characteristics of the target projection object in the projection device coordinate system; and generating projection control parameters based on the feature point coordinates of the projection correction.

4. The projection control method for the projection system according to claim 1, characterized in that, When the sensing device can acquire spatial three-dimensional information, it acquires the factory calibration data of the projection system and the calibration pattern projected by the projection device on the user side, and calculates the sensing device-projection device extrinsic parameter matrix of the projection system. Specifically, this includes: after projecting the calibration pattern on the user side using the projection device, the sensing device acquires an image containing the calibration pattern and scans the three-dimensional information of the calibration pattern; based on the image containing the calibration pattern acquired by the sensing device, the scanned three-dimensional information, and the factory calibration data of the projection system, the sensing device-projection device second extrinsic parameter matrix is ​​derived; the sensing device-projection device extrinsic parameter matrix includes the sensing device-projection device second extrinsic parameter matrix.

5. The projection control method for the projection system according to claim 4, characterized in that, The corresponding projection control parameters are generated based on the extrinsic parameter matrix of the sensing device-projection device and the projection characteristics of the target projection object. Specifically, this includes: acquiring the projection characteristics and three-dimensional information of the target projection object collected by the sensing device; transforming the projection characteristics of the target projection object into the sensing device coordinate system based on the projection characteristics and three-dimensional information of the target projection object collected by the sensing device, and applying the factory calibration data of the projection system, to obtain the three-dimensional coordinates of the projection characteristics of the target projection object in the sensing device coordinate system; and obtaining the corresponding projection control parameters based on the second extrinsic parameter matrix of the sensing device-projection device, the intrinsic parameter matrix of the projection device, and the three-dimensional coordinates of the projection characteristics of the target projection object in the sensing device coordinate system.

6. The projection control method for the projection system according to claim 5, characterized in that, Based on the second extrinsic parameter matrix of the sensing device-projection device, the intrinsic parameter matrix of the projection device, and the three-dimensional coordinates of the projection features of the target object in the sensing device coordinate system, the corresponding projection control parameters are derived. Specifically, this includes: transforming the three-dimensional coordinates of the projection features of the target object in the sensing device coordinate system to the projection device coordinate system based on the second extrinsic parameter matrix of the sensing device-projection device, thereby obtaining the three-dimensional coordinates of the projection features of the target object in the projection device coordinate system; using the intrinsic parameter matrix and distortion coefficients of the projection device, performing projection correction on the three-dimensional coordinates of the projection features of the target object in the projection device coordinate system; and generating projection control parameters based on the feature point coordinates of the projection correction.

7. The projection control method for the projection system according to claim 1, characterized in that, After the projection operation is executed according to the projection control parameters, the projection control method of the projection system further includes: acquiring a projected image containing the target projection object and the current projection content using a sensing device; determining the deviation between the feature points of the target projection object in the projected image and the alignment feature points in the current projection content; the alignment feature points are feature points determined in the projection content based on the feature points of the target object; determining whether the current iteration meets the preset iteration convergence condition; if not, adjusting the current projection control parameters according to the deviation, and using the adjusted projection control parameters as the current projection control parameters; controlling the projection device to execute the projection operation according to the current projection control parameters; and returning to the step "acquiring a projected image containing the target projection object and the current projection content using a sensing device" until the current iteration meets the preset iteration convergence condition.

8. A projection system, characterized in that, include: A control unit and a sensing device and a projection device communicatively connected to the control unit; the sensing device is used to acquire an image of the target projection object; the control unit is used to execute the projection control method of the projection system according to any one of claims 1 to 7 and generate projection control parameters of the target projection object; the projection device is used to execute a projection action according to the projection control parameters.

9. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that the processor executes the computer program to implement the projection control method of the projection system according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the projection control method of the projection system according to any one of claims 1-7.

Images