Projection correction methods, devices, electronic equipment and storage media

Abstract

This application discloses a projection correction method, apparatus, electronic device, and storage medium. The projection correction method, apparatus, electronic device, and storage medium of this application acquire images captured by a camera, including a projection device and the projected image. They can identify the projection device and the projected image in the projection scene image, and calculate the relative positional relationship between the camera and the projection device based on the known three-dimensional information of the projection device. Furthermore, based on the relative positional relationship, they calculate the three-dimensional information of the projected image. Thus, correction parameters for the projected image can be generated based on the three-dimensional information of the projected image. Then, the projection device is controlled to project a distorted image according to the correction parameters. This allows the projection device to correct the edge contours of the projected image to a preset shape, thereby achieving projection image correction without requiring additional devices such as cameras, reducing the cost of projection correction.

Description

Technical Field

[0001] This application belongs to the field of projection technology, and in particular relates to a projection correction method, device, electronic device and storage medium. Background Technology

[0002] The projected image from a projection device is affected by the device's placement, and its shape may be trapezoidal. To correct the projected image to a rectangle, existing technologies typically employ manual or automatic keystone correction methods. Manual keystone correction involves the user manually adjusting the projection device by inputting the projection tilt angle or manually adjusting the final displayed image shape, which is complex. Existing automatic keystone correction technologies usually add devices such as a Time-of-Flight (TOF) camera and projection light structure to the projection device. This allows for 3D reconstruction using the projected light, automatically measuring the tilt angle of the projected image to achieve automatic keystone correction. However, this method requires additional components, has a complex manufacturing process, and is costly. Therefore, a lower-cost projection image correction method is urgently needed. Summary of the Invention

[0003] This application provides a projection image correction method, apparatus, device, and storage medium, which can reduce the cost of projection correction.

[0004] On one hand, embodiments of this application provide a projection correction method, the method comprising:

[0005] Acquire the projection scene image captured by the camera; wherein the projection scene image includes the projection device and the projection screen displayed by the projection device projecting the projected image onto the projection plane;

[0006] Identify projection devices in projection scene images;

[0007] Based on the known three-dimensional information of the projection device, calculate the relative positional relationship between the camera and the projection device;

[0008] Identify the projected image in the projected scene image;

[0009] Calculate the three-dimensional information of the projected image by combining the relative positional relationships;

[0010] Based on the three-dimensional information of the projected image, correction parameters for the projected image are generated; the correction parameters are used to deform the projected image so that the edge contour of the projected image is corrected to a preset shape.

[0011] The projection equipment is controlled to project the distorted image according to the correction parameters.

[0012] Optionally, identifying the projection device in the projection scene image includes:

[0013] Obtain a 3D model of the projection device;

[0014] Extract appearance features from the 3D model and the projected scene image respectively;

[0015] Feature matching is performed on the appearance features extracted from the 3D model and the projected scene image to obtain the feature matching results;

[0016] Based on the feature matching results, the appearance features of the projection device that match the appearance features extracted from the 3D model are determined in the projection scene image.

[0017] Optionally, appearance features are extracted from the 3D model, including:

[0018] Rotate the 3D model to different angles to obtain 2D images at the corresponding angles;

[0019] Extract appearance features from the two-dimensional image at each angle;

[0020] Perform feature matching on the appearance features extracted from the 3D model and the projected scene image, including:

[0021] Feature matching is performed on the appearance features extracted from the 2D image at each angle and the appearance features extracted from the projected scene image.

[0022] Optionally, before calculating the relative positional relationship between the camera and the projection device by combining the known three-dimensional information of the projection device, the method further includes:

[0023] Based on the known 3D coordinates of the 3D model, obtain the 3D coordinates of the appearance features in the 3D model that match the projected scene image;

[0024] Based on the known 3D information of the projection device, calculate the relative positional relationship between the camera and the projection device, including:

[0025] By combining the 3D coordinates of the appearance features in the 3D model that match the projected scene image, the camera extrinsic parameters relative to the projection device are calculated.

[0026] Optionally, the projected image includes preset pattern features to identify the projected image in the projected scene image, including:

[0027] Identify preset pattern features in the projected scene image;

[0028] Based on the relative positional relationships, calculate the three-dimensional information of the projected image, including:

[0029] Calculate the three-dimensional coordinates of the preset pattern features by combining the relative positional relationships;

[0030] Based on the three-dimensional coordinates of the preset pattern features and their positions in the projected image, calculate the three-dimensional coordinates of the four corner points of the projected image.

[0031] Optionally, correction parameters for the projected image are generated based on the three-dimensional information of the projected image, including:

[0032] Calculate the maximum inscribed rectangle of the four corner points of the projected image to obtain the corrected projection area.

[0033] Based on the transformation relationship between the four corner points of the projected image and the four corner points of the projection area, the correction parameters for deforming the projected image are calculated.

[0034] On the other hand, embodiments of this application provide a projection correction device, which includes:

[0035] The acquisition unit is used to acquire the projection scene image captured by the camera; wherein the projection scene image includes the projection device and the projection screen displayed by the projection device projecting the projected image onto the projection plane;

[0036] The first identification unit is used to identify the projection device in the projection scene image;

[0037] The first calculation unit is used to calculate the relative positional relationship between the camera and the projection device by combining the known three-dimensional information of the projection device.

[0038] The second recognition unit is used to recognize the projected image in the projected scene image;

[0039] The second calculation unit is used to calculate the three-dimensional information of the projected image by combining the relative positional relationships;

[0040] The generation unit is used to generate correction parameters for the projected image based on the three-dimensional information of the projected image; wherein, the correction parameters are used to deform the projected image so that the edge contour of the projected image is corrected to a preset shape.

[0041] The control unit is used to control the projection device to project the distorted image according to the correction parameters.

[0042] Optionally, the first identification unit includes:

[0043] Acquire sub-units to acquire the 3D model of the projection device;

[0044] The first extraction subunit is used to extract appearance features from the 3D model and the projected scene image, respectively.

[0045] The feature matching subunit is used to perform feature matching on the appearance features extracted from the 3D model and the projected scene image to obtain the feature matching result.

[0046] The sub-unit is determined to identify the appearance features of the projection device in the projection scene image that match the appearance features extracted from the 3D model, based on the feature matching results.

[0047] Optionally, the first extraction subunit includes:

[0048] Rotation sub-units are used to rotate a 3D model to different angles to obtain a 2D image at the corresponding angle.

[0049] The second extraction subunit is used to extract features from the two-dimensional image at each angle.

[0050] The feature matching subunit is also used to perform feature matching on the appearance features extracted from the two-dimensional image at each angle and the appearance features extracted from the projected scene image.

[0051] Optionally, the acquisition unit is also used to acquire the three-dimensional coordinates of the appearance features in the three-dimensional model that match the projected scene image, based on the known three-dimensional coordinates of the three-dimensional model, before calculating the relative positional relationship between the camera and the projection device by combining the known three-dimensional information of the projection device.

[0052] The first computing unit is also used to calculate the camera extrinsic parameters of the camera relative to the projection device by combining the three-dimensional coordinates of the appearance features in the three-dimensional model that match the image of the projected scene.

[0053] Optionally, the projected image includes preset pattern features, and the second recognition unit is also used to recognize the preset pattern features in the projected scene image;

[0054] The second calculation unit is also used to calculate the three-dimensional coordinates of the preset pattern feature by combining the relative positional relationship; and to calculate the three-dimensional coordinates of the four corner points of the projected image based on the three-dimensional coordinates of the preset pattern feature and the position of the preset pattern feature in the projected image.

[0055] Optionally, the generating unit includes:

[0056] The first calculation subunit is used to calculate the maximum inscribed rectangle of the four corner points of the projected image, and obtain the projected area after the projected image is corrected.

[0057] The second calculation subunit is used to calculate the correction parameters for deforming the projected image based on the transformation relationship between the four corner points of the projected image and the four corner points of the projection area.

[0058] In another aspect, embodiments of this application provide an electronic device, which includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the projection correction method provided in embodiments of this application.

[0059] In another aspect, embodiments of this application provide a storage medium storing computer program instructions, which, when executed by a processor, implement the projection correction method provided in embodiments of this application.

[0060] The projection correction method, apparatus, electronic device, and storage medium of this application embodiment acquire images captured by a camera, including the projection device and the projected image, and can identify the projection device and the projected image in the projection scene image. By combining the known three-dimensional information of the projection device, the relative positional relationship between the camera and the projection device is calculated. Then, based on the relative positional relationship, the three-dimensional information of the projected image is calculated. Thus, correction parameters for the projected image can be generated based on the three-dimensional information of the projected image. Then, the projection device is controlled to project the distorted projected image according to the correction parameters. This allows the projection device to correct the edge contour of the projected image to a preset shape according to the correction parameters, thereby realizing the correction of the projected image without the need to add additional devices such as cameras to the projection device, thus reducing the cost of projection correction. Attached Figure Description

[0061] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 This is a schematic flowchart of a projection correction method provided in one embodiment of this application;

[0063] Figure 2 This is a schematic diagram of a scenario for a projection correction method provided in one embodiment of this application;

[0064] Figure 3 This is a schematic diagram of the projection screen of a projection correction method provided in one embodiment of this application;

[0065] Figure 4 This is a schematic diagram of the projection correction device provided in another embodiment of this application;

[0066] Figure 5 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application. Detailed Implementation

[0067] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0068] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0069] To address the problems of the prior art, embodiments of this application provide a projection correction method, apparatus, electronic device, and storage medium. The projection correction method provided in this application embodiment will be described first below.

[0070] Figure 1 A schematic flowchart of a projection correction method provided in one embodiment of this application is shown. Figure 1 As shown, it includes the following steps:

[0071] Step 101: Obtain the projected scene image captured by the camera.

[0072] The camera is either a camera configured by the executor of this application embodiment or a camera capable of communicating with the executor of this application embodiment. The projected scene image captured by the camera includes a projection device and a projected image displayed on a projection plane by the projection device projecting the projected image. For example, the camera can be a built-in camera of a mobile phone, which runs an application that can control the projection device. The user triggers the projection device to perform keystone correction by operating the application's page. For example, the user can, according to the instructions on the application's page, capture a projected scene image for keystone correction. After capturing the image, the user can trigger the projection device to perform keystone correction by communicating with the projection device.

[0073] For example, such as Figure 2 As shown, the camera 203 can capture the projection device 202 itself, as well as the projected image projected by the projection device 202 onto the wall 201.

[0074] Step 102: Identify the projection device in the projection scene image.

[0075] Projection devices can be identified using known appearance features, such as color and shape. For example, by extracting lines, circles, corners, etc., and combining this with the color of the projection device, specific edges, buttons, corners, and other appearance features of the projection device can be identified in the projected scene image.

[0076] Step 103: Calculate the relative positional relationship between the camera and the projection device based on the known three-dimensional information of the projection device.

[0077] The 3D information of the projection device is known; its 3D information in 3D space is based on its coordinates in the world coordinate system. In one example, the 3D information of the projection device could be the dimensional design parameters of its appearance, which can be obtained through a 3D model of the projection device.

[0078] The projection scene image includes the projection device captured from the camera's perspective. After identifying the image pixel coordinates of the projection device in the projection scene image, and combining this with the known 3D information of the projection device based on the world coordinate system of the projection device, the relative positional relationship between the camera and the projection device can be calculated.

[0079] Specifically, the relative positional relationship between the camera and the projection device can include the coordinate matrix T and the attitude matrix R of the camera relative to the projection device's coordinate system, where the coordinate matrix T and the attitude matrix R are the camera's extrinsic parameters. With the camera's intrinsic parameters (i.e., known) pre-calibrated, the camera's extrinsic parameters can be solved by establishing equations based on the image pixel coordinates and the known three-dimensional information of the projection device.

[0080] The camera's internal parameters include focal lengths Fx and Fy, principal point positions Cx and Cy, and distortion parameters. These parameters can be calibrated at the factory stage, or the focal length, principal point position, and distortion parameters can be directly given based on the theoretical design values ​​of the optical components, or they can be calibrated after leaving the factory. Specifically, calibration can be obtained by taking pictures of a calibration object in the environment from different angles and calculating the results. The calibration object can include multiple calibration feature points with known relative positional relationships. For example, the calibration object can be a classic black and white checkerboard calibration board.

[0081] For example, this can be achieved using the solvePnP algorithm function in OpenCV. Its inputs are the image pixel coordinates of each feature point of the projection device in the two-dimensional image (projection scene image) and the three-dimensional coordinates of each feature point of the projection device. The calculation results are the coordinate matrix T and the attitude matrix R of the camera relative to the coordinate system of the projection device, where the coordinate matrix T and the attitude matrix R are the camera extrinsic parameters of the camera.

[0082] In order to match the appearance features of the projection device identified in the projection scene image with the known three-dimensional information of the projection device, feature matching can be performed between the appearance features identified in the projection scene image and the two-dimensional image of the projection device to determine which appearance feature in the two-dimensional image matches the appearance feature (e.g., feature point) in the projection scene image. Then, the known three-dimensional coordinates of the appearance feature matched in the two-dimensional image are the three-dimensional coordinates of the corresponding appearance feature in the projection scene image.

[0083] Optionally, the two-dimensional image of the projection device can be pre-calibrated and have its three-dimensional information of appearance features pre-saved, or it can be a two-dimensional image generated based on the three-dimensional model of the projection device. For example, a three-dimensional CAD model of the projection device can be obtained, in which the three-dimensional coordinates of each point are known. By rotating the three-dimensional CAD model to different angles, two-dimensional images of the projection device at different angles can be obtained, and the three-dimensional coordinates of each appearance feature displayed in the two-dimensional image are known.

[0084] In one example, the appearance features of the projection device identified in the projection scene image can be feature points. The correspondence between these feature points in the projection scene image and the feature points of the projection device with known 3D coordinates can be established through feature point matching. Feature point matching can utilize algorithms such as SIFT and SURF; alternatively, it can employ deep learning algorithms. A trained neural network model, based on two input images, the extracted feature point descriptors, and the 3D coordinates of the feature points, outputs the feature point matching results.

[0085] Alternatively, deep learning algorithms can also be used to directly calculate camera extrinsic parameters. That is, by using a trained neural network model, based on the two input images, the extracted feature descriptors, and the three-dimensional coordinates of the features, the camera extrinsic parameters can be output.

[0086] Step 104: Identify the projected image in the projected scene image.

[0087] The projected image usually has certain boundaries with the surrounding scene. For example, the projected image onto the projection plane can be an irregular quadrilateral, such as... Figure 3As shown, the projected image can be determined by identifying the location of the quadrilateral in the projected scene image. For example, by extracting straight line features from the projected scene image, four straight lines that can form a closed quadrilateral can be obtained, thereby determining the area of ​​the projected image.

[0088] Alternatively, since the brightness within the projected image is usually different from the surrounding brightness, the area of ​​the projected image can be determined by setting a pixel threshold to perform two-dimensional threshold segmentation of the projected scene image.

[0089] In another example, identifying the projected image can also mean identifying preset pattern features within the projected image. Since the projected image is the image projected by the projection device onto the projection plane, if the projected image includes preset pattern features, the projected image also includes the projection of the preset pattern features from the projected image.

[0090] For example, the projected image can be as follows Figure 3 As shown, the projected image contains two crosses. Therefore, by extracting features such as straight lines and corner points, the preset pattern features in the image can be extracted. Then, multiple preset pattern features that can be combined to form a cross can be determined, and the positioning of the projected image can be obtained.

[0091] It should be noted that step 104 can be executed before step 103, but does not necessarily have to be executed after step 103.

[0092] Step 105: Calculate the three-dimensional information of the projected image based on the relative positional relationships.

[0093] Based on the relative positional relationship, the camera's external parameters can be determined. The camera's internal parameters are already calibrated. Since the camera's internal and external parameters are known, the three-dimensional coordinates can be obtained by using triangulation to reconstruct the three-dimensional coordinates from the pixel coordinates in the image. Thus, the three-dimensional information of the projected image can be calculated based on the pixel coordinates of the projected image in the projected scene image.

[0094] Step 106: Generate correction parameters for the projected image based on the three-dimensional information of the projected image.

[0095] In this embodiment, the projection device corrects the projected image by performing an affine transformation on the projected image. That is, the projection device corrects the projected image digitally. Correction parameters are used to deform the projected image so that the edge contours of the projected image are corrected to a preset shape.

[0096] The default shape is usually rectangular. However, since the projection device is not calibrated, the projected image is usually an irregular quadrilateral. Therefore, it is necessary to correct the projected image from an irregular quadrilateral to a rectangle. The calibration parameters are used to deform the projected image so that the outline of the projected image is corrected to the default shape.

[0097] When generating correction parameters for the projected image based on the normalized 3D information of the projected image, the angle of the projection plane in physical space (the coordinate system of the electronic device taking the picture) can be determined using the normalized 3D information of the projected image. Specifically, the angle can be represented by a normal vector. Furthermore, based on the angle of the projection plane in physical space, the normalized 3D information of the four corner points of the projection plane can be converted into normalized 2D information of the four corner points of the projected image.

[0098] Then, based on the normalized two-dimensional information of the four corner points of the projected image, the normalized two-dimensional information of the four corner points of the maximum inscribed rectangle of the projected image can be calculated. Thus, based on the normalized two-dimensional information of the four corner points of the projected image and the normalized two-dimensional information of the four corner points of the maximum inscribed rectangle, affine transformation parameters are generated to transform the four corner points of the projected image into the four corner points of the maximum inscribed rectangle, thereby obtaining the correction parameters for the projected image.

[0099] The above is one possible implementation method for calculating the correction parameters. In other implementation methods, different approaches may also be used.

[0100] For example, the correction parameter can be the tilt angle of the plane containing the projection image relative to the projection device (or the tilt angle of the projection device relative to the plane containing the projection image). In this way, the projection device can tilt the projected image by the same angle as the tilt angle of the plane containing the projection image relative to the projection device, projecting a distorted image after the tilt angle, thereby compensating for the tilt angle between the projection device and the plane, making the projected image on the plane rectangular. Alternatively, the correction parameter sent to the projection device can also be a trigonometric function calculated based on the tilt angle of the projection image relative to the projection device. The executor of calculating the correction parameter is the executor of this application embodiment. If the executor in this application embodiment is not the projection device, the calculated correction parameter can be sent to the projection device via wired or wireless communication, allowing the projection device to project a distorted image based on the correction parameter. If the executor in this application embodiment is the projection device, the calculation of the correction parameter is performed by the projection device.

[0101] For example, correction parameters can be parameters that enable the projection device to determine the outline of the current projected image. These could be the coordinates of the four sides of the projected image, or the coordinates of the four corner points. Based on this, the projection device can calculate how to adjust the current projected image to a preset shape. For instance, the projection device can calculate a maximum inscribed rectangle within the current projected image (based on the four sides or four corner points to determine the boundaries of the projected image), and calculate the necessary deformation of the projected image based on the deformation between the projected image and the maximum inscribed rectangle. Specifically, the projected image can be deformed using an affine transformation. The affine transformation parameters of the projected image are calculated based on the affine transformation that converts the projected image into the maximum inscribed rectangle, ensuring that the projected image formed on the projection plane after the projection device projects the image deformed according to the affine transformation parameters is rectangular. A specific implementation method can be that there is a correspondence between the preset pattern features in the projected image and the preset pattern features on the projected screen. Each feature point corresponds one-to-one, and the position (coordinates) of the preset pattern features in the projected image is known, as are the coordinates of the preset pattern features in the projected screen. Based on the coordinate correspondence of the same pattern features, the affine transformation relationship H (which is essentially a matrix) between the coordinate systems of the projected image and the projected screen can be calculated. In an optional embodiment of this application, the affine transformation parameter can refer to the aforementioned affine transformation relationship H (i.e., a matrix). Based on the maximum inscribed rectangle (the coordinates of its four vertices) calculated inside the projected screen, combined with the affine transformation relationship H, the deformed projected image (the coordinates of its four vertices) after transforming from the coordinate system of the projected screen to the coordinate system of the projected image can be determined. Thus, the projection device can deform the original projected image into the aforementioned deformed projected image. After the projection device projects the deformed projected image, a rectangular projected screen can be obtained on the projection plane. Optionally, in another optional embodiment of this application, the affine transformation parameter can also refer to the coordinates of the four vertices of the deformed projected image. Specifically, the affine transformation relationship H or the coordinates of the four vertices of the deformed projected image can be included as affine transformation parameters in the optional implementations provided in this application, as long as the projection device can determine the deformed projected image based on the affine transformation parameters.

[0102] For example, correction parameters can be parameters that enable the projection device to determine the difference between the outline of the current projected image and the outline of a preset shape. For instance, taking a rectangle as the preset shape of the target to be adjusted, the correction parameters can be the three-dimensional coordinates of the four corner points of the projected image and the three-dimensional coordinates of the four corner points of the preset shape. The three-dimensional coordinates of the four corner points of the preset shape can be obtained by calculating the maximum inscribed rectangle of the projected image.

[0103] Step 107: Control the projection device to project the distorted image according to the correction parameters.

[0104] In one example, the projection correction method provided in this application embodiment can be executed by an electronic device (such as a mobile phone, tablet computer, or other terminal) that captures images. In this way, the calculated correction parameters can be sent through communication with the projection device (such as wireless communication such as Bluetooth, mobile cellular network, WIFI, or wired communication such as direct USB data cable connection). The correction parameters are sent to the projection device so that the projection device projects the distorted projection image according to the correction parameters, thereby correcting the edge contour of the projected image to a preset shape.

[0105] In another example, the projection correction method provided in this application embodiment can be executed by a projection device. In this way, after calculating the correction parameters, the projected image can be directly deformed. Optionally, the projection correction method provided in this application embodiment can be integrated into a mobile terminal application (APP). In this way, an image can be captured by the mobile terminal's camera, the APP can be run, a communication connection can be established with the projection device through the mobile terminal's communication module, and the correction parameters can be sent to the projection device after being obtained.

[0106] The projection correction method of this application embodiment acquires the projection device and the projected image captured by the camera. It can identify the projection device and the projected image in the projection scene image, and calculate the relative positional relationship between the camera and the projection device by combining the known three-dimensional information of the projection device. Then, it calculates the three-dimensional information of the projected image by combining the relative positional relationship. In this way, correction parameters for the projected image can be generated based on the three-dimensional information of the projected image. Then, the projection device is controlled to project the distorted projected image according to the correction parameters. The projection device can correct the edge contour of the projected image to a preset shape according to the correction parameters, thereby realizing the correction of the projected image. It does not require the addition of additional devices such as cameras to the projection device, thus reducing the cost of projection correction.

[0107] Optionally, when identifying the projection device in the projection scene image in step 102, the following steps may be included:

[0108] Step 1021: Obtain the 3D model of the projection device.

[0109] The three-dimensional model of the projection device is known, and the three-dimensional coordinates of the three-dimensional model are known. For example, the three-dimensional model can be a three-dimensional CAD model.

[0110] Step 1022: Extract appearance features from the 3D model and the projected scene image, respectively.

[0111] Appearance features refer to the visual characteristics of a projection device. These features can include the device's color, edges, vertices, etc. In an image, appearance features can be extracted using image feature extraction methods; for example, edges and vertices of the projection device can be extracted by extracting lines and corner points.

[0112] Image features can be extracted using image processing algorithms from related technologies. For example, corner feature extraction can be performed using the Harris corner detection algorithm to obtain all identified corners in the image.

[0113] Furthermore, when extracting appearance features from a 3D model, the 3D model can be rotated to different angles to obtain 2D images at the corresponding angles, and then appearance features can be extracted from the 2D images at each angle.

[0114] Step 1023: Perform feature matching on the appearance features extracted from the 3D model and the projected scene image to obtain the feature matching results.

[0115] In the field of computer vision, image feature matching is based on image feature points, also known as feature point matching. Feature point matching can use algorithms such as SIFT and SURF; alternatively, it can use deep learning algorithms, which employ a trained neural network model to output the feature point matching results based on two input images, extracted feature descriptors, and the 3D coordinates of the features.

[0116] Step 1024: Based on the feature matching results, determine the appearance features of the projection device in the projection scene image that match the appearance features extracted from the 3D model.

[0117] The feature matching results include the correspondence between the appearance features extracted from the projection scene image and the appearance features extracted from the 3D model. Therefore, based on the feature matching results, the appearance features of the projection device that match the appearance features extracted from the 3D model can be determined in the projection scene image. After identifying the appearance features of the projection device, the position of each appearance feature in the image can be determined. Furthermore, based on the 3D model, the 3D dimensions / coordinates of the corresponding appearance features can be determined. This allows for the correspondence between the 2D and 3D information of the projection device in the image, enabling the establishment of equations and the solution of camera extrinsic parameters.

[0118] Optionally, when performing feature matching between two-dimensional images of a 3D model and projected scene images from multiple different angles, feature matching can be performed on the two-dimensional images and projected scene images at the corresponding angles based on the appearance features extracted from the two-dimensional images and projected scene images at each angle, thereby obtaining the feature matching results of the two-dimensional images and projected scene images at each angle.

[0119] It should be noted that the projection correction method provided in this application can be executed by a projection correction device or a control module within that projection correction device for performing the projection correction method. This application uses the projection correction device performing the projection correction method as an example to illustrate the projection correction apparatus provided in this application.

[0120] like Figure 4 As shown, the projection correction device provided in this application embodiment includes an acquisition unit 41, a first identification unit 42, a first calculation unit 43, a second identification unit 44, a second calculation unit 45, a generation unit 46, and a control unit 47.

[0121] The acquisition unit 41 is used to acquire the projection scene image captured by the camera; wherein, the projection scene image includes the projection device and the projection screen displayed by the projection device projecting the projected image onto the projection plane;

[0122] The first identification unit 42 is used to identify the projection device in the projection scene image;

[0123] The first calculation unit 43 is used to calculate the relative positional relationship between the camera and the projection device by combining the known three-dimensional information of the projection device.

[0124] The second recognition unit 44 is used to recognize the projected image in the projected scene image;

[0125] The second calculation unit 45 is used to calculate the three-dimensional information of the projected image by combining the relative positional relationship;

[0126] The generation unit 46 is used to generate correction parameters for the projected image based on the three-dimensional information of the projected image; wherein, the correction parameters are used to deform the projected image so that the edge contour of the projected image is corrected to a preset shape.

[0127] The control unit 47 is used to control the projection device to project the distorted image according to the correction parameters.

[0128] The projection correction device of this application embodiment acquires the projection device and the projected image captured by the camera. It can identify the projection device and the projected image in the projection scene image, and calculate the relative positional relationship between the camera and the projection device by combining the known three-dimensional information of the projection device. Then, it calculates the three-dimensional information of the projected image by combining the relative positional relationship. In this way, correction parameters for the projected image can be generated based on the three-dimensional information of the projected image. Then, the projection device is controlled to project the distorted projected image according to the correction parameters. The projection device can correct the edge contour of the projected image to a preset shape according to the correction parameters, thereby realizing the correction of the projected image. It does not require the addition of additional devices such as cameras to the projection device, thus reducing the cost of projection correction.

[0129] Optionally, the first identification unit 42 includes:

[0130] Acquire sub-units to acquire the 3D model of the projection device;

[0131] The first extraction subunit is used to extract appearance features from the 3D model and the projected scene image, respectively.

[0132] The feature matching subunit is used to perform feature matching on the appearance features extracted from the 3D model and the projected scene image to obtain the feature matching result.

[0133] The sub-unit is determined to identify the appearance features of the projection device in the projection scene image that match the appearance features extracted from the 3D model, based on the feature matching results.

[0134] Optionally, the first extraction subunit includes:

[0135] Rotation sub-units are used to rotate a 3D model to different angles to obtain a 2D image at the corresponding angle.

[0136] The second extraction subunit is used to extract features from the two-dimensional image at each angle.

[0137] The feature matching subunit is also used to perform feature matching on the appearance features extracted from the two-dimensional image at each angle and the appearance features extracted from the projected scene image.

[0138] Optionally, the acquisition unit 41 is further configured to acquire the three-dimensional coordinates of the appearance features in the three-dimensional model that match the projected scene image, based on the known three-dimensional coordinates of the three-dimensional model, before calculating the relative positional relationship between the camera and the projection device by combining the known three-dimensional information of the projection device.

[0139] The first calculation unit 43 is also used to calculate the camera extrinsic parameters of the camera relative to the projection device by combining the three-dimensional coordinates of the appearance features in the three-dimensional model that match the image of the projected scene.

[0140] Optionally, the projected image includes preset pattern features, and the second recognition unit 44 is also used to recognize the preset pattern features in the projected scene image;

[0141] The second calculation unit 45 is also used to calculate the three-dimensional coordinates of the preset pattern feature by combining the relative positional relationship; and to calculate the three-dimensional coordinates of the four corner points of the projected image based on the three-dimensional coordinates of the preset pattern feature and the position of the preset pattern feature in the projected image.

[0142] Optionally, the generating unit 46 includes:

[0143] The first calculation subunit is used to calculate the maximum inscribed rectangle of the four corner points of the projected image, and obtain the projected area after the projected image is corrected.

[0144] The second calculation subunit is used to calculate the correction parameters for deforming the projected image based on the transformation relationship between the four corner points of the projected image and the four corner points of the projection area.

[0145] This application also provides an electronic device. This electronic device can be a mobile electronic device or a non-mobile electronic device. For example, a mobile electronic device can be a mobile phone, tablet computer, laptop computer, PDA, or personal digital assistant, etc., while a non-mobile electronic device can be a server, personal computer, television set, etc. This application does not impose specific limitations.

[0146] Figure 5 A schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application is shown.

[0147] An electronic device may include a processor 301 and a memory 302 storing computer program instructions.

[0148] Specifically, the processor 301 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0149] Memory 302 may include mass storage for data or instructions. For example, and not limitingly, memory 302 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 302 may include removable or non-removable (or fixed) media. Where appropriate, memory 302 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 302 is non-volatile solid-state memory.

[0150] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of this application.

[0151] The processor 301 implements any of the projection correction methods in the above embodiments by reading and executing computer program instructions stored in the memory 302.

[0152] In one example, the electronic device may also include a communication interface 303 and a bus 310. For example, Figure 5 As shown, the processor 301, memory 302, and communication interface 303 are connected through bus 310 and complete communication with each other.

[0153] The communication interface 303 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0154] Bus 310 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 310 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0155] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0156] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0157] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0158] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0159] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A projection correction method, characterized in that, include: Acquire a projection scene image captured by a camera; wherein the projection scene image includes a projection device and a projection screen displayed by the projection device projecting the image onto a projection plane; A three-dimensional model of the projection device is obtained; appearance features are extracted from the three-dimensional model and the projection scene image respectively; feature matching is performed on the appearance features extracted from the three-dimensional model and the projection scene image to obtain feature matching results; based on the feature matching results, the appearance features of the projection device that match the appearance features extracted from the three-dimensional model are determined in the projection scene image. Based on the known three-dimensional coordinates of the three-dimensional model, obtain the three-dimensional coordinates of the appearance features in the three-dimensional model that match the projected scene image; Based on the known three-dimensional information of the projection device, calculate the relative positional relationship between the camera and the projection device; Identify the projected image in the projected scene image; Based on the relative positional relationships, the three-dimensional information of the projected image is calculated; Based on the three-dimensional information of the projected image, correction parameters for the projected image are generated; wherein, the correction parameters are used to deform the projected image so that the edge contour of the projected image is corrected to a preset shape; The projection device is controlled to project the distorted image according to the correction parameters; The step of extracting appearance features from the three-dimensional model includes: rotating the three-dimensional model to different angles to obtain two-dimensional images at the corresponding angles; and extracting appearance features from the two-dimensional images at each angle. The step of performing feature matching on the appearance features extracted from the 3D model and the projected scene image includes: performing feature matching on the appearance features extracted from the 2D image at each angle and the appearance features extracted from the projected scene image; The step of calculating the relative positional relationship between the camera and the projection device by combining the known three-dimensional information of the projection device includes: calculating the camera extrinsic parameters of the camera relative to the projection device by combining the three-dimensional coordinates of the appearance features in the three-dimensional model that match the image of the projected scene.

2. The projection correction method according to claim 1, characterized in that, The projected scene image includes preset pattern features, and the step of identifying the projected image in the projected scene image includes: Identify the preset pattern features in the projected scene image; The step of calculating the three-dimensional information of the projected image based on the relative positional relationship includes: Based on the relative positional relationship, calculate the three-dimensional coordinates of the preset pattern feature; The three-dimensional coordinates of the four corner points of the projected image are calculated based on the three-dimensional coordinates of the preset pattern feature and the position of the preset pattern feature in the projected image.

3. The projection correction method according to claim 2, characterized in that, The step of generating correction parameters for the projected image based on the three-dimensional information of the projected image includes: Calculate the maximum inscribed rectangle of the four corner points of the projected image to obtain the corrected projection area of ​​the projected image; Based on the transformation relationship of transforming the four corner points of the projected image to the four corner points of the projection area, the correction parameters for deforming the projected image are calculated.

4. A projection correction device, characterized in that, include: An acquisition unit is used to acquire a projection scene image captured by a camera; wherein the projection scene image includes a projection device and a projection screen displayed by the projection device projecting the projected image onto a projection plane; The first identification unit includes: A sub-unit is used to acquire a three-dimensional model of the projection device; The first extraction subunit is used to extract appearance features from the 3D model and the projected scene image, respectively. The feature matching subunit is used to perform feature matching on the appearance features extracted from the 3D model and the projected scene image to obtain the feature matching result. A subunit is defined to determine, based on the feature matching result, the appearance features of the projection device in the projection scene image that match the appearance features extracted from the three-dimensional model; The acquisition unit is further configured to acquire, based on the known three-dimensional coordinates of the three-dimensional model, the three-dimensional features in the three-dimensional model that match the projected scene image; The first calculation unit is used to calculate the relative positional relationship between the camera and the projection device by combining the known three-dimensional information of the projection device; The second recognition unit is used to recognize the projected image in the projected scene image; The second calculation unit is used to calculate the three-dimensional information of the projected image by combining the relative positional relationship; The generation unit is used to generate correction parameters for the projected image based on the three-dimensional information of the projected image; wherein the correction parameters are used to deform the projected image so that the edge contour of the projected image is corrected to a preset shape; A control unit is used to control the projection device to project the distorted projected image according to the correction parameters; The first extraction subunit includes: A rotation subunit is used to rotate the three-dimensional model to different angles to obtain a two-dimensional image at the corresponding angle. The second extraction subunit is used to extract appearance features from the two-dimensional image at each angle. The feature matching subunit is also used to perform feature matching on the appearance features extracted from the two-dimensional image at each angle and the appearance features extracted from the projected scene image; The first calculation unit is specifically used to calculate the camera extrinsic parameters of the camera relative to the projection device by combining the three-dimensional coordinates of the appearance features in the three-dimensional model that match the projected scene image.

5. The projection correction device according to claim 4, characterized in that, The projected scene image includes preset pattern features, and the second recognition unit is further configured to recognize the preset pattern features in the projected scene image; The second calculation unit is also used to calculate the three-dimensional coordinates of the preset pattern feature in combination with the relative positional relationship; and to calculate the three-dimensional coordinates of the four corner points of the projected image based on the three-dimensional coordinates of the preset pattern feature and the position of the preset pattern feature in the projected image.

6. The projection correction device according to claim 4, characterized in that, The generation unit includes: The first calculation subunit is used to calculate the maximum inscribed rectangle of the four corner points of the projected image to obtain the corrected projection area of ​​the projected image. The second calculation subunit is used to calculate the correction parameters for deforming the projected image based on the transformation relationship of transforming the four corner points of the projected image to the four corner points of the projection area.

7. An electronic device, characterized in that, The electronic device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the projection correction method as described in any one of claims 1-3.

8. A storage medium, characterized in that, The storage medium stores computer program instructions, which, when executed by a processor, implement the projection correction method as described in any one of claims 1-3.

Images