Projection control method and device and storage medium

Abstract

The invention discloses a projection control method and device and a storage medium, and the method comprises the steps: obtaining a shot image collected for a current projection image projected on a projection plane, and intercepting a first source image of the current projection image; determining a first mapping relation between the shot image and the first source image; determining projection coordinates of a first source image based on the first mapping relation and the angular point coordinates of the curtain in the shot image under the condition that the angular point coordinates of the curtain in the shot image are determined; and projecting a second source image based on the projection coordinates of the first source image. Thus, aligned display of the projection picture and the curtain can be ensured, better continuity is achieved, and the situation that the projection picture is interrupted does not occur.

Description

Technical Field

[0001] The present application relates to control technology, and in particular to a projection control method, device and storage medium. Background Art

[0002] Currently, in order to realize the screen alignment function of the projection device, a feature map needs to be projected, and the screen alignment operation is performed based on the projected feature map. However, this screen alignment operation will interrupt the projection content being displayed. Summary of the invention

[0003] Embodiments of the present application provide a projection control method, device, and storage medium.

[0004] The technical solution of this application is implemented as follows:

[0005] In a first aspect, a projection control method is provided, the projection control method comprising:

[0006] Acquire a captured image of a current projection picture projected on a projection plane, and intercept a first source image of the current projection picture;

[0007] Determining a first mapping relationship between the captured image and the first source image;

[0008] In the case where the coordinates of the corner points of the curtain in the captured image are determined, the projection coordinates of the first source image are determined based on the first mapping relationship and the coordinates of the corner points of the curtain in the captured image;

[0009] Based on the projection coordinates of the first source image, a second source image is projected.

[0010] In the above solution, determining a first mapping relationship between the captured image and the first source image includes:

[0011] Determining gradient values ​​of a plurality of pixel points in the captured image;

[0012] Using the first window to filter the local maximum gradient value in the captured image to obtain a first pixel point set;

[0013] Determining gradient values ​​of a plurality of pixel points in the first source image;

[0014] Using a second window to filter the local maximum gradient value in the first source image to obtain a second pixel point set;

[0015] Based on the first pixel point set and the second pixel point set, the preset mapping relationship is updated to obtain the first mapping relationship; wherein the preset mapping relationship is a mapping relationship between a camera and an optical machine.

[0016] In the above solution, the updating of the preset mapping relationship based on the first pixel point set and the second pixel point set to obtain the first mapping relationship includes:

[0017] Based on the preset mapping relationship, mapping the second pixel point set to the captured image to obtain a third pixel point set;

[0018] Taking the third pixel point as the center point of the third window, if there is a first pixel point in the third window, taking the first pixel point closest to the third pixel point as the new third pixel point;

[0019] Based on a new third pixel point set corresponding to the third pixel point set and a second pixel point set corresponding to the third pixel point set under the preset mapping relationship, updating the preset mapping relationship to obtain a new preset mapping relationship;

[0020] Based on the new preset mapping relationship, determining a structural similarity index between a first preset central area of ​​the first source image and a matching area of ​​the captured image;

[0021] Iteratively executing the calculation of the structural similarity index P times to obtain P structural similarity indexes;

[0022] A maximum value among the P structural similarity indexes is determined, and a mapping relationship corresponding to the maximum value is used as the first mapping relationship.

[0023] In the above solution, determining a first mapping relationship between the captured image and the first source image includes:

[0024] Determine first coordinate information of a plurality of pixel points in the current projection image in a world coordinate system with the ranging sensor as the origin;

[0025] Based on the rotational translation relationship between the ranging sensor and the optical machine, and the first coordinate information of the plurality of pixel points in the current projection image, determine the second coordinate information of the plurality of pixel points in the first source image in a world coordinate system with the optical machine as the origin;

[0026] Determine, based on the rotational translation relationship between the optical machine and the camera, and the second coordinate information of the plurality of pixels in the first source image, the third coordinate information of the plurality of pixels in the captured image in a world coordinate system with the camera as the origin;

[0027] The first mapping relationship is obtained based on the third coordinate information of multiple pixel points in the captured image and the second coordinate information of multiple pixel points in the first source image.

[0028] In the above solution, determining a first mapping relationship between the captured image and the first source image includes:

[0029] Dividing a preset central area of ​​the captured image into M*N grids;

[0030] Determine the variance of target color component values ​​of a plurality of pixel points in each grid; wherein the target color component value is any one of a red component value, a green component value, and a blue component value;

[0031] Counting the number of grids when the variance is greater than or equal to a first threshold;

[0032] When the number of grids is greater than or equal to a second threshold, a first mapping relationship between the captured image and the first source image is determined; wherein the second threshold is determined by M and N.

[0033] In the above solution, determining a first mapping relationship between the captured image and the first source image includes:

[0034] Perform feature point matching between the captured image and the first source image to determine the number of feature points that match between the captured image and the first source image;

[0035] When the number of feature points is greater than or equal to a third threshold, a first mapping relationship between the captured image and the first source image is determined.

[0036] In the above solution, determining the coordinates of the corner points of the curtain in the captured image includes:

[0037] Extracting a boundary area of ​​the curtain in the captured image;

[0038] Using the first parameter as an input parameter of a preset gradient value calculation formula to obtain a first gradient value calculation formula; wherein different corner point coordinates correspond to different preset gradient value calculation formulas;

[0039] Based on the first gradient value calculation formula, determine a plurality of first gradient values ​​corresponding to a plurality of pixel points on the boundary area;

[0040] Using the second parameter as an input parameter of the preset gradient value calculation formula to obtain a second gradient value calculation formula;

[0041] Based on the second gradient value calculation formula, determine a plurality of second gradient values ​​corresponding to a plurality of pixel points on the boundary area;

[0042] Based on a plurality of first gradient values ​​and a plurality of second gradient values ​​corresponding to a plurality of pixel points on the boundary area, the corresponding corner point coordinates of the curtain are determined.

[0043] In the above solution, the determining of the corresponding corner point coordinates of the curtain based on the multiple first gradient values ​​and the multiple second gradient values ​​corresponding to the multiple pixel points on the boundary area includes:

[0044] determining a first maximum value among the plurality of first gradient values, and determining a second maximum value among the plurality of second gradient values;

[0045] In the case that the first maximum value and the second maximum value correspond to the same target pixel point, the coordinates of the target pixel point are used as the corresponding corner point coordinates of the curtain.

[0046] In a second aspect, a projection control device is provided, the projection control device comprising:

[0047] an acquisition unit, used to acquire a captured image collected for a current projection picture projected on a projection plane, and a capture unit, used to capture a first source image of the current projection picture;

[0048] a determining unit, configured to determine a first mapping relationship between the captured image and the first source image;

[0049] The determining unit is further configured to determine the projection coordinates of the first source image based on the first mapping relationship and the coordinates of the corner points of the curtain in the captured image when the coordinates of the corner points of the curtain in the captured image are determined;

[0050] The projection unit is used to project the second source image based on the projection coordinates of the first source image.

[0051] According to a third aspect, a projection control device is provided, comprising: a processor and a memory configured to store a computer program that can be run on the processor, wherein the processor is configured to execute the steps of the method according to the first aspect when running the computer program.

[0052] According to a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, wherein the computer program implements the steps of the method according to the first aspect when executed by a processor.

[0053] The present application provides a projection control method, device and storage medium, which obtains a captured image of a current projection screen projected on a projection plane, and intercepts a first source image of the current projection screen; determines a first mapping relationship between the captured image and the first source image; determines the projection coordinates of the first source image based on the first mapping relationship and the corner coordinates of the screen in the captured image when the corner coordinates of the screen in the captured image are determined; and projects a second source image based on the projection coordinates of the first source image. In this way, the alignment display of the projection screen and the screen can be ensured, and there is better continuity, and the projection screen will not be interrupted. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 An optional flowchart of the projection control method provided in the embodiment of the present application;

[0055] Figure 2 An optional flowchart of the projection control method provided in the embodiment of the present application;

[0056] Figure 3 An optional flowchart of the projection control method provided in the embodiment of the present application;

[0057] Figure 4 An optional flowchart of the projection control method provided in the embodiment of the present application;

[0058] Figure 5 A schematic diagram of a preset central area of ​​a captured image exemplified in an embodiment of the present application;

[0059] Figure 6 An optional flowchart of the projection control method provided in the embodiment of the present application;

[0060] Figure 7 An optional flowchart of the projection control method provided in the embodiment of the present application;

[0061] Figure 8 An optional structural diagram of a projection control device provided in an embodiment of the present application;

[0062] Fig. 9 An optional structural diagram of a projection control device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0064] The embodiment of the present application provides a projection control method, Figure 1 Schematic diagram of the first process of the projection control method in the embodiment of the present application. Figure 1 As shown, the projection control method may include:

[0065] Step 101: Acquire a captured image of a current projection screen projected on a projection plane, and capture a first source image of the current projection screen.

[0066] The projection screen includes static images and dynamic videos, and the types of the projection screen include dynamic types and static types. Therefore, in some embodiments, obtaining the captured image collected for the current projection screen projected on the projection plane includes:

[0067] When receiving a screen alignment instruction, identifying the type of projection image;

[0068] If the type is static, control the camera to capture the current projection screen and obtain the captured image;

[0069] If the type is dynamic, the projection screen is first controlled to stop, and then the camera is controlled to capture the current projection screen to obtain the captured image.

[0070] Step 102: Determine a first mapping relationship between a captured image and a first source image.

[0071] Step 103: when the coordinates of the corner points of the screen in the captured image are determined, the projection coordinates of the first source image are determined based on the first mapping relationship and the coordinates of the corner points of the screen in the captured image.

[0072] Step 104: Projecting the second source image based on the projection coordinates of the first source image.

[0073] Here, the first source image and the second source image may be source images of the same frame, or may be source images of different frames.

[0074] In an embodiment of the present application, the projection coordinates of the first source image are obtained according to the coordinates of the corner points of the screen in the captured image and a first mapping relationship between the captured image and the first source image, and then based on the projection coordinates of the first source image, the second source image is projected onto the projection plane to ensure that the projection picture is aligned with the screen.

[0075] In an achievable manner, the execution subject of the projection control method may be a processor of a projection device. Based on this, the projection device controls the first camera to capture the current projection screen to obtain a captured image, and intercepts the first source image of the current projection screen; the projection device determines a first mapping relationship between the captured image and the first source image; the projection device determines the projection coordinates of the first source image based on the first mapping relationship and the coordinates of the corner points of the screen in the captured image when determining the coordinates of the corner points of the screen in the captured image; the projection device projects the second source image based on the projection coordinates of the first source image.

[0076] In another achievable manner, the execution subject of the projection control method may be a processor of a terminal. Based on this, the terminal controls the second camera to capture the current projection screen to obtain a captured image, and obtains the first source image of the current projection screen sent by the projection device; the terminal determines the first mapping relationship between the captured image and the first source image; the terminal determines the projection coordinates of the first source image based on the first mapping relationship and the coordinates of the corner points of the screen in the captured image when determining the coordinates of the corner points of the screen in the captured image; the terminal sends the projection coordinates of the first source image to the projection device, so that the projection device projects the second source image based on the projection coordinates of the first source image.

[0077] In the embodiment of the present application, the alignment of the projection image and the screen can be ensured, and there is better continuity, and there will be no interruption of the projection image.

[0078] In some embodiments, reference Figure 2 As shown, the process of determining the first mapping relationship between the captured image and the first source image in step 102 is combined with Figure 2 To explain,

[0079] Step 201: Determine the gradient values ​​of a plurality of pixel points in a captured image.

[0080] Step 202: Using the first window to filter the local maximum gradient value in the captured image, to obtain a first pixel point set.

[0081] In an embodiment of the present application, the first window is used as a reference in the captured image, each window is selected in turn according to a certain step size (for example, one pixel is moved each time), and the local maximum gradient value of the pixel points in each window is screened out to obtain a first pixel point set.

[0082] Step 203: Determine the gradient values ​​of a plurality of pixel points in the first source image.

[0083] In the embodiment of the present application, the Sobel operator may be used to calculate the gradient values ​​of multiple pixel points in the first source image.

[0084] Step 204: Use the second window to filter the local maximum gradient value in the first source image to obtain a second pixel point set.

[0085] In an embodiment of the present application, the second window is used as a reference in the source image, each window is selected in turn according to a certain step size (for example, one pixel is moved each time), and the maximum gradient value of the pixel points in each window is screened out to obtain a second pixel point set.

[0086] In the embodiment of the present application, the size relationship between the first window and the second window is determined by the resolution of the camera and the resolution of the optical machine. For example, the ratio of the resolution of the camera to the resolution of the optical machine is 1:2, if the size of the first window is W*W, then the size of the second window is 2W*2W.

[0087] Step 205: Based on the first pixel point set and the second pixel point set, update the preset mapping relationship to obtain a first mapping relationship; wherein the preset mapping relationship is a mapping relationship between a camera and an optical machine.

[0088] In the embodiment of the present application, a first pixel point matching a second pixel point in the second pixel point set is found in the first pixel point set, and the preset mapping relationship is updated to obtain a first mapping relationship, thereby improving the accuracy of the mapping relationship.

[0089] Step 205 may include:

[0090] Step 1: Based on a preset mapping relationship, the second pixel point set is mapped to the captured image to obtain a third pixel point set.

[0091] Step 2: Taking the third pixel point as the center point of the third window, if the first pixel point exists in the third window, taking the first pixel point closest to the third pixel point as the new third pixel point.

[0092] Step 2 is repeatedly performed until a new third pixel point corresponding to each third pixel point in the third pixel point set is determined.

[0093] The size of the third window may be equal to the size of the first window.

[0094] Step 3: Based on the new third pixel point set corresponding to the third pixel point set and the second pixel point set corresponding to the third pixel point set under the preset mapping relationship, the preset mapping relationship is updated to obtain a new preset mapping relationship.

[0095] Step 4: Based on the new preset mapping relationship, determine the structural similarity index between the first preset central area of ​​the first source image and the matching area of ​​the captured image.

[0096] Structural Similarity Index (SSIM) is a measure of the similarity between two images. The value of SSIM index ranges from -1 to 1, where 1 means that the two images are exactly the same and -1 means that the two images are completely different.

[0097] Step 5: Iterate and calculate the structural similarity index P times to obtain P structural similarity indexes.

[0098] Iteration means taking the result of the last iteration as the input of the next iteration. That is, taking the new third pixel point as the origin of the third window, executing steps 2 to 4, and calculating a structural similarity index. Similarly, after P iterations, P structural similarity indexes can be obtained. The value of P can be 10.

[0099] Step 6: Determine the maximum value of the P structural similarity indexes, and use the mapping relationship corresponding to the maximum value as the first mapping relationship.

[0100] Embodiments of the present application Figure 2 The process of determining the first mapping relationship shown can be applied to a projection device or a terminal.

[0101] In some embodiments, reference Figure 3 As shown, the process of determining the first mapping relationship between the captured image and the first source image in step 102 can also be combined with Figure 3 To explain,

[0102] Step 301: Determine first coordinate information of a plurality of pixel points in a current projection image in a world coordinate system with a distance measuring sensor as an origin.

[0103] In the embodiment of the present application, the depth is calculated by means of a ranging sensor, so in a world coordinate system with the ranging sensor as the origin, the first coordinate information of multiple pixel points in the current projection image is represented as (X1, Y1, Z1), which are all known. The ranging sensor may include an infrared sensor.

[0104] Step 302: Based on the rotational translation relationship between the ranging sensor and the optical machine, and the first coordinate information of multiple pixel points in the current projection image, determine the second coordinate information of multiple pixel points in the first source image in the world coordinate system with the optical machine as the origin.

[0105] In the embodiment of the present application, the rotational translation relationship between the ranging sensor and the optical machine is R1T1. Therefore, according to [X, Y, Z] = [X1, Y1, Z1][R1] + T1, the second coordinate information (X, Y, Z) of multiple pixel points in the source first image in the world coordinate system with the optical machine as the origin can be obtained, and they are all known.

[0106] Step 303: Based on the rotation and translation relationship between the optical machine and the camera, and the second coordinate information of multiple pixel points in the first source image, determine the third coordinate information of multiple pixel points in the captured image in a world coordinate system with the camera as the origin.

[0107] In the embodiment of the present application, the rotational and translation relationship between the optical machine and the camera is R2T2. Therefore, according to [X2, Y2, Z2] = [X, Y, Z][R2] + T2, the third coordinate information (X2, Y2, Z2) of multiple pixel points in the captured image in the world coordinate system with the camera as the origin can be obtained, and they are all known.

[0108] Step 304: Obtain a first mapping relationship based on the third coordinate information of the plurality of pixel points in the captured image and the second coordinate information of the plurality of pixel points in the first source image.

[0109] Embodiments of the present application Figure 3 The process of determining the first mapping relationship shown can be applied to a projection device.

[0110] It should be noted that the first mapping relationship may also be obtained through a SURF feature point detection method, a SIFT feature point detection method, or a Harris feature point detection method.

[0111] In some embodiments, before determining the first mapping relationship between the captured image and the first source image, the conditional process of determining the first mapping relationship is combined with Figure 4 As shown in the following description,

[0112] Step 401: Divide a preset central area of ​​a captured image into M*N grids.

[0113] The preset center area can be Figure 5 The central one-third area shown is not limited to the preset central area.

[0114] M and N may be equal or unequal. Figure 5 Here M is equal to 4 and N is equal to 4.

[0115] Step 402: Determine the variance of target color component values ​​of a plurality of pixel points in each grid; wherein the target color component value is any one of a red component value, a green component value, and a blue component value.

[0116] Each pixel in the captured image corresponds to three color component values, namely, the red component value (ie, R value), the green component value (ie, G value) and the blue component value (ie, B value), and the value range of each color component value is 0 to 255.

[0117] That is, the variance of any color component value among the red component value, the green component value and the blue component value of multiple pixel points in each grid is determined.

[0118] Step 403: Count the number of grids when the statistical variance is greater than or equal to the first threshold.

[0119] In the embodiment of the present application, the variance of the R values ​​of multiple pixels in each grid is determined, and the number of grids when the variance is greater than or equal to the first threshold is counted. Alternatively, the variance of the G values ​​of multiple pixels in each grid is determined, and the number of grids when the variance is greater than or equal to the first threshold is counted. Alternatively, the variance of the B values ​​of multiple pixels in each grid is determined, and the number of grids when the variance is greater than or equal to the first threshold is counted.

[0120] In the embodiment of the present application, when the variance is greater than or equal to the first threshold, it indicates that the grid image corresponding to the variance has content features. The first thresholds corresponding to different color component values ​​may be the same or different.

[0121] Step 404: when the number of grids is greater than or equal to the second threshold, determine a first mapping relationship between the captured image and the first source image.

[0122] In the embodiment of the present application, when the number of grids is greater than or equal to the second threshold, it is determined that the content features of the first source image are rich. The second threshold can be determined by M and N, for example, the second threshold is equal to half of the product of M and N.

[0123] The first mapping relationship refers to a mapping relationship between feature points of the first source image and matching feature points of the captured image.

[0124] In an embodiment of the present application, by dividing the preset central area of ​​the captured image, the number of grids when the variance is greater than or equal to the first threshold is counted, and the grid image corresponding to the variance greater than or equal to the first threshold has content features. When the number of grids is greater than the second threshold, it is considered that the content features of the first source image are rich. Only in this way can the first mapping relationship between the captured image and the first source image be determined.

[0125] Embodiments of the present application Figure 4 The conditional process for determining the first mapping relationship shown can be applied to a projection device or a terminal.

[0126] In some embodiments, before determining the first mapping relationship between the captured image and the first source image, the conditional process of determining the first mapping relationship may also be combined with Figure 6 As shown in the following description,

[0127] Step 601: Perform feature point matching between the captured image and the first source image to determine the number of feature points that match between the captured image and the first source image.

[0128] Step 602: when the number of feature points is greater than or equal to a third threshold, determine a first mapping relationship between the captured image and the first source image.

[0129] In the embodiment of the present application, when the number of feature points is greater than or equal to the third threshold, it is determined that the content features of the first source image are rich.

[0130] The first mapping relationship refers to a mapping relationship between feature points of the first source image and matching feature points of the captured image.

[0131] In an embodiment of the present application, by performing feature matching between the captured image and the first source image, the number of matching feature points is obtained. When the number of feature points is greater than or equal to a third threshold, it is considered that the content features of the first source image are rich. Only in this way can the first mapping relationship between the captured image and the first source image be determined.

[0132] Embodiments of the present application Figure 6 The conditional process for determining the first mapping relationship shown can be applied to a projection device or a terminal.

[0133] In some embodiments, reference Figure 7 As shown, the process of determining the coordinates of the corner points of the curtain in the captured image in step 103 can also be combined with Figure 7 To explain,

[0134] Step 701: extracting the boundary area of ​​the curtain in the captured image.

[0135] In some embodiments, extracting the boundary area of ​​the curtain in the captured image includes:

[0136] Extracting a plurality of darkest pixels in the captured image;

[0137] The plurality of darkest pixel points are flood-filled to obtain a boundary area of ​​the curtain in the captured image.

[0138] It should be noted that, since the boundary area of ​​the curtain is black, a plurality of darkest pixels (ie, black pixels) in the captured image are firstly extracted, and then the plurality of darkest pixels are flood-filled, and finally the boundary area of ​​the curtain in the captured image is extracted.

[0139] Step 702: using the first parameter as an input parameter of a preset gradient value calculation formula to obtain a first gradient value calculation formula; wherein different corner point coordinates correspond to different preset gradient value calculation formulas.

[0140] Step 703: Based on the first gradient value calculation formula, determine multiple first gradient values ​​corresponding to multiple pixel points on the boundary area.

[0141] Step 704: using the second parameter as an input parameter of a preset gradient value calculation formula to obtain a second gradient value calculation formula.

[0142] Step 705: Based on the second gradient value calculation formula, determine multiple second gradient values ​​corresponding to multiple pixel points on the boundary area.

[0143] Step 706: Determine the coordinates of the corresponding corner points of the screen based on the multiple first gradient values ​​and the multiple second gradient values ​​corresponding to the multiple pixel points on the boundary area.

[0144] In some embodiments, step 706 may include: determining a first maximum value among the plurality of first gradient values, and determining a second maximum value among the plurality of second gradient values;

[0145] In the case that the first maximum value and the second maximum value correspond to the same target pixel point, the coordinates of the target pixel point are used as the corresponding corner point coordinates of the curtain.

[0146] It should be noted that when the first maximum value and the second maximum value correspond to the same target pixel point, it is considered that the corner point of the screen is accurately positioned, and the coordinates of the target pixel point are used as the coordinates of the corresponding corner point of the screen.

[0147] In the embodiment of the present application, different corner point coordinates correspond to different preset gradient value calculation formulas.

[0148] Exemplarily, the calculation formula of the preset gradient value corresponding to the coordinates of the upper left corner point is shown in formula (1),

[0149] f1(x,y)=3*f(x+t,y+t)-f(xt,yt)-f(x+t,yt)-f(xt,y+t) (1)

[0150] When t=1 (i.e., the first parameter), the first gradient value calculation formula is as shown in formula (2):

[0151] f1(x,y)=3*f(x+1,y+1)-f(x-1,y-1)-f(x+1,y-1)-f(x-1,y+1) (2)

[0152] When t=2 (i.e., the second parameter), the second gradient value calculation formula is as shown in formula (3),

[0153] f1(x,y)=3*f(x+2,y+2)-f(x-2,y-2)-f(x+2,y-2)-f(x-2,y+2) (3)

[0154] Here, f1 represents the gradient value of the upper left pixel, and f represents the gradient value of the surrounding pixels of the upper left pixel.

[0155] Based on formula (2), multiple first gradient values ​​corresponding to multiple pixel points on the boundary area are calculated, and a first maximum value among the multiple first gradient values ​​is determined.

[0156] Based on formula (3), multiple second gradient values ​​corresponding to multiple pixel points on the boundary area are calculated, and a second maximum value among the multiple second gradient values ​​is determined.

[0157] When the coordinates of the pixel point corresponding to the first maximum value are consistent with the coordinates of the pixel point corresponding to the second maximum value, it is considered that the upper left corner of the screen is accurately positioned, and the coordinates of the pixel point corresponding to the first maximum value or the coordinates of the pixel point corresponding to the second maximum value are used as the coordinates of the upper left corner of the screen. When the coordinates of the pixel point corresponding to the first maximum value are inconsistent with the coordinates of the pixel point corresponding to the second maximum value, it is considered that the upper left corner of the screen has failed to be positioned, and the screen alignment function needs to be retriggered.

[0158] The calculation formula of the preset gradient value corresponding to the coordinates of the upper right corner point is shown in formula (4):

[0159] f2(x,y)=3*f(xt,y+t)-f(x+t,yt)-f(x+t,y+t)-f(xt,yt) (4)

[0160] When t=1 (i.e., the first parameter), the first gradient value calculation formula is as shown in formula (5):

[0161] f2(x,y)=3*f(x-1,y+1)-f(x+1,y-1)-f(x+1,y+1)-f(x-1,y-1) (5)

[0162] When t=2 (i.e., the second parameter), the second gradient value calculation formula is as shown in formula (6),

[0163] f2(x,y)=3*f(x-2,y+2)-f(x+2,y-2)-f(x+2,y+2)-f(x-2,y-2) (6)

[0164] Here, f2 represents the gradient value of the upper right corner pixel, and f represents the gradient value of the surrounding pixels of the upper right corner pixel.

[0165] Based on formula (5), multiple first gradient values ​​corresponding to multiple pixel points on the boundary area are calculated, and a first maximum value among the multiple first gradient values ​​is determined.

[0166] Based on formula (6), multiple second gradient values ​​corresponding to multiple pixel points on the boundary area are calculated, and a second maximum value among the multiple second gradient values ​​is determined.

[0167] When the coordinates of the pixel point corresponding to the first maximum value are consistent with the coordinates of the pixel point corresponding to the second maximum value, it is considered that the upper right corner of the screen is accurately positioned, and the coordinates of the pixel point corresponding to the first maximum value or the coordinates of the pixel point corresponding to the second maximum value are used as the coordinates of the upper right corner of the screen. When the coordinates of the pixel point corresponding to the first maximum value are inconsistent with the coordinates of the pixel point corresponding to the second maximum value, it is considered that the upper right corner of the screen has failed to be positioned, and the screen alignment function needs to be retriggered.

[0168] The calculation formula of the preset gradient value corresponding to the coordinates of the lower left corner point is shown in formula (7):

[0169] f3(x,y)=3*f(x+t,yt)-f(xt,yt)-f(xt,y+t)-f(x+t,y+t) (7)

[0170] When t=1 (i.e., the first parameter), the first gradient value calculation formula is as shown in formula (8):

[0171] f3(x,y)=3*f(x+1,y-1)-f(x-1,y-1)-f(x-1,y+1)-f(x+1,y+1) (8)

[0172] When t=2 (i.e., the second parameter), the second gradient value calculation formula is as shown in formula (9),

[0173] f3(x,y)=3*f(x+2,y-2)-f(x-2,y-2)-f(x-2,y+2)-f(x+2,y+2) (9)

[0174] Here, f3 represents the gradient value of the lower left pixel, and f represents the gradient value of the surrounding pixels of the lower left pixel.

[0175] Based on formula (8), multiple first gradient values ​​corresponding to multiple pixel points on the boundary area are calculated, and a first maximum value among the multiple first gradient values ​​is determined.

[0176] Based on formula (9), multiple second gradient values ​​corresponding to multiple pixel points on the boundary area are calculated, and a second maximum value among the multiple second gradient values ​​is determined.

[0177] When the coordinates of the pixel point corresponding to the first maximum value are consistent with the coordinates of the pixel point corresponding to the second maximum value, it is considered that the lower left corner of the screen is accurately positioned, and the coordinates of the pixel point corresponding to the first maximum value or the coordinates of the pixel point corresponding to the second maximum value are used as the coordinates of the lower left corner of the screen. When the coordinates of the pixel point corresponding to the first maximum value are inconsistent with the coordinates of the pixel point corresponding to the second maximum value, it is considered that the positioning of the lower left corner of the screen has failed, and the screen alignment function needs to be retriggered.

[0178] The calculation formula of the preset gradient value corresponding to the coordinates of the lower right corner point is shown in formula (10):

[0179] f4(x,y)=3*f(xt,yt)-f(x+t,y+t)-f(x+t,yt)-f(xt,y+t) (10)

[0180] When t=1 (i.e., the first parameter), the first gradient value calculation formula is as shown in formula (11):

[0181] f4(x,y)=3*f(x-1,y-1)-f(x+1,y+1)-f(x+1,y-1)-f(x-1,y+1) (11)

[0182] When t=2 (i.e., the second parameter), the second gradient value calculation formula is as shown in formula (12),

[0183] f4(x,y)=3*f(x-2,y-2)-f(x+2,y+2)-f(x+2,y-2)-f(x-2,y+2) (12)

[0184] Here, f4 represents the gradient value of the lower right pixel, and f represents the gradient value of the surrounding pixels of the lower right pixel.

[0185] Based on formula (11), multiple first gradient values ​​corresponding to multiple pixel points on the boundary area are calculated, and a first maximum value among the multiple first gradient values ​​is determined.

[0186] Based on formula (12), multiple second gradient values ​​corresponding to multiple pixel points on the boundary area are calculated, and a second maximum value among the multiple second gradient values ​​is determined.

[0187] When the coordinates of the pixel point corresponding to the first maximum value are consistent with the coordinates of the pixel point corresponding to the second maximum value, it is considered that the lower right corner of the screen is accurately positioned, and the coordinates of the pixel point corresponding to the first maximum value or the coordinates of the pixel point corresponding to the second maximum value are used as the coordinates of the lower right corner of the screen. When the coordinates of the pixel point corresponding to the first maximum value are inconsistent with the coordinates of the pixel point corresponding to the second maximum value, it is considered that the positioning of the lower right corner of the screen has failed, and the screen alignment function needs to be retriggered.

[0188] Embodiments of the present application Figure 7 The process of determining the coordinates of the corner points of the screen in the captured image shown can be applied to a projection device or a terminal.

[0189] In order to implement the method of the embodiment of the present application, based on the same inventive concept, the embodiment of the present application further provides a projection control device, Figure 8 Schematic diagram of the structure of the projection control device in the embodiment of the present application. Figure 8 As shown, the projection control device 80 includes:

[0190] An acquisition unit 801 is used to acquire a captured image collected for a current projection picture projected on a projection plane, and a capture unit is used to capture a first source image of the current projection picture;

[0191] A determining unit 802, configured to determine a first mapping relationship between the captured image and the first source image;

[0192] The determining unit 802 is further configured to determine the projection coordinates of the first source image based on the first mapping relationship and the coordinates of the corner points of the curtain in the captured image when the coordinates of the corner points of the curtain in the captured image are determined;

[0193] The projection unit 803 is configured to project the second source image based on the projection coordinates of the first source image.

[0194] In the embodiment of the present application, the alignment of the projection image and the screen can be ensured, and there is better continuity, and there will be no interruption of the projection image.

[0195] In some embodiments, the determination unit 802 is further configured to determine gradient values ​​of a plurality of pixel points in the captured image;

[0196] Using the first window to filter the local maximum gradient value in the captured image to obtain a first pixel point set;

[0197] Determining gradient values ​​of a plurality of pixel points in the first source image;

[0198] Using a second window to filter the local maximum gradient value in the first source image to obtain a second pixel point set;

[0199] Based on the first pixel point set and the second pixel point set, the preset mapping relationship is updated to obtain the first mapping relationship; wherein the preset mapping relationship is a mapping relationship between a camera and an optical machine.

[0200] In some embodiments, the determining unit 802 is further configured to map the second pixel point set to the captured image based on the preset mapping relationship to obtain a third pixel point set;

[0201] Taking the third pixel point as the center point of the third window, if there is a first pixel point in the third window, taking the first pixel point closest to the third pixel point as the new third pixel point;

[0202] Based on a new third pixel point set corresponding to the third pixel point set and a second pixel point set corresponding to the third pixel point set under the preset mapping relationship, updating the preset mapping relationship to obtain a new preset mapping relationship;

[0203] Based on the new preset mapping relationship, determining a structural similarity index between a first preset central area of ​​the first source image and a matching area of ​​the captured image;

[0204] Iteratively executing the calculation of the structural similarity index P times to obtain P structural similarity indexes;

[0205] A maximum value among the P structural similarity indexes is determined, and a mapping relationship corresponding to the maximum value is used as the first mapping relationship.

[0206] In some embodiments, the determination unit 802 is further configured to determine first coordinate information of a plurality of pixel points in the current projection image in a world coordinate system with the ranging sensor as the origin;

[0207] Based on the rotation and translation relationship between the ranging sensor and the optical machine, and the first coordinate information of the multiple pixel points in the current projection picture, determine the second coordinate information of the multiple pixel points in the first source image in the world coordinate system with the optical machine as the origin;

[0208] Determine, based on the rotational translation relationship between the optical machine and the camera, and the second coordinate information of the plurality of pixels in the first source image, the third coordinate information of the plurality of pixels in the captured image in a world coordinate system with the camera as the origin;

[0209] The first mapping relationship is obtained based on the third coordinate information of multiple pixel points in the captured image and the second coordinate information of multiple pixel points in the first source image.

[0210] In some embodiments, the determination unit 802 is further configured to divide the preset central area of ​​the captured image into M*N grids;

[0211] Determine the variance of target color component values ​​of a plurality of pixel points in each grid; wherein the target color component value is any one of a red component value, a green component value, and a blue component value;

[0212] Counting the number of grids when the variance is greater than or equal to a first threshold;

[0213] When the number of grids is greater than or equal to a second threshold, a first mapping relationship between the captured image and the first source image is determined.

[0214] In some embodiments, the determination unit 802 is further configured to perform feature point matching between the captured image and the first source image, and determine the number of feature points that match between the captured image and the first source image;

[0215] When the number of feature points is greater than or equal to a third threshold, a first mapping relationship between the captured image and the first source image is determined.

[0216] In some embodiments, the determination unit 802 is further configured to extract a boundary region of the curtain in the captured image;

[0217] Using the first parameter as an input parameter of a preset gradient value calculation formula to obtain a first gradient value calculation formula; wherein different corner point coordinates correspond to different preset gradient value calculation formulas;

[0218] Based on the first gradient value calculation formula, determine a plurality of first gradient values ​​corresponding to a plurality of pixel points on the boundary area;

[0219] Using the second parameter as an input parameter of the preset gradient value calculation formula to obtain a second gradient value calculation formula;

[0220] Based on the second gradient value calculation formula, determine a plurality of second gradient values ​​corresponding to a plurality of pixel points on the boundary area;

[0221] Based on a plurality of first gradient values ​​and a plurality of second gradient values ​​corresponding to a plurality of pixel points on the boundary area, the corresponding corner point coordinates of the curtain are determined.

[0222] In some embodiments, the determining unit 802 is further configured to determine a first maximum value among the plurality of first gradient values, and determine a second maximum value among the plurality of second gradient values;

[0223] In the case that the first maximum value and the second maximum value correspond to the same target pixel point, the coordinates of the target pixel point are used as the corresponding corner point coordinates of the curtain.

[0224] The present application also provides another projection device, Fig. 9 Schematic diagram of the structure of the projection control device in the embodiment of the present application. Fig. 9 As shown, the projection control device 90 includes: a processor 901 and a memory 902 configured to store a computer program that can be run on the processor;

[0225] The processor 901 is configured to execute the method steps in the aforementioned embodiment when running a computer program.

[0226] Of course, in practical applications, Fig. 9As shown, the various components in the projection control device 90 are coupled together via a bus system 903. It can be understood that the bus system 903 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 903 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Fig. 9 Various buses are labeled as bus system 903.

[0227] In practical applications, the processor may be at least one of an application-specific integrated circuit (ASIC), a digital signal processing device (DSPD), a programmable logic device (PLD), a field-programmable gate array (FPGA), a controller, a microcontroller, and a microprocessor. It is understandable that for different devices, the electronic device used to implement the functions of the processor may also be other, and the embodiments of the present application do not specifically limit this.

[0228] The above-mentioned memory can be a volatile memory (volatile memory), such as a random access memory (RAM); or a non-volatile memory (non-volatile memory), such as a read-only memory (ROM), a flash memory, a hard disk (HDD) or a solid-state drive (SSD); or a combination of the above-mentioned types of memory, and provide instructions and data to the processor.

[0229] In an exemplary embodiment, the present application also provides a computer-readable storage medium for storing a computer program.

[0230] Optionally, the computer-readable storage medium can be applied to any one of the methods in the embodiments of the present application, and the computer program enables the computer to execute the corresponding processes implemented by the processor in each method in the embodiments of the present application. For the sake of brevity, they are not repeated here.

[0231] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0232] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0233] In addition, all functional units in the embodiments of the present invention may be integrated into one processing module, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units. A person of ordinary skill in the art may understand that all or part of the steps of implementing the above method embodiments may be completed by hardware related to program instructions, and the above program may be stored in a computer-readable storage medium, which, when executed, executes the steps of the above method embodiments; and the above storage medium includes various media that can store program codes, such as mobile storage devices, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0234] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0235] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0236] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0237] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A projection control method, characterized in that: The projection control method comprises: Acquire a captured image of a current projection picture projected on a projection plane, and intercept a first source image of the current projection picture; Determining a first mapping relationship between the captured image and the first source image; In the case where the coordinates of the corner points of the curtain in the captured image are determined, the projection coordinates of the first source image are determined based on the first mapping relationship and the coordinates of the corner points of the curtain in the captured image; Based on the projection coordinates of the first source image, a second source image is projected.

2. The projection control method according to claim 1, characterized in that: The determining a first mapping relationship between the captured image and the first source image includes: Determining gradient values ​​of a plurality of pixel points in the captured image; Using the first window to filter the local maximum gradient value in the captured image to obtain a first pixel point set; Determining gradient values ​​of a plurality of pixel points in the first source image; Using a second window to filter the local maximum gradient value in the first source image to obtain a second pixel point set; Based on the first pixel point set and the second pixel point set, the preset mapping relationship is updated to obtain the first mapping relationship; wherein the preset mapping relationship is a mapping relationship between a camera and an optical machine.

3. The projection control method according to claim 2, characterized in that: The updating of the preset mapping relationship based on the first pixel point set and the second pixel point set to obtain the first mapping relationship includes: Based on the preset mapping relationship, mapping the second pixel point set to the captured image to obtain a third pixel point set; Taking the third pixel point as the center point of the third window, if there is a first pixel point in the third window, taking the first pixel point closest to the third pixel point as the new third pixel point; Based on a new third pixel point set corresponding to the third pixel point set and a second pixel point set corresponding to the third pixel point set under the preset mapping relationship, updating the preset mapping relationship to obtain a new preset mapping relationship; Based on the new preset mapping relationship, determining a structural similarity index between a first preset central area of ​​the first source image and a matching area of ​​the captured image; Iteratively executing the calculation of the structural similarity index P times to obtain P structural similarity indexes; A maximum value among the P structural similarity indexes is determined, and a mapping relationship corresponding to the maximum value is used as the first mapping relationship.

4. The projection control method according to claim 1, characterized in that: The determining a first mapping relationship between the captured image and the first source image includes: Determine first coordinate information of a plurality of pixel points in the current projection image in a world coordinate system with the ranging sensor as the origin; Based on the rotation and translation relationship between the ranging sensor and the optical machine, and the first coordinate information of the multiple pixel points in the current projection picture, determine the second coordinate information of the multiple pixel points in the first source image in the world coordinate system with the optical machine as the origin; Determine, based on the rotational translation relationship between the optical machine and the camera, and the second coordinate information of the plurality of pixels in the first source image, the third coordinate information of the plurality of pixels in the captured image in a world coordinate system with the camera as the origin; The first mapping relationship is obtained based on the third coordinate information of multiple pixel points in the captured image and the second coordinate information of multiple pixel points in the first source image.

5. The projection control method according to any one of claims 1 to 4, characterized in that: The determining a first mapping relationship between the captured image and the first source image includes: Dividing a preset central area of ​​the captured image into M*N grids; Determine the variance of target color component values ​​of a plurality of pixel points in each grid; wherein the target color component value is any one of a red component value, a green component value, and a blue component value; Counting the number of grids when the variance is greater than or equal to a first threshold; When the number of grids is greater than or equal to a second threshold, a first mapping relationship between the captured image and the first source image is determined.

6. The projection control method according to any one of claims 1 to 4, characterized in that: The determining a first mapping relationship between the captured image and the first source image includes: Perform feature point matching between the captured image and the first source image to determine the number of feature points that match between the captured image and the first source image; When the number of feature points is greater than or equal to a third threshold, a first mapping relationship between the captured image and the first source image is determined.

7. The projection control method according to any one of claims 1 to 4, characterized in that: The determining the coordinates of the corner points of the curtain in the captured image includes: Extracting a boundary area of ​​the curtain in the captured image; Using the first parameter as an input parameter of a preset gradient value calculation formula to obtain a first gradient value calculation formula; wherein different corner point coordinates correspond to different preset gradient value calculation formulas; Based on the first gradient value calculation formula, determine a plurality of first gradient values ​​corresponding to a plurality of pixel points on the boundary area; Using the second parameter as an input parameter of the preset gradient value calculation formula to obtain a second gradient value calculation formula; Based on the second gradient value calculation formula, determine a plurality of second gradient values ​​corresponding to a plurality of pixel points on the boundary area; Based on a plurality of first gradient values ​​and a plurality of second gradient values ​​corresponding to a plurality of pixel points on the boundary area, the corresponding corner point coordinates of the curtain are determined.

8. The projection control method according to claim 7, characterized in that: The determining the corresponding corner point coordinates of the curtain based on the plurality of first gradient values ​​and the plurality of second gradient values ​​corresponding to the plurality of pixel points on the boundary area comprises: determining a first maximum value among the plurality of first gradient values, and determining a second maximum value among the plurality of second gradient values; In the case that the first maximum value and the second maximum value correspond to the same target pixel point, the coordinates of the target pixel point are used as the corresponding corner point coordinates of the curtain.

9. A projection control device, characterized in that: The projection control device comprises: an acquisition unit, used to acquire a captured image collected for a current projection picture projected on a projection plane, and a capture unit, used to capture a first source image of the current projection picture; a determining unit, configured to determine a first mapping relationship between the captured image and the first source image; The determining unit is further configured to determine the projection coordinates of the first source image based on the first mapping relationship and the coordinates of the corner points of the curtain in the captured image when the coordinates of the corner points of the curtain in the captured image are determined; The projection unit is used to project the second source image based on the projection coordinates of the first source image.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images