Projection obstacle avoidance method and device, projection equipment and storage medium

Abstract

The invention discloses a projection obstacle avoidance method and device, projection equipment and a storage medium, and the method comprises the steps: obtaining a first gray image, collected by a two-dimensional camera, of a first wall surface region, and obtaining a first depth image, collected by a three-dimensional camera, of the first wall surface region; carrying out obstacle identification on the first grayscale image and the first depth image to obtain an identification result; under the condition that the recognition result represents that the light machine can perform projection in the first wall area, acquiring a second grayscale image except the obstacle in the first grayscale image, and acquiring a second depth image except the obstacle in the first depth image; determining a second wall area of the to-be-projected wall in the second grayscale image based on the second depth image; determining an optimal projection area based on a second wall area of the to-be-projected wall in the second grayscale image; and executing the projection operation of the ray machine based on the optimal projection area.

Description

Technical Field

[0001] The present application relates to the technical field of projection equipment, and in particular to a projection obstacle avoidance method, device, projection equipment and storage medium. Background Art

[0002] Smart projection equipment projects images onto projection media such as curtains, screens, and walls through optical and mechanical components for users to watch. In the process of using smart projection equipment, when projecting onto curtains or screens, the image generally needs to be adaptively aligned behind the curtain, and when projecting onto walls, the image may appear trapezoidal in shape. Automatic trapezoidal correction can adjust the image to a rectangular projection image with a standard aspect ratio. However, the situation on the wall is more complicated than that on the curtains and screens. There may be switches, cabinets, photo frames, posters, air conditioners, dressing tables, and other possible household items on the wall, and there are also corners at the junction of walls. When the above-mentioned obstacles, corners, and other interferences appear in the projection area on the wall, the user needs to manually move the projection device to avoid the obstacles, which is very inconvenient to use. Summary of the invention

[0003] The present application hopes to provide a projection obstacle avoidance method, device, projection equipment and storage medium.

[0004] The technical solution of this application is implemented as follows:

[0005] In a first aspect, a projection obstacle avoidance method is provided, which is applied to a projection device, wherein the projection device includes a two-dimensional camera, a three-dimensional camera, and an optical machine; the method includes:

[0006] Acquire a first grayscale image of a first wall area captured by the two-dimensional camera, and acquire a first depth image of the first wall area captured by the three-dimensional camera;

[0007] Performing obstacle recognition on the first grayscale image and the first depth image to obtain a recognition result;

[0008] The recognition result indicates that the optical machine is capable of acquiring a second grayscale image excluding the obstacle in the first grayscale image and acquiring a second depth image excluding the obstacle in the first depth image when performing projection in the first wall area;

[0009] Determine a second wall area of ​​the wall to be projected in the second grayscale image based on the second depth image;

[0010] Determine an optimal projection area based on a second wall area of ​​the wall to be projected in the second grayscale image;

[0011] Based on the optimal projection area, the projection operation of the optical machine is performed.

[0012] In the above solution, the determining of the second wall area of ​​the wall to be projected in the second grayscale image based on the second depth image includes:

[0013] Converting the second depth image into a three-dimensional point cloud in a world coordinate system;

[0014] Converting the three-dimensional point cloud in the world coordinate system into a two-dimensional top-view image;

[0015] Determine a wall point cloud of the wall to be projected based on the two-dimensional top view image;

[0016] The wall point cloud of the wall to be projected is mapped to the second grayscale image to obtain a second wall area of ​​the wall to be projected in the second grayscale image.

[0017] In the above solution, determining the wall point cloud of the wall to be projected based on the two-dimensional top view image includes:

[0018] Obtaining a minimum circumscribed rectangle of a point cloud region in the two-dimensional top view image;

[0019] Obtaining a first wall point cloud on the boundary of the minimum circumscribed rectangle;

[0020] Performing morphological expansion on the first wall point cloud, and determining the intersection with the two-dimensional top-view image, to obtain all wall point clouds in the two-dimensional top-view image;

[0021] When it is determined that there is at least one wall corner point in the first wall point cloud, all wall point clouds in the two-dimensional top view image are divided into at least two groups of wall point clouds based on the line connecting the projection point and each wall corner point; wherein the projection point is the projection point of the origin of the world coordinate system in the two-dimensional top view;

[0022] The group of wall point clouds with the largest number among the at least two groups of wall point clouds is used as the wall point cloud of the wall to be projected.

[0023] In the above solution, the step of obtaining the first wall point cloud on the boundary of the minimum circumscribed rectangle includes:

[0024] Construct a ray between the projection point and all points on the boundary of the minimum circumscribed rectangle, retain the wall point on the same ray that is farthest from the projection point, and obtain the first wall point cloud.

[0025] In the above solution, after obtaining the first wall point cloud on the boundary of the minimum circumscribed rectangle, the method further includes:

[0026] sorting the first wall point cloud based on the slope between the projection point and each wall point in the first wall point cloud to obtain a sorted first wall point cloud;

[0027] Determine the curvature corresponding to the target wall point based on the target wall point and the N wall points before and after the target wall point; wherein the target wall point is any wall point in the second wall point cloud, and the second wall point cloud is the wall point cloud after excluding the first and last N wall points in the sorted first wall point cloud;

[0028] The wall points with a curvature greater than a preset threshold are taken as the wall corner points.

[0029] In the above solution, the step of obtaining the second wall area of ​​the wall to be projected in the second grayscale image includes:

[0030] The non-obstacle area is expanded with each wall point mapped to the second grayscale image as the center, and the expansion is stopped when other wall points or obstacle points are encountered to obtain the second wall area.

[0031] In the above solution, the determining of the optimal projection area based on the second wall area of ​​the wall to be projected in the second grayscale image includes:

[0032] Mapping four vertices of the projection screen of the optical machine to the second grayscale image to obtain an irregular quadrilateral area;

[0033] Determine an intersection area of ​​the irregular trapezoidal area and a second wall area of ​​the wall to be projected;

[0034] The optimal projection area is determined based on the intersection area.

[0035] In the above solution, determining the optimal projection area based on the intersection area includes:

[0036] Obtain the minimum bounding rectangle of the intersection area;

[0037] Based on the size of the minimum circumscribed rectangle of the intersection area, construct a sliding window of corresponding size;

[0038] Scanning the intersection area through the sliding window, and when it is determined that all the points in the sliding window are wall points, recording at least one center position of the sliding window and at least one distance between the center position of the sliding window and the center position of the irregular trapezoidal area;

[0039] determining a shortest distance among the at least one distance;

[0040] A sliding window corresponding to the center position is determined based on the shortest distance, and is expanded in equal proportion until an obstacle point is encountered and the expansion is stopped to obtain the optimal projection area.

[0041] In the above solution, the performing of the projection operation of the optical machine based on the optimal projection area includes:

[0042] Mapping the four vertices of the optimal projection area to the world coordinate system, and obtaining the maximum inscribed rectangle of a preset aspect ratio;

[0043] Based on the relationship between the world coordinate system and the optical machine coordinate system and the four vertices of the maximum inscribed rectangle, determine the four vertices of the projection screen of the optical machine;

[0044] The projection operation of the optical machine is performed based on four vertices of the projection picture of the optical machine.

[0045] In a second aspect, a projection obstacle avoidance device is provided, which is applied to a projection device, wherein the projection device includes a two-dimensional camera, a three-dimensional camera, and an optical machine; the device includes:

[0046] an acquisition unit, configured to acquire a first grayscale image of a first wall area acquired by the two-dimensional camera, and acquire a first depth image of the first wall area acquired by the three-dimensional camera;

[0047] an identification unit, configured to perform obstacle identification on the first grayscale image and the first depth image to obtain an identification result;

[0048] The acquisition unit is further configured to acquire, when the recognition result indicates that the optical machine can perform projection in the first wall area, a second grayscale image excluding the obstacle in the first grayscale image, and acquire a second depth image excluding the obstacle in the first depth image;

[0049] a determining unit, configured to determine a second wall area of ​​the wall to be projected in the second grayscale image based on the second depth image; and to determine an optimal projection area based on the second wall area of ​​the wall to be projected in the second grayscale image;

[0050] An execution unit is used to execute the projection operation of the optical machine based on the optimal projection area.

[0051] According to a third aspect, a projection device is provided, comprising: a processor and a memory configured to store a computer program that can be run on the processor, wherein the processor is configured to execute the steps of the method according to the first aspect when running the computer program.

[0052] According to a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, wherein the computer program implements the steps of the method according to the first aspect when executed by a processor.

[0053] The present application discloses a projection obstacle avoidance method, device, equipment and storage medium. The present application obtains a second grayscale image and a second depth image by performing obstacle recognition on a first grayscale image acquired by a two-dimensional camera and a first depth image acquired by a three-dimensional image, thereby automatically avoiding obstacles without the need for a user to manually move the projection device to avoid obstacles. Furthermore, through the second depth image, a second wall area of ​​the wall to be projected is determined in the second grayscale image, thereby avoiding the projection image of the optical machine from being projected on objects with a color close to that of the wall and on the corner edge between walls, thereby ensuring the projection effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a schematic diagram of the first process of the projection obstacle avoidance method in an embodiment of the present application;

[0055] Figure 2 This is a second flow chart of the projection obstacle avoidance method in an embodiment of the present application;

[0056] Figure 3 A schematic diagram of a two-dimensional top view image exemplified in an embodiment of the present application;

[0057] Figure 4 Schematic diagram of the flow of a method for determining a wall point cloud of a wall to be projected in an embodiment of the present application.

[0058] Figure 5 This is a schematic diagram of the composition structure of the projection obstacle avoidance device in the embodiment of the present application;

[0059] Figure 6 It is a schematic diagram of the composition structure of the projection device in an embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0061] The present application embodiment provides a projection obstacle avoidance method, Figure 1 The first flow diagram of the projection obstacle avoidance method in the embodiment of the present application is applied to a projection device, which includes a two-dimensional camera (2D camera), a three-dimensional camera (3D camera) and an optical machine. Among them, the 3D camera can be a structured light camera, a time-of-flight camera, a stereo vision camera and a binocular stereo camera. The field of view (FOV) of the 2D camera and the 3D camera is greater than the FOV of the optical machine lens, ensuring that the perception field of view can cover the projection screen.

[0062] like Figure 1 As shown, the projection obstacle avoidance method may specifically include:

[0063] Step 101: Acquire a first grayscale image of a first wall region captured by a two-dimensional camera, and acquire a first depth image of the first wall region captured by a three-dimensional camera.

[0064] The first wall area refers to the range in which the projection image of the optical machine can be projected.

[0065] The image content in the first grayscale image includes at least one of the following: at least one wall, an obstacle, an object with a color texture close to that of the wall (such as a ceiling), and a corner edge between walls. The image content in the first depth image is the same as the image content in the first grayscale image.

[0066] The grayscale image captured by the two-dimensional camera includes length information, width information and grayscale information, and the depth image captured by the three-dimensional camera includes length information, width information and depth information.

[0067] Step 102: performing obstacle recognition on the first grayscale image and the first depth image to obtain a recognition result.

[0068] In a feasible implementation, obstacles in the first grayscale image and the first depth image may be identified through a deep learning neural network.

[0069] In another feasible implementation method, the wall can be extracted through the characteristics of similar grayscale and globally high grayscale value, small depth curvature, basically consistent plane normals formed by point neighborhood, and obvious edges between the wall and obstacles, and the remaining part is the obstacle.

[0070] Step 103: The recognition result indicates that the optical machine can obtain a second grayscale image excluding obstacles in the first grayscale image and obtain a second depth image excluding obstacles in the first depth image when the optical machine is projected in the first wall area.

[0071] In some embodiments, when the recognition result indicates that the obstacle in the first grayscale image and / or the first depth image is a dynamic object (such as a person, pet, etc.), if the anti-shooting eye function is turned on, the anti-shooting eye function is triggered; if the anti-shooting eye function is turned off, a prompt message is output to prompt the user that the projection device is blocked, or to prompt the user to move the pet away; and the monitoring is updated at a preset interval until the dynamic object moves out of the picture, which indicates that the optical machine can project within the first wall area.

[0072] In some embodiments, the recognition result indicates that an obstacle appears in a first preset area (eg, around the image) of the first grayscale image and / or the first depth image, which indicates that the optical machine can perform projection in the first wall area.

[0073] In some embodiments, the recognition result indicates that no obstacle is recognized and a preset area of ​​the entire image is blocked. The second preset area of ​​the first depth image (for example, the center 1 / 3 area of ​​the upper, lower, left, and right sides of the image) is processed and converted into a three-dimensional point cloud and measured. The point cloud distances within the second preset area are sorted. If the median is less than a threshold, most of the second preset area is blocked, and the user is prompted.

[0074] In some embodiments, the recognition result indicates that an obstacle appears in a second preset area of ​​the first grayscale image and / or the first depth image (for example, the center 1 / 3 area of ​​the upper, lower, left, and right sides of the image), which indicates that the optical machine cannot perform normal projection in the first wall area, prompting the user.

[0075] Obstacles refer to objects that are clearly distinguishable from walls, such as switches, cabinets, photo frames, posters, air conditioners, dressing tables, and other possible household items.

[0076] Step 104: Determine a second wall area of ​​the wall to be projected in the second grayscale image based on the second depth image.

[0077] It should be noted that the second depth image and the second grayscale image do not include obstacles that are obviously different from the wall, but may include objects that are mistakenly identified as walls, that is, objects with color and texture close to the wall (such as the ceiling) and corners between walls. Therefore, the present application uses the second depth image to determine the second wall area of ​​the wall to be projected in the second grayscale image, thereby avoiding the projection of the optical machine on the ceiling and / or corners, thereby ensuring the projection effect.

[0078] Step 105: Determine an optimal projection area based on a second wall area of ​​the wall to be projected in the second grayscale image.

[0079] In some embodiments, step 105 may specifically include: mapping four vertices of the projection screen of the optical machine to the second grayscale image to obtain an irregular quadrilateral region;

[0080] Determine an intersection area of ​​the irregular trapezoidal area and a second wall area of ​​the wall to be projected;

[0081] The optimal projection area is determined based on the intersection area.

[0082] In the embodiment of the present application, according to the relationship between the optical machine coordinate system and the world coordinate system, the relationship between the world coordinate system and the three-dimensional camera coordinate system, and the relationship between the three-dimensional camera coordinate system and the two-dimensional camera coordinate system, the four vertices of the optical machine's projection screen are mapped to the second grayscale image. Since there is a certain angle between the projection device and the projection medium (i.e., the first wall area), the lines connecting the four vertices will form an irregular trapezoidal area similar to a trapezoid. The optimal projection area is determined by the intersection area of ​​the irregular trapezoidal area and the second wall area in the second grayscale image.

[0083] In some embodiments, determining the optimal projection area based on the intersection area includes:

[0084] Obtain the minimum bounding rectangle of the intersection area;

[0085] Based on the size of the minimum circumscribed rectangle of the intersection area, construct a sliding window of corresponding size;

[0086] Scanning the intersection area through the sliding window, and when it is determined that all the points in the sliding window are wall points, recording at least one center position of the sliding window and at least one distance between the center position of the sliding window and the center position of the irregular trapezoidal area;

[0087] determining a shortest distance among the at least one distance;

[0088] A sliding window corresponding to the center position is determined based on the shortest distance, and is expanded in equal proportion until an obstacle point is encountered and the expansion is stopped to obtain the optimal projection area.

[0089] In an embodiment of the present application, a minimum bounding rectangle of an intersection area in a second grayscale image is obtained, and a sliding window is constructed on the intersection area. The size of the sliding window may be 1 / 2 of the size of the minimum bounding rectangle, and the sliding step length is s. The intersection area is scanned by a sliding window. When the sliding window is full of wall points, at least one center position of the sliding window and at least one distance between the center position of the sliding window and the center position of the irregular trapezoidal area are recorded. The at least one distance between the center position of the sliding window and the center position of the irregular trapezoidal area is sorted from large to small or from small to large, and the center position of the sliding window corresponding to the shortest distance is obtained as the optimal projection position. The upper, lower, left, and right sides of the sliding window are expanded in equal proportion until the expansion is stopped before the obstacle point is contained in the sliding window to obtain the optimal projection area.

[0090] Step 106: Based on the optimal projection area, perform the projection operation of the optical machine.

[0091] In some embodiments, step 106 may specifically include: mapping four vertices of the optimal projection area to a world coordinate system, and obtaining a maximum inscribed rectangle of a preset aspect ratio;

[0092] Based on the relationship between the world coordinate system and the optical machine coordinate system and the four vertices of the maximum inscribed rectangle, determine the four vertices of the projection screen of the optical machine;

[0093] The projection operation of the optical machine is performed based on four vertices of the projection picture of the optical machine.

[0094] In the embodiment of the present application, for the best projection area on the second grayscale image, its four vertices are mapped to the world coordinate system to obtain the maximum inscribed rectangle of the preset aspect ratio (e.g., 16:9, 16:10, 4:3); then, through the conversion relationship between the world coordinate system and the optical-mechanical coordinate system, and the calibration parameters, the four vertices of the maximum inscribed rectangle are mapped to the optical-mechanical projection screen, and the projection screen correction is completed. When the optical-mechanical component of the projection device projects the corrected projection screen, it can simultaneously complete the automatic focus to ensure the viewing experience.

[0095] Here, the execution entity of steps 101 to 106 may be a processor of the projection device.

[0096] By adopting the above-mentioned technical scheme, the present application obtains a second grayscale image and a second depth image by performing obstacle recognition on the first grayscale image captured by the two-dimensional camera and the first depth image captured by the three-dimensional image, thereby automatically avoiding obstacles without the need for the user to manually move the projection device to avoid obstacles; further, through the second depth image, the second wall area of ​​the wall to be projected is determined in the second grayscale image, thereby avoiding the projection image of the optical machine from being projected on objects with a color close to that of the wall and on the corners between walls, thereby ensuring the projection effect.

[0097] In order to better reflect the purpose of this application, further examples are given based on the above embodiments of this application. Figure 2 : is a second flow chart of the projection obstacle avoidance method in the embodiment of the present application, such as Figure 2 As shown, the projection obstacle avoidance method specifically includes:

[0098] Step 201: Acquire a first grayscale image of a first wall region captured by a two-dimensional camera, and acquire a first depth image of the first wall region captured by a three-dimensional camera.

[0099] The first wall area refers to the range in which the projection image of the optical machine can be projected.

[0100] The image content in the first grayscale image includes at least one of the following: at least one wall, an obstacle, an object with a color texture close to that of the wall (such as a ceiling), and a corner edge between walls. The image content in the first depth image is the same as the image content in the first grayscale image.

[0101] Step 202: performing obstacle recognition on the first grayscale image and the first depth image to obtain a recognition result.

[0102] Step 203: The recognition result indicates that the optical machine can obtain a second grayscale image excluding obstacles in the first grayscale image and a second depth image excluding obstacles in the first depth image when the optical machine is projected in the first wall area.

[0103] Step 204: Convert the second depth image into a three-dimensional point cloud in a world coordinate system.

[0104] According to the relationship between the pixel coordinate system and the 3D camera coordinate system, the second depth image (u, v, z t ) is converted into a 3D point cloud (x t ,y t ,z t ), and then according to the relationship between the 3D camera coordinate system and the world coordinate system, the 3D point cloud (x t ,y t ,z t ) is converted to (x w ,y w ,z w ),Right now:

[0105]

[0106]

[0107] Among them, f x 、f y is the focal length of the camera, c x 、c y Represents the offset of the camera optical axis in the image coordinate system. R and T are the rotation matrix and translation vector between the three-dimensional camera coordinate system and the world coordinate system. R a It is the rotation matrix composed of the pitch angle and roll angle measured by the accelerometer.

[0108] It should be noted that after identifying the obstacles in the first depth image, a second depth image without obstacles is obtained, and the image content of the second depth image includes at least one of the following: at least one wall, an object with a color and texture close to the wall (such as a ceiling), and a corner edge between walls.

[0109] Step 205: Convert the three-dimensional point cloud in the world coordinate system into a two-dimensional top-view image.

[0110] In the embodiment of the present application, if the projection direction of the projection device is the positive direction of the X axis, the left direction of the projection device is the positive direction of the Z axis, and the vertical downward direction of the projection device is the positive direction of the Y axis, the three-dimensional point cloud in the world coordinate system is projected to the XOZ plane according to the positive direction of the Y axis to obtain a two-dimensional top view image. The two-dimensional top view image includes at least one of the following: a wall point cloud of at least one wall and an object point cloud with a color and texture close to the wall.

[0111] Figure 3 A schematic diagram of a two-dimensional top view image of an example of an embodiment of the present application is shown in FIG. Figure 3 As shown, the non-obstacle point cloud is black for the target and white for the background. The non-obstacle point cloud here includes the wall point cloud of wall 1, the wall point cloud of wall 2 and the ceiling point cloud (i.e., the object point cloud with a color and texture close to the wall).

[0112] Step 206: Determine the wall point cloud of the wall to be projected based on the two-dimensional top view image.

[0113] Human eyes can see Figure 3 The number of wall point clouds of Wall 1 is greater than the number of wall point clouds of Wall 2, which means that the area of ​​Wall 1 is greater than the area of ​​Wall 2, so Wall 1 is the wall to be projected.

[0114] As for how to determine the wall point cloud of the wall to be projected based on the two-dimensional top view image, the embodiment of the present application provides a solution such as Figure 4 As shown, it may specifically include:

[0115] Step 401: Obtain the minimum bounding rectangle of the point cloud area in the two-dimensional top view image.

[0116] Step 402: Obtain the first wall point cloud on the minimum circumscribed rectangle boundary.

[0117] In some embodiments, step 402 may specifically include: constructing a ray between the projection point and all points of the minimum circumscribed rectangle boundary, retaining the wall point farthest from the projection point on the same ray, and obtaining the first wall point cloud. The projection point is the projection point of the origin of the world coordinate system in the two-dimensional top view.

[0118] Here, only the target point farthest from the projection point is retained on the same ray, that is, the wall point is retained, and the object points (such as the ceiling point) close to the wall color texture are removed.

[0119] Step 403: perform morphological dilation on the first wall point cloud, and determine the intersection with the two-dimensional top view image to obtain all wall point clouds in the two-dimensional top view image.

[0120] Step 404: When it is determined that there is at least one wall corner point in the first wall point cloud, all wall point clouds in the two-dimensional top view image are divided into at least two groups of wall point clouds based on the line connecting the projection point and each wall corner point; wherein the projection point is the projection point of the origin of the world coordinate system in the two-dimensional top view.

[0121] In some embodiments, after obtaining the first wall point cloud on the boundary of the minimum circumscribed rectangle, the method further includes: sorting the first wall point cloud based on the slope between the projection point and each wall point in the first wall point cloud to obtain a sorted first wall point cloud;

[0122] Based on the target wall point and the N wall points before and after the target wall point, determine the curvature corresponding to the target wall point; wherein the target wall point is any wall point in the second wall point cloud, and the second wall point cloud is the wall point cloud after excluding the first and last N wall points in the sorted first wall point cloud;

[0123] The wall points with a curvature greater than a preset threshold are taken as the wall corner points.

[0124] For each wall point P in the retained first wall point cloud i , sort according to the slope of the straight line formed by the projection point and each wall point in the first wall point cloud, and use the N = 5 points before and after it P j Calculate the curvature c. The curvature of the flat position is small, and the curvature of the corner point of the wall is large, so the corner point of the wall is obtained. The curvature calculation depends on the mathematical theory of vector sum, and the formula is as follows:

[0125]

[0126] Here, the wall points with a curvature greater than a preset threshold are considered as corner points of the wall. The preset curvature threshold can be set according to experiments or experience.

[0127] It should be noted that when the curvatures corresponding to all wall points are less than or equal to the preset curvature threshold, it is considered that there is no corner point on the wall, indicating that only one wall is photographed. In this way, all wall point clouds in the two-dimensional top view image are the wall point clouds of the wall to be projected.

[0128] Step 405: taking the set of wall point clouds with the largest number among the at least two sets of wall point clouds as the wall point cloud of the wall to be projected.

[0129] here, Figure 3 When the number of wall point clouds of wall 1 is greater than the number of wall point clouds of wall 2, it means that the area of ​​wall 1 is greater than the area of ​​wall 2, then wall 1 is the wall to be projected, that is, the second wall point cloud is used as the wall point cloud of the wall to be projected.

[0130] Step 207: Map the wall point cloud of the wall to be projected to the second grayscale image to obtain a second wall area of ​​the wall to be projected in the second grayscale image.

[0131] In some embodiments, step 207 may specifically include: expanding the non-obstacle area centered on each wall point mapped to the second grayscale image, and stopping the expansion when encountering other wall points or obstacle points to obtain the second wall area.

[0132] According to the relationship between the world coordinate system and the 3D camera coordinate system, and the relationship between the 3D camera coordinate system and the 2D camera coordinate system, the wall point cloud of the wall to be projected is mapped to the second grayscale image. The non-obstacle area is expanded with each wall point mapped to the second grayscale image as the center, and the expansion is stopped when other wall points or obstacle points are encountered to form the final second wall area.

[0133] Step 208: Determine the best projection area based on the second wall area of ​​the wall to be projected in the second grayscale image.

[0134] In some embodiments, step 208 may specifically include: mapping four vertices of the projection screen of the optical machine to the second grayscale image to obtain an irregular quadrilateral region;

[0135] Determine an intersection area of ​​the irregular trapezoidal area and a second wall area of ​​the wall to be projected;

[0136] The optimal projection area is determined based on the intersection area.

[0137] In the embodiment of the present application, according to the relationship between the optical machine coordinate system and the world coordinate system, the relationship between the world coordinate system and the three-dimensional camera coordinate system, and the relationship between the three-dimensional camera coordinate system and the two-dimensional camera coordinate system, the four vertices of the optical machine's projection screen are mapped to the second grayscale image. Since there is a certain angle between the projection device and the projection medium (i.e., the first wall area), the lines connecting the four vertices will form an irregular trapezoidal area similar to a trapezoid. The optimal projection area is determined by the intersection area of ​​the irregular trapezoidal area and the second wall area in the second grayscale image.

[0138] In some embodiments, determining the optimal projection area based on the intersection area includes:

[0139] Obtain the minimum bounding rectangle of the intersection area;

[0140] Based on the size of the minimum circumscribed rectangle of the intersection area, construct a sliding window of corresponding size;

[0141] Scanning the intersection area through the sliding window, and when it is determined that all the points in the sliding window are wall points, recording at least one center position of the sliding window and at least one distance between the center position of the sliding window and the center position of the irregular trapezoidal area;

[0142] determining a shortest distance among the at least one distance;

[0143] A sliding window corresponding to the center position is determined based on the shortest distance, and is expanded in equal proportion until an obstacle point is encountered and the expansion is stopped to obtain the optimal projection area.

[0144] In an embodiment of the present application, a minimum bounding rectangle of an intersection area in a second grayscale image is obtained, and a sliding window is constructed on the intersection area. The size of the sliding window may be 1 / 2 of the size of the minimum bounding rectangle, and the sliding step length is s. The intersection area is scanned by a sliding window. When the sliding window is full of wall points, at least one center position of the sliding window and at least one distance between the center position of the sliding window and the center position of the irregular trapezoidal area are recorded. The at least one distance between the center position of the sliding window and the center position of the irregular trapezoidal area is sorted from large to small or from small to large, and the center position of the sliding window corresponding to the shortest distance is obtained as the optimal projection position. The upper, lower, left, and right sides of the sliding window are expanded in equal proportion until the expansion is stopped before the obstacle point is contained in the sliding window to obtain the optimal projection area.

[0145] Step 209: Based on the optimal projection area, perform the projection operation of the optical machine.

[0146] In some embodiments, step 209 may specifically include: mapping four vertices of the optimal projection area to a world coordinate system, and obtaining a maximum inscribed rectangle of a preset aspect ratio;

[0147] Based on the relationship between the world coordinate system and the optical machine coordinate system and the four vertices of the maximum inscribed rectangle, determine the four vertices of the projection screen of the optical machine;

[0148] The projection operation of the optical machine is performed based on four vertices of the projection picture of the optical machine.

[0149] In the embodiment of the present application, for the best projection area on the second grayscale image, its four vertices are mapped to the world coordinate system to obtain the maximum inscribed rectangle of the preset aspect ratio (e.g., 16:9, 16:10, 4:3); then, through the conversion relationship between the world coordinate system and the optical-mechanical coordinate system, and the calibration parameters, the four vertices of the maximum inscribed rectangle are mapped to the optical-mechanical projection screen, and the projection screen correction is completed. When the optical-mechanical component of the projection device projects the corrected projection screen, it can simultaneously complete the automatic focus to ensure the viewing experience.

[0150] By adopting the above-mentioned technical scheme, the present application obtains a second grayscale image and a second depth image by performing obstacle recognition on the first grayscale image captured by the two-dimensional camera and the first depth image captured by the three-dimensional image, thereby automatically avoiding obstacles without the need for the user to manually move the projection device to avoid obstacles; further, through the second depth image, the second wall area of ​​the wall to be projected is determined in the second grayscale image, thereby avoiding the projection image of the optical machine from being projected on objects with a color close to that of the wall and on the corners between walls, thereby ensuring the projection effect.

[0151] In order to implement the method of the embodiment of the present application, based on the same inventive concept, the embodiment of the present application also provides a projection obstacle avoidance device, Figure 5 Schematic diagram of the composition structure of the projection obstacle avoidance device in the embodiment of the present application, which is applied to a projection device, and the projection device includes a two-dimensional camera, a three-dimensional camera and an optical machine; Figure 5 As shown, the projection obstacle avoidance device 50 includes:

[0152] An acquisition unit 501 is used to acquire a first grayscale image of a first wall area acquired by the two-dimensional camera, and to acquire a first depth image of the first wall area acquired by the three-dimensional camera;

[0153] The recognition unit 502 is used to perform obstacle recognition on the first grayscale image and the first depth image to obtain a recognition result;

[0154] The acquisition unit 501 is further configured to acquire, when the recognition result indicates that the optical machine can project within the first wall area, a second grayscale image excluding the obstacle in the first grayscale image, and acquire a second depth image excluding the obstacle in the first depth image;

[0155] A determination unit 503, configured to determine a second wall region of the wall to be projected in the second grayscale image based on the second depth image; and to determine an optimal projection region based on the second wall region of the wall to be projected in the second grayscale image;

[0156] The execution unit 504 is used to execute the projection operation of the optical machine based on the optimal projection area.

[0157] By adopting the above-mentioned technical scheme, the present application obtains a second grayscale image and a second depth image by performing obstacle recognition on the first grayscale image captured by the two-dimensional camera and the first depth image captured by the three-dimensional image, thereby automatically avoiding obstacles without the need for the user to manually move the projection device to avoid obstacles; further, through the second depth image, the second wall area of ​​the wall to be projected is determined in the second grayscale image, thereby avoiding the projection image of the optical machine from being projected on objects with a color close to that of the wall and on the corners between walls, thereby ensuring the projection effect.

[0158] In some embodiments, the determining unit 503 is specifically configured to convert the second depth image into a three-dimensional point cloud in a world coordinate system;

[0159] Converting the three-dimensional point cloud in the world coordinate system into a two-dimensional top-view image;

[0160] Determine a wall point cloud of the wall to be projected based on the two-dimensional top view image;

[0161] The wall point cloud of the wall to be projected is mapped to the second grayscale image to obtain a second wall area of ​​the wall to be projected in the second grayscale image.

[0162] In some embodiments, the determination unit 503 is specifically configured to obtain a minimum bounding rectangle of a point cloud region in the two-dimensional overhead image;

[0163] Obtaining a first wall point cloud on the boundary of the minimum circumscribed rectangle;

[0164] Performing morphological expansion on the first wall point cloud, and determining the intersection with the two-dimensional top-view image, to obtain all wall point clouds in the two-dimensional top-view image;

[0165] When it is determined that there is at least one wall corner point in the first wall point cloud, all wall point clouds in the two-dimensional top view image are divided into at least two groups of wall point clouds based on the line connecting the projection point and each wall corner point; wherein the projection point is the projection point of the origin of the world coordinate system in the two-dimensional top view;

[0166] The group of wall point clouds with the largest number among the at least two groups of wall point clouds is used as the wall point cloud of the wall to be projected.

[0167] In some embodiments, the determination unit 503 is specifically used to construct a ray between the projection point and all points of the minimum circumscribed rectangle boundary, retain the wall point farthest from the projection point on the same ray, and obtain the first wall point cloud.

[0168] In some embodiments, the determining unit 503 is specifically configured to sort the first wall point cloud based on the slope between the projection point and each wall point in the first wall point cloud to obtain a sorted first wall point cloud;

[0169] Based on the target wall point and the N wall points before and after the target wall point, determine the curvature corresponding to the target wall point; wherein the target wall point is any wall point in the second wall point cloud, and the second wall point cloud is the wall point cloud after excluding the first and last N wall points in the sorted first wall point cloud;

[0170] The wall points with a curvature greater than a preset threshold are taken as the wall corner points.

[0171] In some embodiments, the determination unit 503 is specifically configured to expand the non-obstacle area centered on each wall point mapped to the second grayscale image, and stop the expansion when encountering other wall points or obstacle points to obtain the second wall area.

[0172] In some embodiments, the determination unit 503 is specifically configured to map four vertices of the projection screen of the optical machine to the second grayscale image to obtain an irregular quadrilateral region;

[0173] Determine an intersection area of ​​the irregular trapezoidal area and a second wall area of ​​the wall to be projected;

[0174] The optimal projection area is determined based on the intersection area.

[0175] In some embodiments, the determination unit 503 is specifically configured to obtain a minimum bounding rectangle of the intersection area;

[0176] Based on the size of the minimum circumscribed rectangle of the intersection area, construct a sliding window of corresponding size;

[0177] Scanning the intersection area through the sliding window, and when it is determined that all the points in the sliding window are wall points, recording at least one center position of the sliding window and at least one distance between the center position of the sliding window and the center position of the irregular trapezoidal area;

[0178] determining a shortest distance among the at least one distance;

[0179] A sliding window corresponding to the center position is determined based on the shortest distance, and is expanded in equal proportion until an obstacle point is encountered and the expansion is stopped to obtain the optimal projection area.

[0180] In some embodiments, the execution unit 504 is specifically configured to map the four vertices of the optimal projection area to a world coordinate system, and obtain a maximum inscribed rectangle of a preset aspect ratio;

[0181] Based on the relationship between the world coordinate system and the optical machine coordinate system and the four vertices of the maximum inscribed rectangle, determine the four vertices of the projection screen of the optical machine;

[0182] The projection operation of the optical machine is performed based on four vertices of the projection picture of the optical machine.

[0183] The present application also provides another projection device, Figure 6 Schematic diagram of the structure of the projection device in the embodiment of the present application. Figure 6 As shown, the projection device 60 includes: a processor 601 and a memory 602 configured to store a computer program that can be run on the processor;

[0184] The processor 601 is configured to execute the method steps in the aforementioned embodiment when running a computer program.

[0185] Of course, in practical applications, Figure 6 As shown, the various components in the projection device 60 are coupled together via a bus system 603. It is understood that the bus system 603 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 603 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 6 Various buses are labeled as bus system 603.

[0186] In practical applications, the processor may be at least one of an application-specific integrated circuit (ASIC), a digital signal processing device (DSPD), a programmable logic device (PLD), a field-programmable gate array (FPGA), a controller, a microcontroller, and a microprocessor. It is understandable that for different devices, the electronic device used to implement the functions of the processor may also be other, and the embodiments of the present application do not specifically limit this.

[0187] The above-mentioned memory can be a volatile memory (volatile memory), such as a random access memory (RAM); or a non-volatile memory (non-volatile memory), such as a read-only memory (ROM), a flash memory, a hard disk (HDD) or a solid-state drive (SSD); or a combination of the above-mentioned types of memory, and provide instructions and data to the processor.

[0188] In an exemplary embodiment, the present application also provides a computer-readable storage medium for storing a computer program.

[0189] Optionally, the computer-readable storage medium can be applied to any one of the methods in the embodiments of the present application, and the computer program enables the computer to execute the corresponding processes implemented by the processor in each method in the embodiments of the present application. For the sake of brevity, they are not repeated here.

[0190] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0191] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0192] In addition, all functional units in the embodiments of the present invention may be integrated into one processing module, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units. A person of ordinary skill in the art may understand that all or part of the steps of implementing the above method embodiments may be completed by hardware related to program instructions, and the above program may be stored in a computer-readable storage medium, which, when executed, executes the steps of the above method embodiments; and the above storage medium includes various media that can store program codes, such as mobile storage devices, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0193] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0194] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0195] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0196] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A projection obstacle avoidance method, applied to a projection device, characterized in that: The projection device includes a two-dimensional camera, a three-dimensional camera and an optical machine; the method includes: Acquire a first grayscale image of a first wall area captured by the two-dimensional camera, and acquire a first depth image of the first wall area captured by the three-dimensional camera; Performing obstacle recognition on the first grayscale image and the first depth image to obtain a recognition result; The recognition result indicates that the optical machine is capable of acquiring a second grayscale image excluding the obstacle in the first grayscale image and acquiring a second depth image excluding the obstacle in the first depth image when performing projection in the first wall area; Determine a second wall area of ​​the wall to be projected in the second grayscale image based on the second depth image; Determine an optimal projection area based on a second wall area of ​​the wall to be projected in the second grayscale image; Based on the optimal projection area, the projection operation of the optical machine is performed.

2. The method according to claim 1, characterized in that The determining, based on the second depth image, a second wall area of ​​the wall to be projected in the second grayscale image includes: Converting the second depth image into a three-dimensional point cloud in a world coordinate system; Converting the three-dimensional point cloud in the world coordinate system into a two-dimensional top-view image; Determine a wall point cloud of the wall to be projected based on the two-dimensional top view image; The wall point cloud of the wall to be projected is mapped to the second grayscale image to obtain a second wall area of ​​the wall to be projected in the second grayscale image.

3. The method according to claim 2, characterized in that The step of determining the wall point cloud of the wall to be projected based on the two-dimensional top view image includes: Obtaining a minimum circumscribed rectangle of a point cloud region in the two-dimensional top view image; Obtaining a first wall point cloud on the boundary of the minimum circumscribed rectangle; Performing morphological expansion on the first wall point cloud, and determining the intersection with the two-dimensional top-view image, to obtain all wall point clouds in the two-dimensional top-view image; When it is determined that there is at least one wall corner point in the first wall point cloud, all wall point clouds in the two-dimensional top view image are divided into at least two groups of wall point clouds based on the line connecting the projection point and each wall corner point; wherein the projection point is the projection point of the origin of the world coordinate system in the two-dimensional top view; The group of wall point clouds with the largest number among the at least two groups of wall point clouds is used as the wall point cloud of the wall to be projected.

4. The method according to claim 3, characterized in that The step of obtaining a first wall point cloud on the boundary of the minimum circumscribed rectangle includes: Construct a ray between the projection point and all points on the boundary of the minimum circumscribed rectangle, retain the wall point on the same ray that is farthest from the projection point, and obtain the first wall point cloud.

5. The method according to claim 3, characterized in that: After obtaining the first wall point cloud on the minimum circumscribed rectangle boundary, the method further includes: sorting the first wall point cloud based on the slope between the projection point and each wall point in the first wall point cloud to obtain a sorted first wall point cloud; Based on the target wall point and the N wall points before and after the target wall point, determine the curvature corresponding to the target wall point; wherein the target wall point is any wall point in the second wall point cloud, and the second wall point cloud is the wall point cloud after excluding the first and last N wall points in the sorted first wall point cloud; The wall points with a curvature greater than a preset threshold are taken as the wall corner points.

6. The method according to claim 2, characterized in that The step of obtaining a second wall area of ​​the wall to be projected in the second grayscale image includes: The non-obstacle area is expanded with each wall point mapped to the second grayscale image as the center, and the expansion is stopped when other wall points or obstacle points are encountered to obtain the second wall area.

7. The method according to claim 1, characterized in that The determining the optimal projection area based on the second wall area of ​​the wall to be projected in the second grayscale image includes: Mapping four vertices of the projection screen of the optical machine to the second grayscale image to obtain an irregular quadrilateral area; Determine an intersection area of ​​the irregular trapezoidal area and a second wall area of ​​the wall to be projected; The optimal projection area is determined based on the intersection area.

8. The method according to claim 7, characterized in that The determining the optimal projection area based on the intersection area includes: Obtain the minimum bounding rectangle of the intersection area; Based on the size of the minimum circumscribed rectangle of the intersection area, construct a sliding window of corresponding size; Scanning the intersection area through the sliding window, and when it is determined that all the points in the sliding window are wall points, recording at least one center position of the sliding window and at least one distance between the center position of the sliding window and the center position of the irregular trapezoidal area; determining a shortest distance among the at least one distance; A sliding window corresponding to the center position is determined based on the shortest distance, and is expanded in equal proportion until an obstacle point is encountered and the expansion is stopped to obtain the optimal projection area.

9. The method according to claim 1, characterized in that: The performing the projection operation of the optical machine based on the optimal projection area includes: Mapping the four vertices of the optimal projection area to the world coordinate system, and obtaining the maximum inscribed rectangle of a preset aspect ratio; Based on the relationship between the world coordinate system and the optical machine coordinate system and the four vertices of the maximum inscribed rectangle, determine the four vertices of the projection screen of the optical machine; The projection operation of the optical machine is performed based on four vertices of the projection picture of the optical machine.

10. A projection obstacle avoidance device, applied to a projection device, characterized in that: The projection equipment includes a two-dimensional camera, a three-dimensional camera and an optical machine; the device includes: an acquisition unit, configured to acquire a first grayscale image of a first wall area acquired by the two-dimensional camera, and acquire a first depth image of the first wall area acquired by the three-dimensional camera; an identification unit, configured to perform obstacle identification on the first grayscale image and the first depth image to obtain an identification result; The acquisition unit is further configured to acquire, when the recognition result indicates that the optical machine can perform projection in the first wall area, a second grayscale image excluding the obstacle in the first grayscale image, and acquire a second depth image excluding the obstacle in the first depth image; a determining unit, configured to determine a second wall area of ​​the wall to be projected in the second grayscale image based on the second depth image; and to determine an optimal projection area based on the second wall area of ​​the wall to be projected in the second grayscale image; An execution unit is used to execute the projection operation of the optical machine based on the optimal projection area.

11. A projection device, characterized in that: The projection device comprises: a processor and a memory configured to store a computer program that can be run on the processor, Wherein, the processor is configured to execute the steps of the method according to any one of claims 1 to 9 when running the computer program.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images