Projector obstacle avoidance method, device, equipment and medium based on sensor and camera

Abstract

The present application belongs to the technical field of projectors and discloses a projector obstacle avoidance method, apparatus, device and medium based on sensors and cameras, wherein the method comprises: fitting the plane area of ​​the screen according to the measurement results of the TOF sensor; detecting obstacle information in the camera image; transforming the first plane equation and the normal direction of the first plane into the camera coordinate system; calculating the homography matrix between the camera image and the projector image; constructing a virtual image, back-projecting the obstacle information onto the virtual image, binarizing the virtual image according to the virtual coordinates of the obstacle to obtain a binarized image; extracting the maximum connected domain from the binarized image; projecting the unobstructed display area onto the camera image plane; using the homography matrix to transform the camera projection area to the projector image plane; transforming the image to be displayed into the obstacle avoidance area to obtain an obstacle avoidance projection image. The above method can more accurately locate the obstacle avoidance area and maximize the display of the image projected by the projector.

Description

Technical Field

[0001] The present application relates to the field of projector technology, for example, to a projector obstacle avoidance method, device, equipment and medium based on sensors and cameras. Background Art

[0002] When using an LCD projector for display, the usage environment and usage method have a significant impact on the display effect. On the one hand, if there is obvious dirt on the screen display area, or there is an obstruction between the projection and the screen, or in scenarios such as home entertainment, users usually choose a flat wall as the projection target. The wall often has posters, electrical switches, dirt and other objects, which will cause incomplete display or reduced display quality. When using a projector, the projector's optical axis is often difficult to be strictly perpendicular to the screen or wall, which will cause the projected image to be distorted. When the projector has a rotation angle α around the vertical direction or a rotation angle β around the horizontal direction, the display image will have trapezoidal distortion, affecting viewing.

[0003] Prior art, such as CN115150600A, discloses combining optical zoom to achieve image scaling with trapezoidal correction scenarios during the projector's projection process. The target display area of ​​the projector's projection image is determined based on the area of ​​the obstacle and the position of the optical zoom reference point, solving the problem of resolution loss in traditional trapezoidal correction technology during projector obstacle avoidance and display processes. However, this prior art relies solely on the obstacle area and the optical zoom reference point to determine the target display area of ​​the projector's projection image, resulting in low accuracy in projector obstacle avoidance. Prior art, such as CN114205570A, discloses performing feature comparison between two images to determine the mapping relationship between the projection chart and the projection surface in the world coordinate system. Based on the mapping relationship, the captured image is converted into a rectangular area in the world coordinate system, and the playback content is projected into the rectangular area. This prior art obtains a rectangular area through image correction and uses this rectangular area to replace the deformed projection area formed on the projection surface after the projector moves. This fails to eliminate the impact of obstacles between the projector and the screen on the image projected onto the screen.

[0004] In summary, the existing technology has the problem that it is unable to eliminate the influence of obstacles on the image projected on the screen by the projector and / or the accuracy of the projector's obstacle avoidance is not high. Summary of the Invention

[0005] The purpose of this application is to provide a projector obstacle avoidance method, device, equipment, and medium based on sensors and cameras, which can solve the problems in the prior art that obstacles cannot be eliminated from affecting the image projected on the screen and / or the accuracy of projector obstacle avoidance is low. To achieve the above purpose, this application provides a projector obstacle avoidance method based on sensors and cameras, including:

[0006] Obtaining a TOF sensor measurement result, and fitting a plane area of ​​the screen according to the TOF sensor measurement result to obtain a first plane equation and a normal direction of the first plane;

[0007] Acquire a camera image and detect obstacle information in the camera image; the camera image is obtained by photographing the projector image on the plane area with a camera;

[0008] According to the extrinsic parameters between the TOF sensor and the camera, the first plane equation and the first plane normal direction are transformed into a camera coordinate system to obtain a second plane equation and a second plane normal direction;

[0009] Calculating a homography matrix between the camera image and the projector image;

[0010] Constructing a virtual image, back-projecting the obstacle information onto the virtual image to obtain virtual coordinates of the obstacle; binarizing the virtual image according to the virtual coordinates of the obstacle to obtain a binarized image;

[0011] Extracting the maximum connected domain from the binary image to obtain an unobstructed display area;

[0012] Projecting the unobstructed display area onto a camera image plane to obtain a camera projection area;

[0013] The camera projection area is transformed into the projector image plane using the homography matrix to obtain an obstacle avoidance area; and the image to be displayed is transformed into the obstacle avoidance area to obtain an obstacle avoidance projection image.

[0014] Preferably, before fitting the plane area of ​​the screen according to the TOF sensor measurement result, the method further includes:

[0015] Selecting the TOF sensor and the camera;

[0016] Selecting a plurality of relative postures of the TOF sensor and the camera, wherein the relative postures satisfy the requirement that the field of view of the camera covers the projection range of the projector, and the field of view range of the TOF sensor overlaps the field of view range of the camera;

[0017] The relative pose parameters of the TOF sensor and the camera are calibrated under each relative pose, and the target relative pose is determined according to the relative pose parameters; wherein the relative pose parameters include the extrinsic parameters, the camera intrinsic parameters and the distortion parameters.

[0018] Preferably, detecting obstacle information in the camera image includes:

[0019] Converting the camera image into a camera grayscale image;

[0020] Performing image segmentation on the camera grayscale image according to image characteristics of the obstacle to obtain a segmented image; wherein the segmented image is a binary image;

[0021] Obstacle detection is performed on the segmented image to obtain the obstacle information.

[0022] Preferably, the calculating the homography matrix between the camera image and the projector image comprises:

[0023] Acquire a projection display area, and extract a first vertex coordinate set corresponding to the projection display area in the camera image;

[0024] Extracting a second vertex coordinate set corresponding to the projection display area in the projector image;

[0025] The homography matrix is ​​calculated based on the first set of vertex coordinates and the second set of vertex coordinates.

[0026] Preferably, the constructing of the virtual image includes:

[0027] Projecting vertices corresponding to the first vertex coordinate set onto the screen according to the camera intrinsic parameters, the distortion parameters, the second plane equation, and the second plane normal direction to obtain a set of projection points;

[0028] Connect all projection points in the projection point set to obtain a first projection area;

[0029] The area corresponding to the circumscribed rectangle of the first projection area is used as the second projection area;

[0030] The second projection area is discretized to obtain the virtual image.

[0031] Preferably, the back-projecting the obstacle information onto the virtual image to obtain the virtual coordinates of the obstacle includes:

[0032] Back-projecting the obstacle information onto the virtual image according to the camera intrinsic parameters, the distortion parameters, the second plane equation, and the second plane normal direction to obtain an obstacle back-projection area;

[0033] The coordinates of the obstacle back-projection area are discretized and rounded to obtain the virtual coordinates of the obstacle.

[0034] Preferably, projecting the barrier-free display area onto a camera image plane to obtain a camera projection area comprises:

[0035] The unobstructed display area is projected onto the camera image plane according to the camera intrinsic parameters, the distortion parameters, the second plane equation and the second plane normal direction to obtain the camera projection area.

[0036] Preferably, the step of transforming the image to be displayed into the obstacle avoidance area to obtain an obstacle avoidance projection image includes:

[0037] The image to be displayed is transformed into the obstacle avoidance area by image affine transformation or interpolation method to obtain the obstacle avoidance projection image.

[0038] This application provides a projector obstacle avoidance device based on sensors and cameras, including:

[0039] A plane area fitting module is used to obtain a TOF sensor measurement result, and fit the plane area of ​​the screen according to the TOF sensor measurement result to obtain a first plane equation and a first plane normal direction;

[0040] an obstacle information detection module, configured to acquire a camera image and detect obstacle information in the camera image; the camera image is obtained by photographing the projector image on the plane area with a camera;

[0041] a parameter transformation module, configured to transform the first plane equation and the first plane normal direction into a camera coordinate system according to extrinsic parameters between the TOF sensor and the camera, to obtain a second plane equation and a second plane normal direction;

[0042] A homography matrix transformation module, used to calculate the homography matrix between the camera image and the projector image;

[0043] A virtual image binarization module is used to construct a virtual image, back-project the obstacle information onto the virtual image to obtain virtual coordinates of the obstacle; and binarize the virtual image according to the virtual coordinates of the obstacle to obtain a binarized image;

[0044] A maximum connected domain extraction module is used to extract the maximum connected domain from the binary image to obtain an unobstructed display area;

[0045] An obstacle-free display area projection module, configured to project the obstacle-free display area onto a camera image plane to obtain a camera projection area;

[0046] The image transformation module is used to transform the camera projection area to the projector image plane using the homography matrix to obtain an obstacle avoidance area; and transform the image to be displayed into the obstacle avoidance area to obtain an obstacle avoidance projection image.

[0047] The present application also provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, it implements any of the above-mentioned projector obstacle avoidance methods based on sensors and cameras and / or any of the above-mentioned steps of the projector obstacle avoidance method based on sensors and cameras.

[0048] The present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it implements any of the above-mentioned projector obstacle avoidance methods based on sensors and cameras and / or any of the above-mentioned steps of the projector obstacle avoidance method based on sensors and cameras.

[0049] The present application discloses a sensor- and camera-based projector obstacle avoidance method, comprising obtaining a time-of-flight (TOF) sensor measurement result, fitting a planar area of ​​a screen based on the TOF sensor measurement result, and obtaining a first plane equation and a first plane normal direction. A camera image is obtained, and obstacle information in the camera image is detected. The camera image is obtained by capturing a projector image on the planar area with a camera. Based on the extrinsic parameters between the TOF sensor and the camera, the first plane equation and the first plane normal direction are transformed into a camera coordinate system to obtain a second plane equation and a second plane normal direction. A homography matrix is ​​calculated between the camera image and the projector image. A virtual image is constructed, and the obstacle information is back-projected onto the virtual image to obtain virtual coordinates of the obstacle. The virtual image is binarized based on the virtual coordinates of the obstacle to obtain a binarized image. A maximum connected domain is extracted from the binarized image to obtain an unobstructed display area. The unobstructed display area is projected onto the camera image plane to obtain a camera projection area. The camera projection area is transformed onto the projector image plane using the homography matrix to obtain an obstacle avoidance area. The image to be displayed is transformed into the obstacle avoidance area to obtain an obstacle avoidance projection image. Using a TOF sensor and camera, the obstacle-free area can be directly extracted on the screen. This method is simple and intuitive, highly flexible, and can be well integrated with other functions such as fusion keystone correction. This provides good compatibility when additional functions or sensors are needed. By combining the TOF sensor, camera, and the external parameters between them, the display area can be converted between different image planes, more accurately locating the obstacle avoidance area, eliminating the impact of obstacles on the image projected by the projector on the screen, and thus maximizing the display of the projector's projected image. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a schematic diagram of the relative positions of a projector and a screen according to an embodiment;

[0051] Figure 2 is a schematic diagram of the relative positions of a projector, a TOF sensor, and a camera according to an embodiment;

[0052] Figure 3 1 is a flow chart of a projector obstacle avoidance method based on sensors and cameras according to an embodiment;

[0053] Figure 4 Schematic diagram of a process for determining the relative position of a target according to an embodiment;

[0054] Figure 5 A schematic diagram of a process for detecting obstacle information according to an embodiment;

[0055] Figure 6 1 is a flow chart of calculating a homography matrix according to an embodiment;

[0056] Figure 7 This is a schematic block diagram of the structure of a projector obstacle avoidance device based on sensors and cameras according to one embodiment;

[0057] Figure 8 The figure is a schematic block diagram of the structure of a computer device according to an embodiment.

[0058] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0060] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "above", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of features, integers, steps, operations, elements, modules, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, modules, components, and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any module and all combinations of one or more associated listed items.

[0061] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0062] In one embodiment, referring to Figure 1-Figure 3 , is a flow chart of a projector obstacle avoidance method based on sensors and cameras disclosed in this application, the method comprising:

[0063] S1: Obtain a TOF sensor measurement result, and perform fitting on a plane area of ​​the screen according to the TOF sensor measurement result to obtain a first plane equation and a first plane normal direction.

[0064] The projector obstacle avoidance method of the embodiment of the present application is used in scenarios where there is an obstacle between the projector and the screen, there is dirt on the screen, or when the screen is a wall, there are posters or electrical switches on the wall.

[0065] The first plane equation is a parametric equation of the plane area of ​​the screen relative to the TOF sensor. The formula of the first plane equation is:

[0066] A1X+B1Y+C1Z+D1=0;

[0067] The normal direction of the first plane is expressed as The first plane normal direction is a vector.

[0068] S2: Acquire a camera image and detect obstacle information in the camera image; the camera image is obtained by photographing the projector image on the plane area with a camera.

[0069] When the obstacle avoidance function is triggered, the projector projects a pure white image, and the camera captures the projector image on the flat surface, generating a camera image. Because the projector projects a pure white image onto the screen, the image captured by the camera carries obstacle information, including obstacle type, range, and location.

[0070] S3: According to the extrinsic parameters between the TOF sensor and the camera, transform the first plane equation and the first plane normal direction into a camera coordinate system to obtain a second plane equation and a second plane normal direction.

[0071] The formula for the second plane equation is:

[0072] A2X+B2Y+C2Z+D2=0;

[0073] The normal direction of the second plane is expressed as

[0074] S4: Calculate the homography matrix between the camera image and the projector image.

[0075] Acquire a projection display area, and extract a first vertex coordinate set corresponding to the projection display area in the camera image;

[0076] Extracting a second vertex coordinate set corresponding to the projection display area in the projector image;

[0077] The homography matrix is ​​calculated based on the first set of vertex coordinates and the second set of vertex coordinates.

[0078] The homography matrix can reflect the conversion relationship between the camera image and the projector image. Optionally, the projector projects a patterned image onto the screen, and the camera captures the patterned image on the screen. By extracting feature points, the point pairs required for calculating the homography matrix are obtained.

[0079] S5: Constructing a virtual image, back-projecting the obstacle information onto the virtual image to obtain virtual coordinates of the obstacle; and binarizing the virtual image according to the virtual coordinates of the obstacle to obtain a binarized image.

[0080] Projecting vertices corresponding to the first vertex coordinate set onto the screen according to the camera intrinsic parameters, the distortion parameters, the second plane equation, and the second plane normal direction to obtain a set of projection points;

[0081] Connect all projection points in the projection point set to obtain a first projection area;

[0082] The area corresponding to the circumscribed rectangle of the first projection area is used as the second projection area;

[0083] The second projection area is discretized to obtain the virtual image.

[0084] The virtual image is a matrix of size n×n.

[0085] S6: Extracting the maximum connected domain from the binary image to obtain an unobstructed display area.

[0086] The maximum connected domain that meets specific proportions and shapes in the binary image is extracted using general search methods or computational geometry methods.

[0087] Combining the discretization parameters, the position or range of the maximum connected domain on the second projection area or virtual image can be obtained, which is the unobstructed display area observed on the screen after obstacle avoidance. In general, the unobstructed display area is a quadrilateral.

[0088] S7: Projecting the barrier-free display area onto a camera image plane to obtain a camera projection area.

[0089] The unobstructed display area is projected onto the camera image plane according to the camera intrinsic parameters, the distortion parameters, the second plane equation and the second plane normal direction to obtain the camera projection area.

[0090] S8: using the homography matrix to transform the camera projection area to the projector image plane to obtain an obstacle avoidance area; transforming the image to be displayed into the obstacle avoidance area to obtain an obstacle avoidance projection image.

[0091] The image to be displayed is transformed into the obstacle avoidance area by image affine transformation or interpolation method to obtain the obstacle avoidance projection image.

[0092] An image affine transformation allows for translation, rotation, and other operations on an image through a series of geometric transformations. This transformation maintains the image's flatness and parallelism. Flatness means that straight lines remain straight after an affine transformation. Parallelism means that parallel lines remain parallel after an affine transformation.

[0093] Image affine transformation includes three steps. The first step is to determine the parameters of the transformation matrix; the second step is to calculate the corresponding positions of the target image and the original image, which in the embodiment of the present application is to calculate the corresponding positions of the obstacle avoidance area and the image to be displayed; the third step is to assign a value to the position of the target image, which in the embodiment of the present application is to assign a value to the position of the obstacle avoidance area.

[0094] Interpolation methods include unilinear interpolation, bilinear interpolation, spline interpolation, edge-based image interpolation, original low-resolution image edge-based method, and region-guided image interpolation algorithm, which are not limited here.

[0095] A sensor- and camera-based projector obstacle avoidance method according to an embodiment of the present application includes obtaining time-of-flight (TOF) sensor measurement results, fitting a planar area of ​​a screen based on the TOF sensor measurement results, and obtaining a first plane equation and a first plane normal direction. A camera image is obtained, and obstacle information in the camera image is detected. The camera image is obtained by capturing a projector image on the planar area with the camera. Based on extrinsic parameters between the TOF sensor and the camera, the first plane equation and the first plane normal direction are transformed into a camera coordinate system to obtain a second plane equation and a second plane normal direction. A homography matrix is ​​calculated between the camera image and the projector image. A virtual image is constructed, and obstacle information is back-projected onto the virtual image to obtain virtual coordinates of the obstacle. The virtual image is binarized based on the virtual coordinates of the obstacle to obtain a binarized image. A maximum connected domain is extracted from the binarized image to obtain an unobstructed display area. The unobstructed display area is projected onto the camera image plane to obtain a camera projection area. The camera projection area is transformed onto the projector image plane using the homography matrix to obtain an obstacle avoidance area. The image to be displayed is transformed into the obstacle avoidance area to obtain an obstacle avoidance projection image. Using a TOF sensor and camera, the obstacle-free area can be directly extracted on the screen. This method is simple and intuitive, highly flexible, and can be well integrated with other functions such as fusion keystone correction. This provides good compatibility when additional functions or sensors are needed. By combining the TOF sensor, camera, and the external parameters between them, the display area can be converted between different image planes, more accurately locating the obstacle avoidance area, eliminating the impact of obstacles on the image projected by the projector on the screen, and thus maximizing the display of the projector's projected image.

[0096] In one embodiment, referring to Figure 4 Before fitting the plane area of ​​the screen according to the TOF sensor measurement result, the method further includes:

[0097] S11 ′: Select the TOF sensor and the camera.

[0098] Select a TOF sensor and camera that meets your technical specifications.

[0099] S12': selecting a plurality of relative postures of the TOF sensor and the camera, wherein the relative postures satisfy that the field of view of the camera covers the projection range of the projector, and the field of view range of the TOF sensor overlaps the field of view range of the camera.

[0100] The relative position of the TOF sensor and the camera is as follows Figure 2 As shown, it is ensured that the field of view of the camera completely covers the entire projection range of the projector, and the field of view of the TOF sensor and the field of view of the camera overlap as much as possible.

[0101] S13': calibrating relative pose parameters of the TOF sensor and the camera under each relative pose, and determining the target relative pose according to the relative pose parameters; wherein the relative pose parameters include the extrinsic parameters, camera intrinsic parameters and distortion parameters.

[0102] First, the intrinsic parameters and distortion parameters of the camera are calibrated. On this basis, the extrinsic parameters between the TOF sensor and the camera are further calibrated. The target relative pose between the TOF sensor and the camera is selected according to the parameter calibration accuracy and size requirements. The target relative pose is the optimal relative pose.

[0103] As mentioned above, before fitting the screen's planar area based on the TOF sensor's measurement results, the process also involves selecting a TOF sensor and camera. Multiple relative poses are selected, ensuring that the camera's field of view covers the projector's projection range and that the TOF sensor's field of view overlaps with the camera's. The relative pose parameters of the TOF sensor and camera are calibrated for each relative pose, and the target relative pose is determined based on the relative pose parameters. The relative pose parameters include extrinsic parameters, camera intrinsic parameters, and distortion parameters. The target relative pose between the TOF sensor and camera is selected based on the calibration accuracy and size requirements. The target relative pose is the optimal relative pose.

[0104] In one embodiment, referring to Figure 5 , the detecting obstacle information in the camera image includes:

[0105] S22: Convert the camera image into a camera grayscale image.

[0106] Before step S22 , the method further includes step S21 : acquiring a camera image.

[0107] The camera image is a color image, such as an RGB image. Methods for converting the camera image into a camera grayscale image include averaging, maximum-minimum-average, and weighted-average methods. The embodiment of this application uses the averaging method as an example.

[0108] The averaging method takes the average value of the RGB three-channel pixel values ​​of each pixel in the camera image as the pixel value of the corresponding pixel in the camera grayscale image.

[0109] S23: performing image segmentation on the camera grayscale image according to image characteristics of the obstacle to obtain a segmented image; wherein the segmented image is a binary image.

[0110] The method for performing image segmentation on the camera grayscale image may use a threshold-based segmentation method, a region-based segmentation method, or an edge detection-based segmentation method, which is not limited here.

[0111] The pixel value of each pixel in the obstacle area of ​​the segmented image is 0, and the pixel value of each pixel in other areas except the obstacle area is 1.

[0112] Optionally, a threshold-based segmentation method is used to perform image segmentation on the camera grayscale image to obtain an obstacle area, and the pixel value of each pixel in the obstacle area is assigned to 0 to obtain a segmented image.

[0113] S24: Perform obstacle detection on the segmented image to obtain the obstacle information.

[0114] Methods for performing obstacle detection on segmented images include sliding window-based target detection methods, texture-based target detection methods, and neural network-based target detection methods. The neural network-based target detection method can be R-CNN, Fast R-CNN, or Yolo, which is not limited here.

[0115] Obstacle information includes obstacle type, obstacle range and obstacle location.

[0116] As described above, detecting obstacle information in the camera image includes converting the camera image into a camera grayscale image, performing image segmentation on the camera grayscale image based on image characteristics of the obstacle, and obtaining a segmented image; wherein the segmented image is a binary image. Obstacle detection is performed on the segmented image to obtain the obstacle information. The obstacle information includes obstacle type, obstacle range, and obstacle location.

[0117] In one embodiment, referring to Figure 6 , the calculating of the homography matrix between the camera image and the projector image comprises:

[0118] S41: Acquire a projection display area, and extract a first vertex coordinate set corresponding to the projection display area in the camera image.

[0119] The projection display area is a quadrilateral, and the first vertex coordinate set includes the four vertices C of the projection display area. p 1. C p 2. C p 3 and C p 4.

[0120] S42: Extracting a second vertex coordinate set corresponding to the projection display area in the projector image.

[0121] The coordinates of the four vertices of the second vertex coordinate set are (0, 0), (0, height), (width, 0) and (width, height), respectively, where width represents the width of the projector image and height represents the height of the projector image.

[0122] C in the first vertex coordinate set p 1 corresponds to the point (0, 0) in the second vertex coordinate set, C p 2 corresponds to the point (0, height), C p 3 corresponds to point (width, 0), C p 4 corresponds to the (width, height) point.

[0123] S43: Calculating the homography matrix according to the first vertex coordinate set and the second vertex coordinate set.

[0124] A homography matrix is ​​calculated based on the correspondence between the points in the first vertex coordinate set and the points in the second vertex coordinate set.

[0125] The homography matrix can reflect the transformation relationship between the camera image and the projector image.

[0126] As described above, calculating the homography matrix between the camera image and the projector image includes obtaining the projected display area, extracting a first set of vertex coordinates corresponding to the projected display area in the camera image, extracting a second set of vertex coordinates corresponding to the projected display area in the projector image, and calculating the homography matrix based on the first and second vertex coordinates. The homography matrix can reflect the transformation relationship between the camera image and the projector image.

[0127] In one embodiment, constructing the virtual image includes:

[0128] S51: Projecting vertices corresponding to the first vertex coordinate set onto the screen according to the camera intrinsic parameters, the distortion parameters, the second plane equation, and the second plane normal direction to obtain a projection point set.

[0129] The four points in the first vertex coordinate set are projected onto the screen respectively to obtain four projection points C1, C2, C3 and C4, and the four projection points constitute a projection point set.

[0130] S52: Connect all projection points in the projection point set to obtain a first projection area.

[0131] Connect C1, C2, C3 and C4 in sequence to obtain a first projection area, which is a quadrilateral.

[0132] S53: Using the area corresponding to the circumscribed rectangle of the first projection area as the second projection area.

[0133] The second projection area is denoted as M.

[0134] S54: discretize the second projection area to obtain the virtual image.

[0135] The second projection area M is discretized with a fixed step size within the second projection area M to construct a matrix with all initial values ​​of 1 and a size of n×n. This matrix is ​​a single-channel virtual image.

[0136] The plane corresponding to the virtual image can be used for back-projecting obstacle information.

[0137] As described above, constructing a virtual image involves projecting the vertices corresponding to the first vertex coordinate set onto the screen based on the camera intrinsic parameters, distortion parameters, the second plane equation, and the direction of the second plane normal, to obtain a set of projection points. All projection points in the set of projection points are connected to obtain a first projection area. The area corresponding to the circumscribed rectangle of the first projection area is used as the second projection area. The second projection area is discretized to obtain a virtual image. The plane corresponding to the virtual image can be used for back-projecting obstacle information.

[0138] In one embodiment, back-projecting the obstacle information onto the virtual image to obtain virtual coordinates of the obstacle includes:

[0139] S55: Back-projecting the obstacle information onto the virtual image according to the camera intrinsic parameters, the distortion parameters, the second plane equation, and the second plane normal direction to obtain an obstacle back-projection area.

[0140] Specifically, the geometric information of the obstacle, such as the obstacle range and obstacle position in the obstacle information, is back-projected onto the virtual image to obtain the obstacle back-projection area.

[0141] S56: Discretize and round the coordinates of the obstacle back-projection area to obtain the virtual coordinates of the obstacle.

[0142] The rounding method may be rounding down, rounding up, or other rounding methods, which are not limited here. The pixel value corresponding to the virtual coordinate of the obstacle in the virtual image is 0.

[0143] Pixels with a pixel value of 0 in the virtual image represent obstacles, and image areas with a pixel value of 1 in the virtual image represent unobstructed display areas.

[0144] As described above, obstacle information is back-projected onto the virtual image to obtain the obstacle's virtual coordinates. This involves back-projecting the obstacle information onto the virtual image based on the camera's intrinsic parameters, distortion parameters, the second plane equation, and the direction of the second plane's normal to obtain the obstacle's back-projected area. The coordinates of the obstacle's back-projected area are discretized and rounded to integers to obtain the obstacle's virtual coordinates. Pixels with a pixel value of 0 in the virtual image represent obstacles, while image regions with a pixel value of 1 in the virtual image represent unobstructed display areas.

[0145] Reference Figure 7, is a schematic block diagram of the structure of a projector obstacle avoidance device based on sensors and cameras disclosed in this application, the device comprising:

[0146] A plane area fitting module 10 is used to obtain a TOF sensor measurement result, and to fit the plane area of ​​the screen according to the TOF sensor measurement result to obtain a first plane equation and a first plane normal direction;

[0147] The obstacle information detection module 20 is used to obtain a camera image and detect obstacle information in the camera image; the camera image is obtained by photographing the projector image on the plane area with a camera;

[0148] a parameter transformation module 30, configured to transform the first plane equation and the first plane normal direction into a camera coordinate system according to extrinsic parameters between the TOF sensor and the camera, to obtain a second plane equation and a second plane normal direction;

[0149] A homography matrix transformation module 40, configured to calculate a homography matrix between the camera image and the projector image;

[0150] A virtual image binarization module 50 is configured to construct a virtual image, back-project the obstacle information onto the virtual image to obtain virtual coordinates of the obstacle, and binarize the virtual image according to the virtual coordinates of the obstacle to obtain a binarized image.

[0151] A maximum connected domain extraction module 60 is configured to extract the maximum connected domain from the binary image to obtain an unobstructed display area;

[0152] An unobstructed display area projection module 70 is used to project the unobstructed display area onto a camera image plane to obtain a camera projection area;

[0153] The image transformation module 80 is used to transform the camera projection area to the projector image plane using the homography matrix to obtain an obstacle avoidance area; and transform the image to be displayed into the obstacle avoidance area to obtain an obstacle avoidance projection image.

[0154] As described above, the projector obstacle avoidance device based on sensors and cameras can implement the projector obstacle avoidance method based on sensors and cameras.

[0155] In one embodiment, the sensor and camera-based projector obstacle avoidance device further includes:

[0156] A sensor and camera selection module, configured to select the TOF sensor and the camera;

[0157] A relative posture selection module is used to select a plurality of relative postures of the TOF sensor and the camera, wherein the relative posture satisfies that the field of view of the camera covers the projection range of the projector, and the field of view range of the TOF sensor overlaps the field of view range of the camera;

[0158] The target relative pose determination module is used to calibrate the relative pose parameters of the TOF sensor and the camera under each relative pose, and determine the target relative pose according to the relative pose parameters; wherein the relative pose parameters include the extrinsic parameters, camera intrinsic parameters and distortion parameters.

[0159] In one embodiment, the obstacle information detection module 20 further includes:

[0160] A camera image conversion unit, configured to convert the camera image into a camera grayscale image;

[0161] An image segmentation unit, configured to segment the camera grayscale image according to image characteristics of the obstacle to obtain a segmented image; wherein the segmented image is a binary image;

[0162] The obstacle detection unit is used to perform obstacle detection on the segmented image to obtain the obstacle information.

[0163] In one embodiment, the homography matrix transformation module 40 further includes:

[0164] A first vertex coordinate set extraction unit is configured to obtain a projection display area and extract a first vertex coordinate set corresponding to the projection display area in the camera image;

[0165] A second vertex coordinate set extraction unit, configured to extract a second vertex coordinate set corresponding to the projection display area in the projector image;

[0166] A homography matrix calculation unit is configured to calculate the homography matrix according to the first vertex coordinate set and the second vertex coordinate set.

[0167] In one embodiment, the virtual image binarization module 50 further includes:

[0168] a first projection unit, configured to project vertices corresponding to the first vertex coordinate set onto the screen according to the camera intrinsic parameters, the distortion parameters, the second plane equation, and the second plane normal direction, to obtain a projection point set;

[0169] a projection point connecting unit, configured to connect all projection points in the projection point set to obtain a first projection area;

[0170] a second projection area construction unit, configured to use an area corresponding to a circumscribed rectangle of the first projection area as a second projection area;

[0171] The second projection area discretization unit is used to discretize the second projection area to obtain the virtual image.

[0172] In one embodiment, the virtual image binarization module 50 further includes:

[0173] an obstacle information back-projection unit, configured to back-project the obstacle information onto the virtual image according to the camera intrinsic parameters, the distortion parameters, the second plane equation, and the second plane normal direction, to obtain an obstacle back-projection area;

[0174] The obstacle virtual coordinate calculation unit is used to discretize the coordinates of the obstacle back-projection area and round them to integers to obtain the obstacle virtual coordinates.

[0175] In one embodiment, the barrier-free display area projection module 70 further includes:

[0176] An unobstructed display area projection unit is used to project the unobstructed display area onto the camera image plane according to the camera intrinsic parameters, the distortion parameters, the second plane equation and the second plane normal direction to obtain the camera projection area.

[0177] In one embodiment, the image transformation module 80 further includes:

[0178] An image transformation unit is used to transform the image to be displayed into the obstacle avoidance area by image affine transformation or interpolation method to obtain the obstacle avoidance projection image.

[0179] Reference Figure 8 In the embodiment of the present application, a computer device is also provided, the internal structure of which can be as follows: Figure 8As shown. The computer device includes a processor, memory, network interface and database connected via a system bus. The processor designed for the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating device, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store an accessible display area, etc. The network interface of the computer device is used to communicate with an external terminal via a network connection. Furthermore, the above-mentioned computer device may also be provided with an input device and a display screen, etc. When the computer program is executed by a processor, it realizes a projector obstacle avoidance method based on sensors and cameras, comprising the following steps: obtaining a TOF sensor measurement result, fitting a plane area of ​​a screen according to the TOF sensor measurement result, and obtaining a first plane equation and a first plane normal direction; obtaining a camera image, and detecting obstacle information in the camera image; the camera image is obtained by photographing a projector image on the plane area by a camera; transforming the first plane equation and the first plane normal direction into a camera coordinate system according to external parameters between the TOF sensor and the camera, and obtaining a second plane equation and a second plane normal direction; Normal direction; calculate the homography matrix between the camera image and the projector image; construct a virtual image, back-project the obstacle information to the virtual image to obtain the virtual coordinates of the obstacle; binarize the virtual image according to the virtual coordinates of the obstacle to obtain a binary image; perform maximum connected domain extraction on the binary image to obtain an obstacle-free display area; project the obstacle-free display area to the camera image plane to obtain a camera projection area; use the homography matrix to transform the camera projection area to the projector image plane to obtain an obstacle avoidance area; transform the image to be displayed into the obstacle avoidance area to obtain an obstacle avoidance projection image. Those skilled in the art will understand that Figure 8 The structure shown in is merely a block diagram of a portion of the structure related to the present application solution and does not constitute a limitation on the computer device to which the present application solution is applied.

[0180] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, a projector obstacle avoidance method based on a sensor and a camera is implemented, comprising the following steps: obtaining a TOF sensor measurement result, fitting a plane area of ​​a screen according to the TOF sensor measurement result, and obtaining a first plane equation and a first plane normal direction; obtaining a camera image, and detecting obstacle information in the camera image; the camera image is obtained by photographing a projector image on the plane area by a camera; transforming the first plane equation and the first plane normal direction into a camera coordinate system according to external parameters between the TOF sensor and the camera; Under the above, the equation of the second plane and the normal direction of the second plane are obtained; the homography matrix between the camera image and the projector image is calculated; a virtual image is constructed, and the obstacle information is back-projected onto the virtual image to obtain the virtual coordinates of the obstacle; the virtual image is binarized according to the virtual coordinates of the obstacle to obtain a binarized image; the maximum connected domain is extracted from the binarized image to obtain an obstacle-free display area; the obstacle-free display area is projected onto the camera image plane to obtain a camera projection area; the camera projection area is transformed into the projector image plane using the homography matrix to obtain an obstacle avoidance area; the image to be displayed is transformed into the obstacle avoidance area to obtain an obstacle avoidance projection image.

[0181] It can be understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0182] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided in this application and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0183] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, apparatus, article, or method comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, apparatus, article, or method. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, apparatus, article, or method comprising the element.

[0184] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A projector obstacle avoidance method based on sensors and cameras, characterized in that: include: Obtaining a TOF sensor measurement result, and fitting a plane area of ​​the screen according to the TOF sensor measurement result to obtain a first plane equation and a normal direction of the first plane; Acquire a camera image and detect obstacle information in the camera image; the camera image is obtained by photographing the projector image on the plane area with a camera; According to the extrinsic parameters between the TOF sensor and the camera, the first plane equation and the first plane normal direction are transformed into a camera coordinate system to obtain a second plane equation and a second plane normal direction; Calculating a homography matrix between the camera image and the projector image; Constructing a virtual image, and back-projecting the obstacle information onto the virtual image to obtain virtual coordinates of the obstacle; Binarizing the virtual image according to the virtual coordinates of the obstacle to obtain a binary image; Extracting the maximum connected domain from the binary image to obtain an unobstructed display area; Projecting the unobstructed display area onto a camera image plane to obtain a camera projection area; Using the homography matrix, the camera projection area is transformed to the projector image plane to obtain an obstacle avoidance area; The image to be displayed is transformed into the obstacle avoidance area to obtain an obstacle avoidance projection image.

2. The projector obstacle avoidance method based on sensors and cameras according to claim 1, characterized in that: Before fitting the plane area of ​​the screen according to the TOF sensor measurement result, the method further includes: Selecting the TOF sensor and the camera; Selecting a plurality of relative postures of the TOF sensor and the camera, wherein the relative postures satisfy the requirement that the field of view of the camera covers the projection range of the projector, and the field of view range of the TOF sensor overlaps the field of view range of the camera; The relative pose parameters of the TOF sensor and the camera are calibrated under each relative pose, and the target relative pose is determined according to the relative pose parameters; wherein the relative pose parameters include the extrinsic parameters, the camera intrinsic parameters and the distortion parameters.

3. The projector obstacle avoidance method based on sensors and cameras according to claim 1, characterized in that: The detecting obstacle information in the camera image includes: Converting the camera image into a camera grayscale image; Performing image segmentation on the camera grayscale image according to image characteristics of the obstacle to obtain a segmented image; wherein the segmented image is a binary image; Obstacle detection is performed on the segmented image to obtain the obstacle information.

4. The projector obstacle avoidance method based on sensors and cameras according to claim 2, characterized in that: The calculating of the homography matrix between the camera image and the projector image includes: Acquire a projection display area, and extract a first vertex coordinate set corresponding to the projection display area in the camera image; Extracting a second vertex coordinate set corresponding to the projection display area in the projector image; The homography matrix is ​​calculated based on the first set of vertex coordinates and the second set of vertex coordinates.

5. The projector obstacle avoidance method based on sensors and cameras according to claim 4, characterized in that: The constructing of the virtual image includes: Projecting vertices corresponding to the first vertex coordinate set onto the screen according to the camera intrinsic parameters, the distortion parameters, the second plane equation, and the second plane normal direction to obtain a set of projection points; Connect all projection points in the projection point set to obtain a first projection area; The area corresponding to the circumscribed rectangle of the first projection area is used as the second projection area; The second projection area is discretized to obtain the virtual image.

6. The projector obstacle avoidance method based on sensors and cameras according to claim 2, characterized in that: The back-projecting the obstacle information onto the virtual image to obtain virtual coordinates of the obstacle includes: Back-projecting the obstacle information onto the virtual image according to the camera intrinsic parameters, the distortion parameters, the second plane equation, and the second plane normal direction to obtain an obstacle back-projection area; The coordinates of the obstacle back-projection area are discretized and rounded to obtain the virtual coordinates of the obstacle.

7. The projector obstacle avoidance method based on sensors and cameras according to claim 2, characterized in that: The step of projecting the barrier-free display area onto a camera image plane to obtain a camera projection area includes: The unobstructed display area is projected onto the camera image plane according to the camera intrinsic parameters, the distortion parameters, the second plane equation and the second plane normal direction to obtain the camera projection area.

8. The projector obstacle avoidance method based on sensors and cameras according to claim 1, characterized in that: The step of transforming the image to be displayed into the obstacle avoidance area to obtain an obstacle avoidance projection image includes: The image to be displayed is transformed into the obstacle avoidance area by image affine transformation or interpolation method to obtain the obstacle avoidance projection image.

9. A projector obstacle avoidance device based on sensors and cameras, characterized in that: include: A plane area fitting module is used to obtain a TOF sensor measurement result, and fit the plane area of ​​the screen according to the TOF sensor measurement result to obtain a first plane equation and a first plane normal direction; an obstacle information detection module, configured to acquire a camera image and detect obstacle information in the camera image; the camera image is obtained by photographing the projector image on the plane area with a camera; a parameter transformation module, configured to transform the first plane equation and the first plane normal direction into a camera coordinate system according to extrinsic parameters between the TOF sensor and the camera, to obtain a second plane equation and a second plane normal direction; A homography matrix transformation module, used to calculate the homography matrix between the camera image and the projector image; A virtual image binarization module is used to construct a virtual image, back-project the obstacle information onto the virtual image, and obtain virtual coordinates of the obstacle; Binarizing the virtual image according to the virtual coordinates of the obstacle to obtain a binary image; A maximum connected domain extraction module is used to extract the maximum connected domain from the binary image to obtain an unobstructed display area; An obstacle-free display area projection module, configured to project the obstacle-free display area onto a camera image plane to obtain a camera projection area; An image transformation module, configured to transform the camera projection area to a projector image plane using the homography matrix to obtain an obstacle avoidance area; The image to be displayed is transformed into the obstacle avoidance area to obtain an obstacle avoidance projection image.

10. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the steps of the sensor-and-camera-based projector obstacle avoidance method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the sensor-and-camera-based projector obstacle avoidance method according to any one of claims 1 to 8 are implemented.

Images