Projector obstacle avoidance method and device based on sensor and camera, equipment and medium
Through the combination of TOF sensor and camera, the fitting equation and normal direction of the screen plane area are obtained, obstacle information is detected, the homography matrix is calculated, virtual images are constructed, and binarization and maximum connectivity domain extraction are solved. The projector is solved The problem of low obstacle avoidance accuracy has achieved high-precision obstacle avoidance effect.
Patent Information
- Application Number
- CN202310548094.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-15
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2043-05-15
AI Technical Summary
The prior art cannot effectively eliminate the impact of obstacles on the image projected by the projector on the screen, and the projector's obstacle avoidance accuracy is not high.
Using a combination of TOF sensor and camera, by obtaining the fitting equations and normal directions of the screen plane area, detecting obstacle information, calculating the homography matrix, constructing virtual images, performing binarization and maximum connectivity domain extraction, and determining that no The obstacle display area is projected to the projector image plane through a homography matrix to achieve obstacle avoidance.
It achieves higher accuracy obstacle avoidance, eliminates the impact of obstacles on the projector projected image, maximizes the projection effect of the display area, and has flexibility and compatibility.
Smart Images

Figure CN118972527A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of projectors, and for example, to a sensor- and camera-based projector obstacle avoidance method, device, equipment and medium. Background Art
[0002] When using an LCD projector for display, the use environment and usage method have a significant impact on the display effect. On the one hand, if there is obvious dirt in the screen display area, or there is an obstruction between the projection and the screen, or in scenarios such as home entertainment, users usually choose a flat wall as the projection target. There are often posters, electrical switches, dirt and other objects on the wall, which will cause incomplete display or reduced display quality. In the process of using the projector, the projector optical axis is often difficult to be strictly perpendicular to the screen or wall, which will cause the projection picture to be distorted. When the projector has a rotation angle α around the vertical direction, or a rotation angle β around the horizontal direction, the display picture will have trapezoidal distortion, affecting viewing.
[0003] Prior art such as CN115150600A discloses that during the projection process of the projector, the target display area of the projected image of the projector is determined according to the area of the obstacle and the position of the optical zoom reference point, which solves the problem of resolution loss in the process of the projector obstacle avoidance display and the like in the conventional trapezoidal correction technology. However, the prior art only relies on the obstacle area and the optical zoom reference point to determine the target display area of the projected image of the projector, and the accuracy of the projector obstacle avoidance is not high. Prior art such as CN114205570A discloses that the mapping relationship between the projection chart and the projection surface in the world coordinate system is determined by feature comparison of two images, and the rectangular area converted to the world coordinate system is obtained according to the mapping relationship, and the playback content is projected to the rectangular area. The prior art obtains a rectangular area by image correction, and uses the rectangular area to replace the deformed projection area formed on the projection surface after the projector moves, and cannot eliminate the influence of obstacles between the projector and the screen on the image projected by the projector on the screen.
[0004] In summary, the prior art has the problem that the influence of obstacles on the image projected on the screen by the projector cannot be eliminated and / or the accuracy of the obstacle avoidance of the projector is not high. Summary of the invention
[0005] The purpose of the present application is to provide a projector obstacle avoidance method, device, equipment and medium based on sensors and cameras, which can solve the problem that the existing technology cannot eliminate the influence of obstacles on the image projected by the projector on the screen and / or the accuracy of the projector obstacle avoidance is not high. To achieve the above purpose, the present application provides a projector obstacle avoidance method based on sensors and cameras, including:
[0006] Obtaining a TOF sensor measurement result, and fitting a plane area of the screen according to the TOF sensor measurement result to obtain a first plane equation and a first plane normal direction;
[0007] Acquire a camera image and detect obstacle information in the camera image; the camera image is obtained by photographing the projector image on the plane area with a camera;
[0008] According to the external parameters between the TOF sensor and the camera, the first plane equation and the first plane normal direction are transformed into a camera coordinate system to obtain a second plane equation and a second plane normal direction;
[0009] Calculating a homography matrix between the camera image and the projector image;
[0010] Constructing a virtual image, back-projecting the obstacle information onto the virtual image to obtain virtual coordinates of the obstacle; binarizing the virtual image according to the virtual coordinates of the obstacle to obtain a binarized image;
[0011] Extracting the maximum connected domain from the binary image to obtain an obstacle-free display area;
[0012] Projecting the barrier-free display area onto a camera image plane to obtain a camera projection area;
[0013] The camera projection area is transformed to the projector image plane using the homography matrix to obtain an obstacle avoidance area; the image to be displayed is transformed to the obstacle avoidance area to obtain an obstacle avoidance projection image.
[0014] Preferably, before fitting the plane area of the screen according to the measurement result of the TOF sensor, the method further includes:
[0015] Selecting the TOF sensor and the camera;
[0016] Selecting a plurality of relative postures of the TOF sensor and the camera, wherein the relative postures satisfy that the field of view of the camera covers the projection range of the projector, and the field of view range of the TOF sensor overlaps the field of view range of the camera;
[0017] The relative posture parameters of the TOF sensor and the camera are calibrated under each relative posture, and the target relative posture is determined according to the relative posture parameters; wherein the relative posture parameters include the external parameters, the camera internal parameters and the distortion parameters.
[0018] Preferably, the detecting obstacle information in the camera image includes:
[0019] Converting the camera image into a camera grayscale image;
[0020] Performing image segmentation on the camera grayscale image according to the image characteristics of the obstacle to obtain a segmented image; wherein the segmented image is a binary image;
[0021] Obstacle detection is performed on the segmented image to obtain the obstacle information.
[0022] Preferably, the calculating the homography matrix between the camera image and the projector image comprises:
[0023] Acquire a projection display area, and extract a first vertex coordinate set corresponding to the projection display area in the camera image;
[0024] Extracting a second vertex coordinate set corresponding to the projection display area in the projector image;
[0025] The homography matrix is calculated based on the first set of vertex coordinates and the second set of vertex coordinates.
[0026] Preferably, the constructing of a virtual image comprises:
[0027] According to the camera intrinsic parameters, the distortion parameters, the second plane equation and the second plane normal direction, projecting the vertices corresponding to the first vertex coordinate set onto the screen to obtain a projection point set;
[0028] Connect all projection points in the projection point set to obtain a first projection area;
[0029] Taking the area corresponding to the circumscribed rectangle of the first projection area as the second projection area;
[0030] The second projection area is discretized to obtain the virtual image.
[0031] Preferably, the back-projecting the obstacle information onto the virtual image to obtain the virtual coordinates of the obstacle includes:
[0032] Back-projecting the obstacle information onto the virtual image according to the camera intrinsic parameters, the distortion parameters, the second plane equation, and the second plane normal direction to obtain an obstacle back-projection area;
[0033] The coordinates of the obstacle back-projection area are discretized and rounded to obtain the virtual coordinates of the obstacle.
[0034] Preferably, projecting the barrier-free display area onto a camera image plane to obtain a camera projection area comprises:
[0035] According to the camera intrinsic parameters, the distortion parameters, the second plane equation and the second plane normal direction, the barrier-free display area is projected onto the camera image plane to obtain the camera projection area.
[0036] Preferably, the step of transforming the image to be displayed into the obstacle avoidance area to obtain an obstacle avoidance projection image includes:
[0037] The image to be displayed is transformed into the obstacle avoidance area by image affine transformation or interpolation method to obtain the obstacle avoidance projection image.
[0038] The present application provides a projector obstacle avoidance device based on a sensor and a camera, comprising:
[0039] A plane area fitting module, used to obtain a measurement result of a TOF sensor, and to fit a plane area of the screen according to the measurement result of the TOF sensor to obtain a first plane equation and a first plane normal direction;
[0040] An obstacle information detection module, used to obtain a camera image and detect obstacle information in the camera image; the camera image is obtained by photographing the projector image on the plane area with a camera;
[0041] A parameter transformation module, used for transforming the first plane equation and the first plane normal direction into a camera coordinate system according to the external parameters between the TOF sensor and the camera, to obtain a second plane equation and a second plane normal direction;
[0042] A homography matrix transformation module, used for calculating the homography matrix between the camera image and the projector image;
[0043] A virtual image binarization module is used to construct a virtual image, back-project the obstacle information onto the virtual image to obtain virtual coordinates of the obstacle; binarize the virtual image according to the virtual coordinates of the obstacle to obtain a binarized image;
[0044] A maximum connected domain extraction module, used to extract the maximum connected domain from the binary image to obtain an obstacle-free display area;
[0045] An obstacle-free display area projection module, used to project the obstacle-free display area onto a camera image plane to obtain a camera projection area;
[0046] The image transformation module is used to transform the camera projection area to the projector image plane using the homography matrix to obtain an obstacle avoidance area; and transform the image to be displayed into the obstacle avoidance area to obtain an obstacle avoidance projection image.
[0047] The present application also provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, it implements any of the above-mentioned sensor and camera-based projector obstacle avoidance methods and / or any of the above-mentioned sensor and camera-based projector obstacle avoidance methods.
[0048] The present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it implements any of the above-mentioned projector obstacle avoidance methods based on sensors and cameras and / or any of the above-mentioned steps of the projector obstacle avoidance method based on sensors and cameras.
[0049] The present application discloses a projector obstacle avoidance method based on a sensor and a camera, comprising obtaining a measurement result of a TOF sensor, fitting a plane area of a screen according to the measurement result of the TOF sensor, and obtaining a first plane equation and a first plane normal direction. Obtain a camera image, and detect obstacle information in the camera image. The camera image is obtained by photographing a projector image on a plane area with a camera. According to the external parameters between the TOF sensor and the camera, the first plane equation and the first plane normal direction are transformed into a camera coordinate system to obtain a second plane equation and a second plane normal direction. Calculate the homography matrix between the camera image and the projector image. Construct a virtual image, back-project the obstacle information onto the virtual image, and obtain the virtual coordinates of the obstacle. Binarize the virtual image according to the virtual coordinates of the obstacle to obtain a binarized image. Perform maximum connected domain extraction on the binarized image to obtain an obstacle-free display area. Project the obstacle-free display area onto the camera image plane to obtain the camera projection area. Use the homography matrix to transform the camera projection area to the projector image plane to obtain an obstacle avoidance area. Transform the image to be displayed into the obstacle avoidance area to obtain an obstacle avoidance projection image. Through TOF sensors and cameras, obstacle-free areas can be directly extracted on the screen. The method is simple and intuitive, with high flexibility, and can be well combined with other functions such as fusion keystone correction. When other functions or other sensors need to be added, it has good compatibility. By combining TOF sensors, cameras and the external parameters between the two, the display area can be converted between different image planes, and the obstacle avoidance area can be more accurately located, eliminating the impact of obstacles on the image projected by the projector on the screen, thereby maximizing the display of the image projected by the projector. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 is a schematic diagram of the relative positions of a projector and a screen according to an embodiment;
[0051] Figure 2 is a schematic diagram of the relative positions of a projector, a TOF sensor and a camera according to an embodiment;
[0052] Figure 3 1 is a flow chart of a projector obstacle avoidance method based on sensors and cameras according to an embodiment;
[0053] Figure 4 A schematic diagram of a process for determining a relative position and posture of a target according to an embodiment;
[0054] Figure 5 A schematic diagram of a process of detecting obstacle information according to an embodiment;
[0055] Figure 6 A schematic diagram of a process for calculating a homography matrix according to an embodiment;
[0056] Figure 7 It is a structural schematic block diagram of a projector obstacle avoidance device based on a sensor and a camera according to an embodiment;
[0057] Figure 8 The present invention is a schematic block diagram of the structure of a computer device according to an embodiment.
[0058] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0059] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0060] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "a", "an", "above", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of features, integers, steps, operations, elements, modules, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, modules, components, and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any module and all combinations of one or more associated listed items.
[0061] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as herein.
[0062] In one embodiment, referring to Figure 1-Figure 3 , is a flow chart of a projector obstacle avoidance method based on sensors and cameras disclosed in the present application, the method comprising:
[0063] S1: Obtain a measurement result of a TOF sensor, and fit a plane area of the screen according to the measurement result of the TOF sensor to obtain a first plane equation and a first plane normal direction.
[0064] The projector obstacle avoidance method of the embodiment of the present application is used in scenarios where there is an obstacle between the projector and the screen, there is dirt on the screen, or when the screen is a wall, there are posters or electrical switches on the wall.
[0065] The first plane equation is a parametric equation of the plane area of the screen relative to the TOF sensor. The formula of the first plane equation is:
[0066] A1X+B1Y+C1Z+D1=0;
[0067] The normal direction of the first plane is expressed as The first plane normal direction is a vector.
[0068] S2: Acquire a camera image and detect obstacle information in the camera image; the camera image is obtained by photographing the projector image on the plane area with a camera.
[0069] After the obstacle avoidance function is triggered, the projector projects a pure white image, and the camera captures the projector image on the plane area to obtain the camera image. Since the projector projects a pure white image to the screen at this time, the image captured by the camera carries obstacle information, including obstacle category, obstacle range, and obstacle location.
[0070] S3: According to the external parameters between the TOF sensor and the camera, transform the first plane equation and the first plane normal direction into a camera coordinate system to obtain a second plane equation and a second plane normal direction.
[0071] The formula for the second plane equation is:
[0072] A2X+B2Y+C2Z+D2=0;
[0073] The normal direction of the second plane is expressed as
[0074] S4: Calculate the homography matrix between the camera image and the projector image.
[0075] Acquire a projection display area, and extract a first vertex coordinate set corresponding to the projection display area in the camera image;
[0076] Extracting a second vertex coordinate set corresponding to the projection display area in the projector image;
[0077] The homography matrix is calculated based on the first set of vertex coordinates and the second set of vertex coordinates.
[0078] The homography matrix can reflect the conversion relationship between the camera image and the projector image. Optionally, the projector projects a patterned image onto the screen, and the camera captures the patterned image on the screen, and the point pairs required for calculating the homography matrix are obtained by extracting feature points.
[0079] S5: constructing a virtual image, back-projecting the obstacle information onto the virtual image to obtain virtual coordinates of the obstacle; binarizing the virtual image according to the virtual coordinates of the obstacle to obtain a binary image.
[0080] According to the camera intrinsic parameters, the distortion parameters, the second plane equation and the second plane normal direction, projecting the vertices corresponding to the first vertex coordinate set onto the screen to obtain a projection point set;
[0081] Connect all projection points in the projection point set to obtain a first projection area;
[0082] Taking the area corresponding to the circumscribed rectangle of the first projection area as the second projection area;
[0083] The second projection area is discretized to obtain the virtual image.
[0084] The virtual image is a matrix of size n×n.
[0085] S6: extracting the maximum connected domain from the binary image to obtain an obstacle-free display area.
[0086] The maximum connected domain that meets specific proportions and shapes in the binary image is extracted using general search methods or computational geometry methods.
[0087] Combining the discretization parameters, the position or range of the maximum connected domain on the second projection area or virtual image can be obtained, that is, the barrier-free display area observed on the screen after obstacle avoidance. In general, the barrier-free display area is a quadrilateral.
[0088] S7: Projecting the barrier-free display area onto a camera image plane to obtain a camera projection area.
[0089] According to the camera intrinsic parameters, the distortion parameters, the second plane equation and the second plane normal direction, the barrier-free display area is projected onto the camera image plane to obtain the camera projection area.
[0090] S8: using the homography matrix to transform the camera projection area to the projector image plane to obtain an obstacle avoidance area; transforming the image to be displayed into the obstacle avoidance area to obtain an obstacle avoidance projection image.
[0091] The image to be displayed is transformed into the obstacle avoidance area by image affine transformation or interpolation method to obtain the obstacle avoidance projection image.
[0092] Image affine transformation means that an image can be translated, rotated, and other operations through a series of geometric transformations. This transformation can maintain the straightness and parallelism of the image. Straightness means that after the image is transformed, the straight lines are still straight lines. Parallelism means that after the image is transformed, the parallel lines are still parallel lines.
[0093] Image affine transformation includes three steps. The first step is to determine the parameters of the transformation matrix; the second step is to calculate the corresponding positions of the target image and the original image, and in the embodiment of the present application, it is to calculate the corresponding positions of the obstacle avoidance area and the image to be displayed; the third step is to assign a value to the position of the target image, and in the embodiment of the present application, it is to assign a value to the position of the obstacle avoidance area.
[0094] Interpolation methods include unilinear interpolation, bilinear interpolation, spline interpolation, edge-based image interpolation, original low-resolution image edge-based methods, and region-guided image interpolation algorithms, which are not limited here.
[0095] A sensor- and camera-based projector obstacle avoidance method according to an embodiment of the present application includes obtaining a TOF sensor measurement result, fitting a plane area of a screen according to the TOF sensor measurement result, and obtaining a first plane equation and a first plane normal direction. Obtain a camera image and detect obstacle information in the camera image. The camera image is obtained by photographing a projector image on a plane area with a camera. According to the external parameters between the TOF sensor and the camera, the first plane equation and the first plane normal direction are transformed into a camera coordinate system to obtain a second plane equation and a second plane normal direction. Calculate the homography matrix between the camera image and the projector image. Construct a virtual image, back-project the obstacle information to the virtual image, and obtain the virtual coordinates of the obstacle. Binarize the virtual image according to the virtual coordinates of the obstacle to obtain a binarized image. Perform maximum connected domain extraction on the binarized image to obtain an obstacle-free display area. Project the obstacle-free display area onto the camera image plane to obtain the camera projection area. Use the homography matrix to transform the camera projection area to the projector image plane to obtain an obstacle avoidance area. Transform the image to be displayed into the obstacle avoidance area to obtain an obstacle avoidance projection image. Through TOF sensors and cameras, obstacle-free areas can be directly extracted on the screen. The method is simple and intuitive, with high flexibility, and can be well combined with other functions such as fusion keystone correction. When other functions or other sensors need to be added, it has good compatibility. By combining TOF sensors, cameras and the external parameters between the two, the display area can be converted between different image planes, and the obstacle avoidance area can be more accurately located, eliminating the impact of obstacles on the image projected by the projector on the screen, thereby maximizing the display of the image projected by the projector.
[0096] In one embodiment, referring to Figure 4 Before fitting the plane area of the screen according to the measurement result of the TOF sensor, the method further includes:
[0097] S11 ′: Select the TOF sensor and the camera.
[0098] Select a TOF sensor and camera that meets your technical specifications.
[0099] S12': Select a plurality of relative postures of the TOF sensor and the camera, wherein the relative postures satisfy that the field of view of the camera covers the projection range of the projector, and the field of view range of the TOF sensor overlaps the field of view range of the camera.
[0100] The relative position of the TOF sensor and the camera is as follows Figure 2 As shown, it is ensured that the field of view of the camera completely covers the entire projection range of the projector, and the field of view of the TOF sensor and the field of view of the camera overlap as much as possible.
[0101] S13': calibrating the relative posture parameters of the TOF sensor and the camera under each relative posture, and determining the target relative posture according to the relative posture parameters; wherein the relative posture parameters include the external parameters, the camera intrinsic parameters and the distortion parameters.
[0102] First, the intrinsic parameters and distortion parameters of the camera are calibrated. On this basis, the extrinsic parameters between the TOF sensor and the camera are further calibrated. The target relative pose between the TOF sensor and the camera is selected according to the parameter calibration accuracy and size requirements. The target relative pose is the optimal relative pose.
[0103] As mentioned above, before fitting the plane area of the screen according to the measurement results of the TOF sensor, it also includes selecting the TOF sensor and the camera, selecting a variety of relative postures of the TOF sensor and the camera, and the relative posture satisfies that the field of view of the camera covers the projection range of the projector, and the field of view range of the TOF sensor overlaps the field of view range of the camera. The relative posture parameters of the TOF sensor and the camera are calibrated under each relative posture, and the target relative posture is determined according to the relative posture parameters; wherein the relative posture parameters include external parameters, camera internal parameters and distortion parameters. The target relative posture between the TOF sensor and the camera is selected according to the parameter calibration accuracy and size requirements, and the target relative posture is the optimal relative posture.
[0104] In one embodiment, referring to Figure 5 , the detecting obstacle information in the camera image includes:
[0105] S22: Convert the camera image into a camera grayscale image.
[0106] Before step S22, the method further includes step S21: acquiring a camera image.
[0107] The camera image is a color image, such as an RGB image. Methods for converting the camera image into a camera grayscale image include an average method, a maximum-minimum average method, and a weighted average method. The embodiment of the present application takes the average method as an example.
[0108] The averaging method takes the average of the pixel values of the RGB three channels of each pixel in the camera image as the pixel value of the corresponding pixel in the camera grayscale image.
[0109] S23: performing image segmentation on the camera grayscale image according to image characteristics of the obstacle to obtain a segmented image; wherein the segmented image is a binary image.
[0110] The method for performing image segmentation on the camera grayscale image may use a threshold-based segmentation method, a region-based segmentation method, or an edge detection-based segmentation method, which is not limited here.
[0111] The pixel value of each pixel in the obstacle area of the segmented image is 0, and the pixel value of each pixel in other areas except the obstacle area is 1.
[0112] Optionally, a threshold-based segmentation method is used to perform image segmentation on the camera grayscale image to obtain an obstacle area, and a pixel value of each pixel in the obstacle area is assigned to 0 to obtain a segmented image.
[0113] S24: Perform obstacle detection on the segmented image to obtain the obstacle information.
[0114] Methods for performing obstacle detection on segmented images include a sliding window-based target detection method, a texture-based target detection method, and a neural network-based target detection method. The neural network-based target detection method may be R-CNN, Fast R-CNN, or Yolo, which is not limited here.
[0115] Obstacle information includes obstacle type, obstacle range and obstacle location.
[0116] As described above, detecting obstacle information in the camera image includes converting the camera image into a camera grayscale image, performing image segmentation on the camera grayscale image according to the image characteristics of the obstacle, and obtaining a segmented image; wherein the segmented image is a binary image. Obstacle detection is performed on the segmented image to obtain the obstacle information. The obstacle information includes obstacle category, obstacle range, and obstacle position.
[0117] In one embodiment, referring to Figure 6 , the calculating of the homography matrix between the camera image and the projector image comprises:
[0118] S41: Acquire a projection display area, and extract a first vertex coordinate set corresponding to the projection display area in the camera image.
[0119] The projection display area is a quadrilateral, and the first vertex coordinate set includes four vertices C of the projection display area. p 1. C p 2. C p 3 and C p 4.
[0120] S42: Extracting a second vertex coordinate set corresponding to the projection display area in the projector image.
[0121] The coordinates of the four vertices of the second vertex coordinate set are (0, 0), (0, height), (width, 0) and (width, height), respectively, where width represents the width of the projector image and height represents the height of the projector image.
[0122] C in the first vertex coordinate set p 1 corresponds to the point (0, 0) in the second vertex coordinate set, C p 2 corresponds to the point (0, height), C p 3 corresponds to (width, 0), C p 4 corresponds to the (width, height) point.
[0123] S43: Calculate the homography matrix according to the first vertex coordinate set and the second vertex coordinate set.
[0124] A homography matrix is calculated according to the correspondence between the points in the first vertex coordinate set and the points in the second vertex coordinate set.
[0125] The homography matrix can reflect the transformation relationship between the camera image and the projector image.
[0126] As described above, calculating the homography matrix between the camera image and the projector image includes obtaining the projection display area, extracting the first vertex coordinate set corresponding to the projection display area in the camera image, extracting the second vertex coordinate set corresponding to the projection display area in the projector image, and calculating the homography matrix based on the first vertex coordinate set and the second vertex coordinate set. The homography matrix can reflect the conversion relationship between the camera image and the projector image.
[0127] In one embodiment, constructing a virtual image includes:
[0128] S51: According to the camera intrinsic parameters, the distortion parameters, the second plane equation and the second plane normal direction, project the vertices corresponding to the first vertex coordinate set onto the screen to obtain a projection point set.
[0129] The four points in the first vertex coordinate set are projected onto the screen respectively to obtain four projection points C1, C2, C3 and C4, and the four projection points constitute a projection point set.
[0130] S52: Connect all projection points in the projection point set to obtain a first projection area.
[0131] C1, C2, C3 and C4 are connected in sequence to obtain a first projection area, which is a quadrilateral.
[0132] S53: taking the area corresponding to the circumscribed rectangle of the first projection area as the second projection area.
[0133] The second projection area is denoted by M.
[0134] S54: discretize the second projection area to obtain the virtual image.
[0135] The second projection area M is discretized with a fixed step size within the second projection area M to construct a matrix with all initial values being 1 and a size of n×n. The matrix is a single-channel virtual image.
[0136] The plane corresponding to the virtual image can be used for back-projecting obstacle information.
[0137] As described above, constructing a virtual image includes projecting the vertices corresponding to the first vertex coordinate set onto the screen according to the camera intrinsic parameters, the distortion parameters, the second plane equation and the second plane normal direction to obtain a projection point set. Connect all projection points in the projection point set to obtain a first projection area. The area corresponding to the circumscribed rectangle of the first projection area is used as the second projection area. Discretize the second projection area to obtain a virtual image. The plane corresponding to the virtual image can be used for back-projection obstacle information.
[0138] In one embodiment, the back-projecting the obstacle information onto the virtual image to obtain the virtual coordinates of the obstacle includes:
[0139] S55: Back-projecting the obstacle information onto the virtual image according to the camera intrinsic parameters, the distortion parameters, the second plane equation, and the second plane normal direction to obtain an obstacle back-projection area.
[0140] Specifically, the geometric information of the obstacle, such as the obstacle range and the obstacle position in the obstacle information, is back-projected onto the virtual image to obtain the obstacle back-projection area.
[0141] S56: Discretizing and rounding the coordinates of the obstacle back-projection area to obtain virtual coordinates of the obstacle.
[0142] The rounding method may be rounding down, rounding to the nearest integer, or other rounding methods, which are not limited here. The pixel value corresponding to the virtual coordinate of the obstacle in the virtual image is 0.
[0143] Pixels with a pixel value of 0 in the virtual image represent obstacles, and image areas with a pixel value of 1 in the virtual image represent obstacle-free display areas.
[0144] As described above, the obstacle information is back-projected onto the virtual image to obtain the virtual coordinates of the obstacle, including back-projecting the obstacle information onto the virtual image according to the camera intrinsic parameters, the distortion parameters, the second plane equation and the second plane normal direction to obtain the obstacle back-projection area. The coordinates of the obstacle back-projection area are discretized and rounded to obtain the virtual coordinates of the obstacle. Pixels with a pixel value of 0 in the virtual image represent obstacles, and image areas with a pixel value of 1 in the virtual image represent obstacle-free display areas.
[0145] Reference Figure 7, is a schematic structural block diagram of a projector obstacle avoidance device based on a sensor and a camera disclosed in the present application, the device comprising:
[0146] A plane area fitting module 10 is used to obtain a measurement result of a TOF sensor, and to fit the plane area of the screen according to the measurement result of the TOF sensor to obtain a first plane equation and a first plane normal direction;
[0147] The obstacle information detection module 20 is used to obtain a camera image and detect obstacle information in the camera image; the camera image is obtained by photographing the projector image on the plane area with a camera;
[0148] A parameter transformation module 30, configured to transform the first plane equation and the first plane normal direction into a camera coordinate system according to external parameters between the TOF sensor and the camera, to obtain a second plane equation and a second plane normal direction;
[0149] A homography matrix transformation module 40, used to calculate the homography matrix between the camera image and the projector image;
[0150] A virtual image binarization module 50 is used to construct a virtual image, back-project the obstacle information onto the virtual image to obtain virtual coordinates of the obstacle; and binarize the virtual image according to the virtual coordinates of the obstacle to obtain a binarized image;
[0151] A maximum connected domain extraction module 60 is used to extract the maximum connected domain from the binary image to obtain an unobstructed display area;
[0152] An obstacle-free display area projection module 70, used to project the obstacle-free display area onto a camera image plane to obtain a camera projection area;
[0153] The image transformation module 80 is used to transform the camera projection area to the projector image plane using the homography matrix to obtain an obstacle avoidance area; and transform the image to be displayed into the obstacle avoidance area to obtain an obstacle avoidance projection image.
[0154] As described above, the sensor-and-camera-based projector obstacle avoidance device can implement the sensor-and-camera-based projector obstacle avoidance method.
[0155] In one embodiment, the sensor and camera based projector obstacle avoidance device further comprises:
[0156] A sensor and camera selection module, used for selecting the TOF sensor and the camera;
[0157] A relative posture selection module, used to select a plurality of relative postures of the TOF sensor and the camera, wherein the relative posture satisfies that the field of view of the camera covers the projection range of the projector, and the field of view range of the TOF sensor overlaps the field of view range of the camera;
[0158] The target relative posture determination module is used to calibrate the relative posture parameters of the TOF sensor and the camera under each relative posture, and determine the target relative posture according to the relative posture parameters; wherein the relative posture parameters include the external parameters, the camera internal parameters and the distortion parameters.
[0159] In one embodiment, the obstacle information detection module 20 further includes:
[0160] A camera image conversion unit, used for converting the camera image into a camera grayscale image;
[0161] An image segmentation unit, used to perform image segmentation on the camera grayscale image according to the image characteristics of the obstacle to obtain a segmented image; wherein the segmented image is a binary image;
[0162] The obstacle detection unit is used to perform obstacle detection on the segmented image to obtain the obstacle information.
[0163] In one embodiment, the homography matrix transformation module 40 further includes:
[0164] A first vertex coordinate set extraction unit, used to obtain a projection display area, and extract a first vertex coordinate set corresponding to the projection display area in the camera image;
[0165] A second vertex coordinate set extraction unit, used to extract a second vertex coordinate set corresponding to the projection display area in the projector image;
[0166] A homography matrix calculation unit is used to calculate the homography matrix according to the first vertex coordinate set and the second vertex coordinate set.
[0167] In one embodiment, the virtual image binarization module 50 further includes:
[0168] A first projection unit, configured to project vertices corresponding to the first vertex coordinate set onto the screen according to the camera intrinsic parameters, the distortion parameters, the second plane equation, and the second plane normal direction, to obtain a projection point set;
[0169] A projection point connecting unit, used to connect all projection points in the projection point set to obtain a first projection area;
[0170] A second projection area construction unit, configured to use an area corresponding to a circumscribed rectangle of the first projection area as a second projection area;
[0171] The second projection area discretization unit is used to discretize the second projection area to obtain the virtual image.
[0172] In one embodiment, the virtual image binarization module 50 further includes:
[0173] an obstacle information back-projection unit, configured to back-project the obstacle information onto the virtual image according to the camera intrinsic parameters, the distortion parameters, the second plane equation, and the second plane normal direction, to obtain an obstacle back-projection area;
[0174] The obstacle virtual coordinate calculation unit is used to discretize and round the coordinates of the obstacle back-projection area to obtain the obstacle virtual coordinates.
[0175] In one embodiment, the barrier-free display area projection module 70 further includes:
[0176] An obstacle-free display area projection unit is used to project the obstacle-free display area onto the camera image plane according to the camera intrinsic parameters, the distortion parameters, the second plane equation and the second plane normal direction to obtain the camera projection area.
[0177] In one embodiment, the image transformation module 80 further includes:
[0178] An image transformation unit is used to transform the image to be displayed into the obstacle avoidance area by image affine transformation or interpolation method to obtain the obstacle avoidance projection image.
[0179] Reference Figure 8 In an embodiment of the present application, a computer device is also provided, the internal structure of which can be as follows: Figure 8As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor designed for the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating device, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store an accessible display area, etc. The network interface of the computer device is used to communicate with an external terminal via a network connection. Furthermore, the above-mentioned computer device can also be provided with an input device and a display screen, etc. When the computer program is executed by a processor, a sensor- and camera-based projector obstacle avoidance method is implemented, comprising the following steps: obtaining a TOF sensor measurement result, fitting a plane area of a screen according to the TOF sensor measurement result, and obtaining a first plane equation and a first plane normal direction; obtaining a camera image, and detecting obstacle information in the camera image; the camera image is obtained by photographing a projector image on the plane area with a camera; transforming the first plane equation and the first plane normal direction into a camera coordinate system according to external parameters between the TOF sensor and the camera, and obtaining a second plane equation and a second plane normal direction; Normal direction; calculate the homography matrix between the camera image and the projector image; construct a virtual image, back-project the obstacle information to the virtual image, and obtain the virtual coordinates of the obstacle; binarize the virtual image according to the virtual coordinates of the obstacle to obtain a binary image; perform maximum connected domain extraction on the binary image to obtain an obstacle-free display area; project the obstacle-free display area to the camera image plane to obtain the camera projection area; use the homography matrix to transform the camera projection area to the projector image plane to obtain the obstacle avoidance area; transform the image to be displayed into the obstacle avoidance area to obtain an obstacle avoidance projection image. Those skilled in the art will understand that Figure 8 The structure shown in is merely a block diagram of a portion of the structure related to the present application solution and does not constitute a limitation on the computer device to which the present application solution is applied.
[0180] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, a sensor- and camera-based projector obstacle avoidance method is implemented, comprising the following steps: obtaining a TOF sensor measurement result, fitting a plane area of a screen according to the TOF sensor measurement result, and obtaining a first plane equation and a first plane normal direction; obtaining a camera image, and detecting obstacle information in the camera image; the camera image is obtained by photographing a projector image on the plane area with a camera; transforming the first plane equation and the first plane normal direction into a camera coordinate system according to external parameters between the TOF sensor and the camera; , obtain the second plane equation and the second plane normal direction; calculate the homography matrix between the camera image and the projector image; construct a virtual image, back-project the obstacle information to the virtual image, and obtain the virtual coordinates of the obstacle; binarize the virtual image according to the virtual coordinates of the obstacle to obtain a binary image; perform maximum connected domain extraction on the binary image to obtain an obstacle-free display area; project the obstacle-free display area to the camera image plane to obtain the camera projection area; use the homography matrix to transform the camera projection area to the projector image plane to obtain the obstacle avoidance area; transform the image to be displayed into the obstacle avoidance area to obtain an obstacle avoidance projection image.
[0181] It can be understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.
[0182] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided in this application and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0183] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.
[0184] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. A projector obstacle avoidance method based on sensors and cameras, It is characterized in that include: Obtaining a TOF sensor measurement result, and fitting a plane area of the screen according to the TOF sensor measurement result to obtain a first plane equation and a first plane normal direction; Acquire a camera image and detect obstacle information in the camera image; the camera image is obtained by photographing the projector image on the plane area with a camera; According to the external parameters between the TOF sensor and the camera, the first plane equation and the first plane normal direction are transformed into a camera coordinate system to obtain a second plane equation and a second plane normal direction; Calculating a homography matrix between the camera image and the projector image; Constructing a virtual image, back-projecting the obstacle information onto the virtual image, and obtaining virtual coordinates of the obstacle; Binarizing the virtual image according to the virtual coordinates of the obstacle to obtain a binary image; Extracting the maximum connected domain from the binary image to obtain an obstacle-free display area; Projecting the barrier-free display area onto a camera image plane to obtain a camera projection area; Using the homography matrix, the camera projection area is transformed to the projector image plane to obtain an obstacle avoidance area; The image to be displayed is transformed into the obstacle avoidance area to obtain an obstacle avoidance projection image.
2. The projector obstacle avoidance method based on sensors and cameras according to claim 1, It is characterized in that Before fitting the plane area of the screen according to the TOF sensor measurement result, the method further includes: Selecting the TOF sensor and the camera; Selecting a plurality of relative postures of the TOF sensor and the camera, wherein the relative postures satisfy that the field of view of the camera covers the projection range of the projector, and the field of view range of the TOF sensor overlaps the field of view range of the camera; The relative posture parameters of the TOF sensor and the camera are calibrated under each relative posture, and the target relative posture is determined according to the relative posture parameters; wherein the relative posture parameters include the external parameters, the camera internal parameters and the distortion parameters.
3. The projector obstacle avoidance method based on sensors and cameras according to claim 1, It is characterized in that The detecting obstacle information in the camera image includes: Converting the camera image into a camera grayscale image; Performing image segmentation on the camera grayscale image according to the image characteristics of the obstacle to obtain a segmented image; wherein the segmented image is a binary image; Obstacle detection is performed on the segmented image to obtain the obstacle information.
4. The projector obstacle avoidance method based on sensors and cameras according to claim 2, It is characterized in that The calculating the homography matrix between the camera image and the projector image comprises: Acquire a projection display area, and extract a first vertex coordinate set corresponding to the projection display area in the camera image; Extracting a second vertex coordinate set corresponding to the projection display area in the projector image; The homography matrix is calculated based on the first set of vertex coordinates and the second set of vertex coordinates.
5. The projector obstacle avoidance method based on sensors and cameras according to claim 4, It is characterized in that The constructing of the virtual image comprises: According to the camera intrinsic parameters, the distortion parameters, the second plane equation and the second plane normal direction, projecting the vertices corresponding to the first vertex coordinate set onto the screen to obtain a projection point set; Connect all projection points in the projection point set to obtain a first projection area; Taking the area corresponding to the circumscribed rectangle of the first projection area as the second projection area; The second projection area is discretized to obtain the virtual image.
6. The projector obstacle avoidance method based on sensors and cameras according to claim 2, It is characterized in that The step of back-projecting the obstacle information onto the virtual image to obtain virtual coordinates of the obstacle includes: Back-projecting the obstacle information onto the virtual image according to the camera intrinsic parameters, the distortion parameters, the second plane equation, and the second plane normal direction to obtain an obstacle back-projection area; The coordinates of the obstacle back-projection area are discretized and rounded to obtain the virtual coordinates of the obstacle.
7. The projector obstacle avoidance method based on sensors and cameras according to claim 2, It is characterized in that The step of projecting the barrier-free display area onto a camera image plane to obtain a camera projection area includes: According to the camera intrinsic parameters, the distortion parameters, the second plane equation and the second plane normal direction, the barrier-free display area is projected onto the camera image plane to obtain the camera projection area.
8. The projector obstacle avoidance method based on sensors and cameras according to claim 1, It is characterized in that The step of transforming the image to be displayed into the obstacle avoidance area to obtain an obstacle avoidance projection image includes: The image to be displayed is transformed into the obstacle avoidance area by image affine transformation or interpolation method to obtain the obstacle avoidance projection image.
9. A projector obstacle avoidance device based on sensors and cameras, It is characterized in that include: A plane area fitting module, used to obtain a measurement result of a TOF sensor, and to fit a plane area of the screen according to the measurement result of the TOF sensor to obtain a first plane equation and a first plane normal direction; An obstacle information detection module, used to obtain a camera image and detect obstacle information in the camera image; the camera image is obtained by photographing the projector image on the plane area with a camera; A parameter transformation module, used for transforming the first plane equation and the first plane normal direction into a camera coordinate system according to the external parameters between the TOF sensor and the camera, to obtain a second plane equation and a second plane normal direction; A homography matrix transformation module, used for calculating the homography matrix between the camera image and the projector image; A virtual image binarization module is used to construct a virtual image, and back-project the obstacle information onto the virtual image to obtain virtual coordinates of the obstacle; Binarizing the virtual image according to the virtual coordinates of the obstacle to obtain a binary image; A maximum connected domain extraction module, used to extract the maximum connected domain from the binary image to obtain an obstacle-free display area; An obstacle-free display area projection module, used to project the obstacle-free display area onto a camera image plane to obtain a camera projection area; An image transformation module, used to transform the camera projection area to the projector image plane using the homography matrix to obtain an obstacle avoidance area; The image to be displayed is transformed into the obstacle avoidance area to obtain an obstacle avoidance projection image.
10. A computer device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the processor executes the computer program, the steps of the sensor-and-camera-based projector obstacle avoidance method according to any one of claims 1 to 8 are implemented.
11. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the sensor-and-camera-based projector obstacle avoidance method according to any one of claims 1 to 8 are implemented.
Citation Information
Patent Citations
Projection equipment and projection obstacle avoidance method
CN115604445A
Method and apparatus for determining display area of projection picture, and projector
WO2022205813A1
Target coordinate determination method and apparatus
WO2022205814A1
