Omnidirectional correction-based projector automatic obstacle avoidance method, device, equipment and medium

Abstract

The application belongs to the technical field of projectors, and discloses a projector automatic obstacle avoidance method and device based on omnidirectional correction, equipment and medium, wherein the method comprises the following steps: selecting multiple relative installation positions of a projector and a camera; calibrating the projector and the camera according to each relative installation position, and determining a target relative installation position of the projector and the camera according to a calibration parameter; triangulating image features of a camera image to obtain target calibration parameters and a three-dimensional coordinate set; fitting a screen plane according to the three-dimensional coordinate set to obtain a screen plane, a vertical rotation angle and a horizontal rotation angle; inversely projecting obstacle information onto the screen plane, constructing a virtual image according to obstacle coordinates, inversely projecting the obstacle information onto the virtual image to obtain an obstacle-free display area; and the above method, in combination with the rotation of the projector in a three-dimensional space, can realize automatic obstacle avoidance of the projector and simultaneously perform omnidirectional correction on the projector.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of projectors, for example to a projector automatic obstacle avoidance method and device based on omnidirectional correction, equipment and medium. BACKGROUND

[0002] When using a projector to display, the use environment and use mode have a significant impact on the display effect. For example, if the optical axis of the projector does not point directly at the screen, according to the perspective transformation, the display content on the screen will appear as trapezoidal distortion. When there is obvious dirt on the screen display area, there is an obstacle between the projection and the screen, or there is a poster or an electrical switch on the wall as the projection target, it will cause incomplete display or display quality degradation.

[0003] The prior art such as CN106817550B discloses a projection method and device of a projector. The prior art detects whether there is an obstacle between two projectors and the screen. When there is an obstacle between the first projector and the screen, and there is no obstacle between the second projector and the screen, the second projector synchronously projects the current projection content of the first projector. The prior art does not disclose how to achieve obstacle avoidance display on one projector.

[0004] The prior art such as CN105791784A discloses using a projector to obtain the obstacle between the projector and the screen and the shadow image on the screen, sending the shadow image to an image processing module, the image processing module calculating the area of the obstacle and the screen shadow. The distance between the obstacle and the projector is calculated, the distance between the obstacle and the screen is calculated, the pixel position that is blocked is obtained, the light output of the blocked pixel is controlled, and the corrected picture is obtained. The prior art can detect the obstacle and prevent the obstacle from being directly irradiated by the light emitted by the projector, but cannot achieve obstacle avoidance display of the projector.

[0005] In summary, the prior art has the problem of being unable to achieve obstacle avoidance display of the projector. SUMMARY

[0006] The present application aims to provide a projector automatic obstacle avoidance method, device, equipment and medium based on omnidirectional correction, which can solve the problem that the prior art cannot achieve obstacle avoidance display of the projector and / or cannot directly extract an obstacle-free area on a screen or a wall. To achieve the above-mentioned purpose, the present application provides a projector automatic obstacle avoidance method based on omnidirectional correction, comprising:

[0007] Selecting a plurality of relative installation positions of the projector and the camera; each of the relative installation positions satisfies that the field of view of the camera covers the field of view of the projector;

[0008] According to each of the relative installation positions, calibrate the projector and the camera to obtain calibration parameters; and according to the calibration parameters, determine a target relative installation position of the projector and the camera;

[0009] Obtain a normal rotation angle of the projector and a camera image, triangulate image features of the camera image to obtain target calibration parameters and a three-dimensional coordinate set;

[0010] According to the three-dimensional coordinate set, perform plane fitting on the screen to obtain a screen plane, a vertical rotation angle and a horizontal rotation angle;

[0011] Extract obstacle information in the camera image, and back-project the obstacle information onto the screen plane to obtain an obstacle coordinate set;

[0012] According to the obstacle coordinate, construct a virtual image, and back-project the obstacle information onto the virtual image to obtain an obstacle-free display area;

[0013] According to the normal rotation angle, the vertical rotation angle, the horizontal rotation angle and the target calibration parameters, project the obstacle-free display area onto a camera image plane to obtain a camera projection area;

[0014] Project the camera projection area onto a projector image plane to obtain a target obstacle avoidance area.

[0015] Preferably, the calibration of the projector and the camera according to each of the relative installation positions to obtain calibration parameters comprises:

[0016] Fix a focusing distance of the projector;

[0017] According to the focusing distance, calibrate the calibration parameters at each of the relative installation positions, the calibration parameters including camera intrinsic parameters, projector intrinsic parameters, distortion parameters and extrinsic parameters between the projector and the camera.

[0018] Preferably, the triangulation of the image features of the camera image to obtain target calibration parameters and a three-dimensional coordinate set comprises:

[0019] Extract the image features of the camera image, the camera image being obtained by the camera shooting a projector image;

[0020] According to the target calibration parameters, construct a bundle adjustment error function;

[0021] Minimize the bundle adjustment error function to obtain the target calibration parameters and the three-dimensional coordinate set.

[0022] Preferably, the extraction of the obstacle information in the camera image comprises:

[0023] convert the camera image into a gray image;

[0024] perform image segmentation on the gray image to obtain a binary image;

[0025] perform obstacle detection on the binary image to obtain the obstacle information.

[0026] Preferably, the back-projection of the obstacle information onto the screen plane to obtain a set of obstacle coordinates comprises:

[0027] calculating a homography matrix between the camera image plane and the projector image plane;

[0028] calculating a second set of coordinates in the camera image corresponding to the first set of coordinates in the projector image according to the homography matrix;

[0029] back-projecting the second set of coordinates onto the screen plane to obtain the set of obstacle coordinates.

[0030] Preferably, the construction of a virtual image according to the set of obstacle coordinates comprises:

[0031] connecting the obstacle coordinates in the set of obstacle coordinates to obtain a first projection region;

[0032] taking the circumscribed rectangle of the first projection region as a second projection region;

[0033] discretizing the second projection region to obtain the virtual image.

[0034] Preferably, the back-projection of the obstacle information onto the virtual image to obtain an obstacle-free display region comprises:

[0035] back-projecting the obstacle information onto the virtual image according to the normal rotation angle, the vertical rotation angle, the horizontal rotation angle, and the target calibration parameter to obtain a set of virtual image coordinates;

[0036] setting the pixel value of the coordinate point in the set of virtual image coordinates to a first preset value to obtain the obstacle-free display region.

[0037] The present application provides a projector automatic obstacle avoidance device based on omnidirectional correction, comprising:

[0038] a relative installation position selection module for selecting a plurality of relative installation positions of a projector and a camera; each of the relative installation positions satisfies that the field of view of the camera covers the field of view of the projector;

[0039] a target relative installation position determination module configured to calibrate the projector and the camera according to each of the relative installation positions to obtain calibration parameters, and determine a target relative installation position of the projector and the camera according to the calibration parameters;

[0040] an image feature triangulation module configured to obtain a normal rotation angle of the projector and a camera image, triangulate image features of the camera image to obtain target calibration parameters and a three-dimensional coordinate set;

[0041] a plane fitting module configured to perform plane fitting on a screen according to the three-dimensional coordinate set to obtain a screen plane, a vertical rotation angle and a horizontal rotation angle;

[0042] a first back projection module configured to extract obstacle information in the camera image, and back project the obstacle information onto the screen plane to obtain an obstacle coordinate set;

[0043] a second back projection module configured to construct a virtual image according to the obstacle coordinate set, and back project the obstacle information onto the virtual image to obtain an obstacle-free display area;

[0044] a first projection module configured to project the obstacle-free display area onto a camera image plane according to the normal rotation angle, the vertical rotation angle, the horizontal rotation angle and the target calibration parameters to obtain a camera projection area;

[0045] a second projection module configured to project the camera projection area onto a projector image plane to obtain a target obstacle avoidance area.

[0046] The application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the method for automatically avoiding obstacles of a projector based on omnidirectional correction according to any one of the preceding embodiments and / or the steps of the method for automatically avoiding obstacles of a projector based on omnidirectional correction according to any one of the preceding embodiments.

[0047] The application further provides a computer readable storage medium storing a computer program, wherein the computer program is executed by a processor to implement the method for automatically avoiding obstacles of a projector based on omnidirectional correction according to any one of the preceding embodiments and / or the steps of the method for automatically avoiding obstacles of a projector based on omnidirectional correction according to any one of the preceding embodiments.

[0048] The application discloses an omnidirectional correction-based automatic obstacle avoidance method for a projector, which comprises the following steps: selecting multiple relative installation positions of the projector and a camera; each relative installation position satisfies that a field of view of the camera covers a field of view of the projector. The projector can be an LCD projector or a DMD projector, and the application range is relatively wide. According to each relative installation position, the projector and the camera are calibrated to obtain calibration parameters; and the target relative installation position of the projector and the camera is determined according to the calibration parameters. The normal rotation angle of the projector and a camera image are obtained, the image features of the camera image are triangulated to obtain target calibration parameters and a three-dimensional coordinate set. According to the three-dimensional coordinate set, a screen plane, a vertical rotation angle and a horizontal rotation angle are obtained by fitting the screen. The obstacle information of an obstacle in the camera image is extracted, the obstacle information is inversely projected onto the screen plane to obtain an obstacle coordinate set. According to the obstacle coordinates, a virtual image is constructed, the obstacle information is inversely projected onto the virtual image to obtain an obstacle-free display area. According to the normal rotation angle, the vertical rotation angle, the horizontal rotation angle and the target calibration parameters, the obstacle-free display area is projected onto the camera image plane to obtain a camera projection area. The camera projection area is projected onto the projector image plane to obtain a target obstacle avoidance area. The method can describe the rotation of the projector in the three-dimensional space by introducing the normal rotation angle, the vertical rotation angle and the horizontal rotation angle of the projector, and can realize the automatic obstacle avoidance of the projector and the omnidirectional correction of the projector at the same time. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 A schematic diagram of the relative position of the projector and the camera in an embodiment;

[0050] Figure 2 A schematic diagram of the projector image in an embodiment;

[0051] Figure 3 A schematic diagram of the relative position and posture of the projector and the screen in an embodiment;

[0052] Figure 4 A schematic diagram of the flow of the omnidirectional correction-based automatic obstacle avoidance method for the projector in an embodiment;

[0053] Figure 5 A schematic diagram of the flow of the extraction of the obstacle information in the camera image in an embodiment;

[0054] Figure 6 A schematic diagram of the flow of the inverse projection of the obstacle information onto the screen plane in an embodiment;

[0055] Figure 7 A schematic diagram of the flow of the construction of the virtual image in an embodiment;

[0056] Figure 8A structure schematic block diagram of an embodiment of the omnidirectional correction-based projector automatic obstacle avoidance device;

[0057] Figure 9 A structure schematic block diagram of an embodiment of the computer device.

[0058] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of the present application more clear, the following further describes the present application in combination with the embodiments and the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0060] Those skilled in the art can understand that, unless specifically stated, the singular forms "a", "an" and "the" used herein also include the plural forms. It should be further understood that the use of the term "include" in the specification of the present application means that the features, integers, steps, operations, elements, modules and / or components exist, but do not exclude the existence or addition of one or more other features, integers, steps, operations, elements, modules, components and / or their combinations. It should be understood that when we say that an element is "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or there can be an intermediate element. In addition, the "connection" or "coupling" used herein can include wireless connection or wireless coupling. The phrase "and / or" used herein includes all or any module and all combinations of the associated listed items.

[0061] Those skilled in the art can understand that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as that generally understood by those skilled in the art to which the present application belongs. It should also be understood that terms such as those defined in a general dictionary should be understood to have meanings consistent with those in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as such.

[0062] In one embodiment, referring to Figure 4 is a flowchart of an omnidirectional correction-based projector automatic obstacle avoidance method disclosed by the present application, the method comprising:

[0063] S1: selecting a plurality of relative mounting positions of the projector and the camera; each of the relative mounting positions satisfies that the field of view of the camera covers the field of view of the projector.

[0064] In one embodiment, referring to Figure 1The camera can be on the left of the projector or on the right of the projector, which is not limited herein. The relative installation positions include different baseline distances and pose combinations of the projector and the camera, and each relative installation position satisfies that the field of view of the camera covers the field of view of the projector.

[0065] It should be noted that the lens of the projector has a refocusing function, including a manual refocusing function or an automatic focusing function.

[0066] Before step S1, the projector and the camera satisfying the technical specification requirements are selected.

[0067] S2: calibrate the projector and the camera according to each of the relative installation positions to obtain calibration parameters; and determine a target relative installation position of the projector and the camera according to the calibration parameters.

[0068] Fix the focusing distance of the projector;

[0069] Calibrate the calibration parameters according to the focusing distance under each of the relative installation positions, the calibration parameters including camera intrinsic parameters, projector intrinsic parameters, distortion parameters and extrinsic parameters between the projector and the camera. The distortion parameters include first distortion parameters and second distortion parameters, and the extrinsic parameters include first extrinsic parameters and second extrinsic parameters. The intrinsic parameters are related to the optical structure and focal length of the camera lens, and the extrinsic parameters are related to the relative position, relative rotation and translation between the projector and the camera.

[0070] S3: obtain a normal rotation angle of the projector and a camera image, triangulate image features of the camera image to obtain target calibration parameters and a three-dimensional coordinate set.

[0071] Extract the image features of the camera image, which is obtained by the camera shooting a projector image;

[0072] Construct a bundle adjustment error function according to the target calibration parameters;

[0073] Minimize the bundle adjustment error function to obtain the target calibration parameters and the three-dimensional coordinate set.

[0074] The target calibration parameters include target camera intrinsic parameters, target projector intrinsic parameters, target distortion parameters and target extrinsic parameters between the projector and the camera.

[0075] Since the projector is often triggered to automatically focus when it is working online, the parameters of the projector will change. When the projector is triggered to intelligently avoid obstacles, the projector will project a projector image, and the camera module will image the projector image to obtain a camera image, wherein the projector image is a specific pattern image. It should be noted that the projector image is pre-set, and only needs to ensure that reliable image features can be extracted from the camera image after the camera images the projector image. There are many forms of projector images that meet the above requirements. For example, Figure 2 , Figure 2 is an example of a projector image.

[0076] By triangulating the image features, the target calibration parameters can be obtained, which are the optimal combination of internal and external parameters.

[0077] S4: Plane fitting is performed on the screen according to the three-dimensional coordinate set to obtain a screen plane, a vertical rotation angle and a horizontal rotation angle.

[0078] Based on the three-dimensional coordinates in the three-dimensional coordinate set, plane fitting is performed on the screen to obtain a parameter equation AX+BY+CZ+D=0 of the screen relative to the coordinate system of the projector or the camera, and then the plane normal direction The screen plane can be determined from the parameter equation and the plane normal direction. The first distance between the screen plane and the projector or the camera is calculated, and the plane normal direction The vertical rotation angle α and the horizontal rotation angle β are calculated. Specifically, the plane normal direction and the first distance can represent the relative pose of the projector and the screen, which includes the rotation amount and the translation amount. The rotation can be represented using Euler angles, and the vertical rotation angle α and the horizontal rotation angle β can be calculated. Figure 3 is a schematic diagram of the relative pose of the projector and the screen.

[0079] The screen can be a wall or a curtain, which is not limited here.

[0080] S5: Extracting obstacle information in the camera image and inversely projecting the obstacle information onto the screen plane to obtain a set of obstacle coordinates.

[0081] The extraction of the obstacle information in the camera image includes:

[0082] Converting the camera image into a grayscale image;

[0083] Image segmentation is performed on the grayscale image to obtain a binary image;

[0084] Obstacle detection is performed on the binary image to obtain the obstacle information.

[0085] detecting an obstacle class, an obstacle range and an obstacle position in a binary image using an image detection method to obtain obstacle information, the obstacle information including coordinates and a range of the obstacle in the binary image, the image detection method including image segmentation and target recognition.

[0086] back-projecting the obstacle information onto the screen plane to obtain an obstacle coordinate set, including:

[0087] calculating a homography matrix between the camera image plane and the projector image plane;

[0088] calculating a second coordinate set in the camera image corresponding to the first coordinate set in the projector image according to the homography matrix;

[0089] back-projecting the second coordinate set onto the screen plane to obtain the obstacle coordinate set.

[0090] The homography matrix can reflect a conversion relationship between the camera image and the projector image.

[0091] S6: constructing a virtual image according to the obstacle coordinates, and back-projecting the obstacle information onto the virtual image to obtain an obstacle-free display area.

[0092] connecting the obstacle coordinates in the obstacle coordinate set to obtain a first projection area;

[0093] taking a circumscribed rectangle of the first projection area as a second projection area;

[0094] discretizing the second projection area to obtain the virtual image.

[0095] back-projecting the obstacle information onto the virtual image according to the normal rotation angle, the vertical rotation angle, the horizontal rotation angle and the target calibration parameter to obtain a virtual image coordinate set;

[0096] setting a pixel value of a coordinate point in the virtual image coordinate set to a first preset value to obtain the obstacle-free display area.

[0097] The pixel value of a pixel point in the virtual image is 1 or 0, and after the pixel value of the coordinate point in the virtual image coordinate set is set to the first preset value, the obstacle-free display area can be obtained by using a search method or a computational geometry method in image processing, and the obstacle-free display area is a final display area viewed on the screen.

[0098] S7: Project the barrier-free display region to the camera image plane according to the normal rotation angle, the vertical rotation angle, the horizontal rotation angle and the target calibration parameter, to obtain a camera projection region.

[0099] Project the boundary and coordinates of the barrier-free display region in the virtual image to the camera image plane according to the camera target intrinsic parameter, the target distortion parameter, the normal rotation angle, the vertical rotation angle, the horizontal rotation angle, the first distance and the discretization step.

[0100] S8: Project the camera projection region to the projector image plane to obtain a target obstacle avoidance region.

[0101] The target obstacle avoidance region obtained by projecting the camera projection region to the projector image plane using the homography matrix meets the requirements of automatic obstacle avoidance and omnidirectional correction at the same time.

[0102] After obtaining the target obstacle avoidance region in step S8, only the image affine transformation or image interpolation method is needed to transform the image to be displayed into the target obstacle avoidance region, so as to realize the functions of projector automatic obstacle avoidance and projector omnidirectional correction at the same time.

[0103] The omnidirectional correction-based projector automatic obstacle avoidance method provided by the embodiment of the application comprises the following steps: selecting a plurality of relative installation positions of a projector and a camera; each relative installation position meets the requirement that the field of view of the camera covers the field of view of the projector. The projector can be an LCD projector or a DMD projector, and the application range is relatively wide. The projector and the camera are calibrated according to each relative installation position, to obtain calibration parameters; and a target relative installation position of the projector and the camera is determined according to the calibration parameters. The normal rotation angle of the projector and a camera image are obtained, the image features of the camera image are triangulated, to obtain target calibration parameters and a three-dimensional coordinate set. The screen is planar fitted according to the three-dimensional coordinate set, to obtain a screen plane, a vertical rotation angle and a horizontal rotation angle. Obstacle information of an obstacle in the camera image is extracted, and the obstacle information is inversely projected onto the screen plane, to obtain an obstacle coordinate set. A virtual image is constructed according to the obstacle coordinates, and the obstacle information is inversely projected onto the virtual image, to obtain a barrier-free display region. The barrier-free display region is projected to the camera image plane according to the normal rotation angle, the vertical rotation angle, the horizontal rotation angle and the target calibration parameter, to obtain a camera projection region. The camera projection region is projected to the projector image plane, to obtain a target obstacle avoidance region. The method can describe the rotation of the projector in the three-dimensional space by introducing the normal rotation angle, the vertical rotation angle and the horizontal rotation angle of the projector, and can realize the omnidirectional correction of the projector while realizing the automatic obstacle avoidance of the projector.

[0104] In one embodiment, the calibration of the projector and the camera according to each of the relative installation positions to obtain the calibration parameters comprises:

[0105] S21: fixing the focusing distance of the projector.

[0106] The focusing distance of the projector is fixed according to the relative installation position of the projector and the camera. The projector lens usually has a refocusing function, which includes a manual refocusing function and an automatic focusing function.

[0107] The focusing distance corresponding to the image with the highest resolution on the screen is selected by testing the resolution of the image projected by the projector on the screen at different focusing distances.

[0108] S22: calibrating the calibration parameters according to the focusing distance at each of the relative installation positions, the calibration parameters including the camera intrinsic parameters, the projector intrinsic parameters, the distortion parameters, and the extrinsic parameters between the projector and the camera.

[0109] The embodiments of the present application regard the projector as the inverse process of the camera imaging, calibrate the camera intrinsic parameters, the projector intrinsic parameters, the first distortion parameters, the second distortion parameters, the first extrinsic parameters and the second extrinsic parameters based on the pinhole camera model, respectively.

[0110] After the calibration of the calibration parameters according to the focusing distance at each of the relative installation positions, the target relative installation position between the projector and the camera is selected according to the calibration accuracy and size requirements of all or part of the calibration parameters.

[0111] As described above, the calibration of the projector and the camera according to each of the relative installation positions to obtain the calibration parameters comprises: fixing the focusing distance of the projector, calibrating the calibration parameters according to the focusing distance at each of the relative installation positions, the calibration parameters including the camera intrinsic parameters, the projector intrinsic parameters, the distortion parameters, and the extrinsic parameters between the projector and the camera. The target relative installation position between the projector and the camera is selected according to the calibration accuracy and size requirements of all or part of the calibration parameters.

[0112] In one embodiment, referring to Figure 4 , the triangulation of the image features of the camera image to obtain the target calibration parameters and the three-dimensional coordinate set comprises:

[0113] S32: extracting the image features of the camera image, the camera image being obtained by the camera shooting the projector image.

[0114] After the projector projects the projector image onto the screen, the camera captures the projector image to obtain a camera image. An image feature extraction method based on a neural network or a traditional image feature extraction method can be used to extract the image features of the camera image. The image feature extraction method based on a neural network includes a method based on a VGG network or a Resnet network, and the traditional image feature extraction method includes an LBP (Local Binary Patterns) method, an HOG (Histogram of Oriented Gradient) method, and an image feature extraction method based on a SIFT (Scale Invariant Feature Transform) operator.

[0115] Before step S32, step S31 of obtaining the normal rotation angle of the projector and the camera image is further included.

[0116] When the projector is triggered to run, the normal rotation angle θ of the projector around the normal of the normal is obtained by an IMU sensor or the like. When the projector is triggered to automatically avoid obstacles, the projector projects a projector image onto the screen, and the camera images the image on the screen to obtain a camera image.

[0117] S33: Constructing a bundle adjustment error function according to the target calibration parameters.

[0118] Taking the calibration parameters obtained in step S22 as initial values, a bundle adjustment error function L(K c , K p , D p , D c , R, T) is constructed, where K c is a camera intrinsic parameter, K p is a projector intrinsic parameter, D p is a first distortion parameter, D c is a second distortion parameter, R is a first extrinsic parameter, and T is a second extrinsic parameter.

[0119] S34: Minimizing the bundle adjustment error function to obtain the target calibration parameters and the set of three-dimensional coordinates.

[0120] The bundle adjustment error function is minimized by a least square method or other methods to obtain the target calibration parameters and the set of three-dimensional coordinates corresponding to the image features of the camera image, and the target calibration parameters include a camera target intrinsic parameter K a projector target intrinsic parameter K a first target distortion parameter D a second target distortion parameter D a first target extrinsic parameter R​b and the second target extrinsic parameter T b .

[0121] The embodiment of the present application again regards the projector projection as the inverse process of camera imaging, based on the binocular ranging principle and bundle adjustment method, and uses the pinhole imaging model to triangulate the extracted image features.

[0122] As described above, the image features of the camera image are triangulated to obtain the target calibration parameters and the three-dimensional coordinate set, including extracting the image features of the camera image, which is obtained by the camera pair shooting the projector image. The bundle adjustment error function is constructed according to the target calibration parameters. The bundle adjustment error function is minimized to obtain the target calibration parameters and the three-dimensional coordinate set. Based on the binocular ranging principle and bundle adjustment method, the pinhole imaging model is used to triangulate the extracted image features.

[0123] In one embodiment, with reference to Figure 5 , the extraction of the obstacle information in the camera image includes:

[0124] S511: converting the camera image into a gray-scale image.

[0125] Before step S511, it further includes updating the camera image. The projector projects a pure white image with the same size as the projector image onto the screen, and the camera images the pure white image to update the camera image. The camera image is a color image in RGB format or YUV format.

[0126] The embodiment of the present application takes the camera image as a color image in RGB format as an example, calculates the pixel values of the R channel, the G channel and the B channel of each pixel point of the camera image, and takes the average value of the pixel values of the R channel, the G channel and the B channel as the gray value of the pixel point. The gray values of each pixel point in the camera image are calculated in sequence according to the order of rows or columns to obtain a gray-scale image.

[0127] S512: performing image segmentation on the gray-scale image to obtain a binary image.

[0128] According to the characteristics of the obstacle in the gray-scale image, an image segmentation method is used to perform image segmentation on the gray-scale image, and the image segmentation method includes a threshold-based segmentation method, a region-based segmentation method and an edge detection-based segmentation method.

[0129] The pixel value of the region corresponding to the obstacle in the binary image is 0, and the pixel value of other regions is 1.

[0130] S513: performing obstacle detection on the binary image to obtain the obstacle information.

[0131] The obstacle information is obtained by using a method based on connected domain analysis to detect obstacles in the binary image. The obstacle information includes coordinates and a range of the obstacles in the binary image.

[0132] As described above, the obstacle information in the camera image is extracted by converting the camera image into a gray-scale image, and performing image segmentation on the gray-scale image to obtain a binary image. The obstacle information is obtained by detecting obstacles in the binary image. The obstacle information includes coordinates and a range of the obstacles in the binary image.

[0133] In one embodiment, referring to Figure 6 , the back projection of the obstacle information onto the screen plane to obtain the obstacle coordinate set includes:

[0134] S52: calculating a homography matrix between the camera image plane and the projector image plane.

[0135] According to image features of the camera image, the homography matrix between the camera image plane and the projector image plane is calculated.

[0136] Before step S52, there is also step S51: extracting obstacle information in the camera image.

[0137] S53: calculating a second coordinate set in the camera image corresponding to the first coordinate set in the projector image according to the homography matrix.

[0138] The coordinates C p1 , C p2 , C p3 and C p4 of the four vertices (0, 0), (0, height), (width, 0) and (width, height) of the pure white image projected by the projector in the updated camera image are calculated using the homography matrix. p1 , C p2 , C p3 and C p4 , wherein width represents the width of the pure white image, and height represents the height of the pure white image.

[0139] It should be noted that when the projector and the camera satisfy the relative position relationship as shown in Figure 1 , C p1 , C p2 , C p3 and C p4 are completely contained in the updated camera image.

[0140] S54: back projecting the second coordinate set onto the screen plane to obtain the obstacle coordinate set.

[0141] The camera target intrinsic parameters The second target distortion parameters The normal rotation angle θ, the vertical rotation angle α, the horizontal rotation angle β, and the first distance d are used to project C p1 , C p2 , C p3 , and C p4 Four points are sequentially inversely projected onto the screen plane to obtain four projection points C1, C2, C3, and C4, and C1, C2, C3, and C4 form an obstacle coordinate set.

[0142] As described above, inversely projecting the obstacle information onto the screen plane to obtain the obstacle coordinate set includes calculating a homography matrix between the camera image plane and the projector image plane, and calculating a second coordinate set in the camera image corresponding to the first coordinate set in the projector image plane according to the homography matrix. The second coordinate set is inversely projected onto the screen plane to obtain the obstacle coordinate set, and the obstacle coordinate set includes four vertices.

[0143] In one embodiment, with reference to Figure 7 , the virtual image is constructed according to the obstacle coordinates, including:

[0144] S61: connecting the obstacle coordinates in the obstacle coordinate set to obtain a first projection area.

[0145] The four obstacle coordinates C1, C2, C3, and C4 in the obstacle coordinate set obtained in step S54 are sequentially connected to obtain a first projection area, and the first projection area is a quadrilateral.

[0146] S62: taking a circumscribed rectangle of the first projection area as a second projection area.

[0147] In order to facilitate representation and subsequent processing, the circumscribed rectangle of the first projection area is taken as the second projection area.

[0148] S63: discretizing the second projection area to obtain the virtual image.

[0149] The second projection area is discretized according to a preset step length to obtain the virtual image. The virtual image is represented as a matrix, and the virtual image is an initial value of all 1s, and is an n×n single-channel image, where n can be 100 or other values, which are not limited here.

[0150] As described above, the virtual image is constructed according to the obstacle coordinates, including connecting the obstacle coordinates in the obstacle coordinate set to obtain a first projection area. The circumscribed rectangle of the first projection area is taken as a second projection area, and the second projection area is discretized to obtain the virtual image, and the virtual image is represented as a matrix.

[0151] In one embodiment, the back-projection of the obstacle information to the virtual image to obtain the obstacle-free display region comprises:

[0152] S64: back-project the obstacle information to the virtual image according to the normal rotation angle, the vertical rotation angle, the horizontal rotation angle, and the target calibration parameter to obtain a virtual image coordinate set.

[0153] According to the normal rotation angle θ, the vertical rotation angle α, the horizontal rotation angle β, the camera target internal parameter, the second distortion parameter, and the first distance, the obstacle geometric information in the obstacle information is back-projected to the virtual image to obtain a plurality of coordinate points, all the coordinate points are discretized and rounded to obtain a virtual image coordinate set.

[0154] S65: set the pixel value of the coordinate point in the virtual image coordinate set to a first preset value to obtain the obstacle-free display region.

[0155] The first preset value can be 0 or 1, which is not limited here. The embodiment of the present application takes 0 as an example.

[0156] The pixel point with a pixel value of 0 in the virtual image represents an obstacle, and all the pixel points with a pixel value of 1 form an obstacle-free display region. Therefore, after setting the pixel value of the coordinate point in the virtual image coordinate set to the first preset value, extracting the obstacle-free display region is converted into extracting a sub-region meeting specific proportion, size, shape, etc. requirements in the virtual image. The obstacle-free display region can be extracted by a search method or a computational geometry method, and the obstacle-free display region is the region finally viewed on the screen.

[0157] As described above, the back-projection of the obstacle information to the virtual image to obtain the obstacle-free display region comprises back-projecting the obstacle information to the virtual image according to the normal rotation angle, the vertical rotation angle, the horizontal rotation angle, and the target calibration parameter to obtain a virtual image coordinate set. The pixel value of the coordinate point in the virtual image coordinate set is set to a first preset value to obtain the obstacle-free display region. The obstacle-free display region can be extracted by a search method or a computational geometry method, and the obstacle-free display region is the region finally viewed on the screen.

[0158] Referring to Figure 8 is a structural schematic block diagram of a projector automatic obstacle avoidance device based on omnidirectional correction disclosed by the present application, the device comprises:

[0159] The relative installation position selection module 10 is used for selecting a plurality of relative installation positions of the projector and the camera; each of the relative installation positions satisfies that the field of view of the camera covers the field of view of the projector.

[0160] The target relative installation position determination module 20 is configured to calibrate the projector and the camera according to each of the relative installation positions to obtain calibration parameters; and determine a target relative installation position of the projector and the camera according to the calibration parameters.

[0161] The image feature triangulation module 30 is configured to obtain a normal rotation angle of the projector and a camera image, and triangulate image features of the camera image to obtain target calibration parameters and a three-dimensional coordinate set.

[0162] The plane fitting module 40 is configured to perform plane fitting on the screen according to the three-dimensional coordinate set to obtain a screen plane, a vertical rotation angle and a horizontal rotation angle.

[0163] The first back projection module 50 is configured to extract obstacle information in the camera image, and back project the obstacle information onto the screen plane to obtain an obstacle coordinate set.

[0164] The second back projection module 60 is configured to construct a virtual image according to the obstacle coordinate set, and back project the obstacle information onto the virtual image to obtain an obstacle-free display area.

[0165] The first projection module 70 is configured to project the obstacle-free display area onto a camera image plane according to the normal rotation angle, the vertical rotation angle, the horizontal rotation angle and the target calibration parameters to obtain a camera projection area.

[0166] The second projection module 80 is configured to project the camera projection area onto a projector image plane to obtain a target obstacle avoidance area.

[0167] As described above, the omnidirectional correction-based projector automatic obstacle avoidance device can implement an omnidirectional correction-based projector automatic obstacle avoidance method.

[0168] In one embodiment, the target relative installation position determination module 20 further comprises:

[0169] A focusing distance fixing unit is configured to fix a focusing distance of the projector.

[0170] A calibration unit is configured to calibrate the calibration parameters according to the focusing distance at each of the relative installation positions, wherein the calibration parameters include camera intrinsic parameters, projector intrinsic parameters, distortion parameters and extrinsic parameters between the projector and the camera.

[0171] In one embodiment, the image feature triangulation module 30 further comprises:

[0172] An image feature extraction unit is configured to extract the image features of the camera image, wherein the camera image is obtained by the camera shooting a projector image.

[0173] an error function construction unit configured to construct a bundle adjustment error function according to the target calibration parameter;

[0174] a minimization error function unit configured to minimize the bundle adjustment error function to obtain the target calibration parameter and the three-dimensional coordinate set.

[0175] In an embodiment, the first back projection module 50 further comprises:

[0176] an image conversion unit configured to convert the camera image into a gray-scale image;

[0177] an image segmentation unit configured to perform image segmentation on the gray-scale image to obtain a binary image;

[0178] an obstacle detection unit configured to perform obstacle detection on the binary image to obtain the obstacle information.

[0179] In an embodiment, the first back projection module 50 further comprises:

[0180] a homography matrix calculation unit configured to calculate a homography matrix between the camera image plane and the projector image plane;

[0181] a second coordinate set calculation unit configured to calculate a second coordinate set corresponding to the first coordinate set in the projector image according to the homography matrix;

[0182] a first back projection unit configured to back project the second coordinate set to the screen plane to obtain the obstacle coordinate set.

[0183] In an embodiment, the second back projection module 60 further comprises:

[0184] an obstacle coordinate connection unit configured to connect obstacle coordinates in the obstacle coordinate set to obtain a first projection region;

[0185] a second projection region construction unit configured to take a circumscribed rectangle of the first projection region as a second projection region;

[0186] a discretization unit configured to discretize the second projection region to obtain the virtual image.

[0187] In an embodiment, the second back projection module 60 further comprises:

[0188] a second back projection unit, configured to back project the obstacle information to the virtual image according to the normal rotation angle, the vertical rotation angle, the horizontal rotation angle and the target calibration parameter, to obtain a virtual image coordinate set;

[0189] a pixel value setting unit, configured to set a pixel value of a coordinate point in the virtual image coordinate set as a first preset value, to obtain the obstacle-free display area.

[0190] Reference Figure 9 The computer device in the embodiments of the present application can have an internal structure as shown in Figure 9 The computer device includes a processor, a memory, a network interface and a database connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating device, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is configured to store the obstacle-free display area and the like. The network interface of the computer device is configured to communicate with an external terminal through a network connection. Further, the computer device can be further provided with an input device, a display screen and the like. The computer program is executed by the processor to implement the omnidirectional correction-based automatic obstacle avoidance method for a projector, including the following steps: selecting a plurality of relative mounting positions of a projector and a camera; each of the relative mounting positions satisfies that a field of view of the camera covers a field of view of the projector; calibrating the projector and the camera according to each of the relative mounting positions to obtain calibration parameters; determining a target relative mounting position of the projector and the camera according to the calibration parameters; obtaining a normal rotation angle of the projector and a camera image, triangulating image features of the camera image to obtain target calibration parameters and a three-dimensional coordinate set; fitting a screen plane according to the three-dimensional coordinate set to obtain the screen plane, a vertical rotation angle and a horizontal rotation angle; extracting obstacle information in the camera image, back projecting the obstacle information onto the screen plane to obtain an obstacle coordinate set; constructing a virtual image according to the obstacle coordinate, back projecting the obstacle information to the virtual image to obtain an obstacle-free display area; and projecting the obstacle-free display area to a camera image plane according to the normal rotation angle, the vertical rotation angle, the horizontal rotation angle and the target calibration parameter to obtain a camera projection area; projecting the camera projection area to a projector image plane to obtain a target obstacle avoidance area. Those skilled in the art can understand that Figure 9 The structure shown in the embodiments of the present application is only a block diagram of part of the structure related to the present application, and does not constitute a limitation on the computer device to which the present application is applied.

[0191] An embodiment of the present application also provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement an omnidirectional correction-based automatic obstacle avoidance method for a projector, including the following steps: selecting a plurality of relative installation positions of a projector and a camera; each of the relative installation positions satisfies that a field of view of the camera covers a field of view of the projector; calibrating the projector and the camera according to each of the relative installation positions to obtain calibration parameters; determining a target relative installation position of the projector and the camera according to the calibration parameters; obtaining a normal rotation angle of the projector and a camera image, triangulating image features of the camera image to obtain target calibration parameters and a three-dimensional coordinate set; performing plane fitting on a screen according to the three-dimensional coordinate set to obtain a screen plane, a vertical rotation angle and a horizontal rotation angle; extracting obstacle information in the camera image, inversely projecting the obstacle information onto the screen plane to obtain an obstacle coordinate set; constructing a virtual image according to the obstacle coordinate set, inversely projecting the obstacle information onto the virtual image to obtain an obstacle-free display area; projecting the obstacle-free display area onto a camera image plane according to the normal rotation angle, the vertical rotation angle, the horizontal rotation angle and the target calibration parameters to obtain a camera projection area; and projecting the camera projection area onto a projector image plane to obtain a target obstacle avoidance area.

[0192] It can be understood that the computer readable storage medium in the embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0193] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, storage, databases, or other media in this application and in examples used herein can include non-volatile and / or volatile memory. Non-volatile memory can include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM), or external cache memory. As an illustration but not limitation, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0194] It should be noted that in this document, the terms "comprising", "including", or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements recited, but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitations, an element defined by the phrase "comprising a" does not exclude the existence of additional identical elements in the process, method, article, or apparatus including the element.

[0195] The above description is only the preferred embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation, or direct or indirect application in other related technical fields, based on the content of the present application specification and drawings, is also included in the patent protection scope of the present application.

Claims

1. An omnidirectional correction-based projector automatic obstacle avoidance method, characterized in that, The method comprises the following steps: selecting a plurality of relative mounting positions of a projector and a camera; each of the relative mounting positions satisfies that a field of view of the camera covers a field of view of the projector; calibrating the projector and the camera according to each of the relative mounting positions to obtain calibration parameters; determining a target relative mounting position of the projector and the camera according to the calibration parameters; obtaining a normal rotation angle of the projector and a camera image, triangulating image features of the camera image to obtain target calibration parameters and a three-dimensional coordinate set; performing plane fitting on a screen according to the three-dimensional coordinate set to obtain a screen plane, a vertical rotation angle and a horizontal rotation angle; extracting obstacle information in the camera image, and inversely projecting the obstacle information onto the screen plane to obtain an obstacle coordinate set; constructing a virtual image according to the obstacle coordinate set, and inversely projecting the obstacle information onto the virtual image to obtain an obstacle-free display area; projecting the obstacle-free display area onto a camera image plane according to the normal rotation angle, the vertical rotation angle, the horizontal rotation angle and the target calibration parameters to obtain a camera projection area; projecting the camera projection area onto a projector image plane to obtain a target obstacle avoidance area.

2. The omni-directional correction based projector automatic obstacle avoidance method according to claim 1, wherein, The calibration of the projector and the camera according to each of the relative mounting positions to obtain calibration parameters comprises the following steps: fixing a focusing distance of the projector; calibrating the calibration parameters according to the focusing distance at each of the relative mounting positions, wherein the calibration parameters include camera internal parameters, projector internal parameters, distortion parameters and external parameters between the projector and the camera.

3. The omni-directional correction based projector automatic obstacle avoidance method according to claim 1, wherein, The triangulation of the image features of the camera image to obtain target calibration parameters and a three-dimensional coordinate set comprises the following steps: extracting the image features of the camera image, which is obtained by the camera shooting a projector image; constructing a bundle adjustment error function according to the target calibration parameters; minimizing the bundle adjustment error function to obtain the target calibration parameters and the three-dimensional coordinate set.

4. The omni-directional correction based projector automatic obstacle avoidance method according to claim 1, wherein, The extraction of the obstacle information in the camera image comprises the following steps: converting the camera image into a gray image; performing image segmentation on the gray image to obtain a binary image; performing obstacle detection on the binary image to obtain the obstacle information.

5. The omni-directional correction based projector automatic obstacle avoidance method according to claim 3, wherein, The inverse projection of the obstacle information onto the screen plane to obtain an obstacle coordinate set comprises the following steps: calculating a homography matrix between the camera image plane and the projector image plane; calculating a second coordinate set corresponding to a first coordinate set in the projector image according to the homography matrix; inversely projecting the second coordinate set onto the screen plane to obtain the obstacle coordinate set.

6. The omni-directional correction based projector automatic obstacle avoidance method according to claim 1, wherein, The construction of a virtual image according to the obstacle coordinate set comprises the following steps: connecting obstacle coordinates in the obstacle coordinate set to obtain a first projection area; taking a circumscribed rectangle of the first projection area as a second projection area; discretizing the second projection area to obtain the virtual image.

7. The omni-directional correction based projector automatic obstacle avoidance method according to claim 1, wherein, The obstacle information is reversely projected to the virtual image to obtain an obstacle-free display area, including: According to the normal rotation angle, the vertical rotation angle, the horizontal rotation angle, and the target calibration parameter, the obstacle information is reversely projected to the virtual image to obtain a virtual image coordinate set; A pixel value of a coordinate point in the virtual image coordinate set is set to a first preset value to obtain the obstacle-free display area.

8. An omnidirectional correction-based projector automatic obstacle avoidance device, characterized in that, Comprise: A relative installation position selection module is configured to select a plurality of relative installation positions of a projector and a camera, each of the relative installation positions satisfying that a field of view of the camera covers a field of view of the projector; A target relative installation position determination module is configured to calibrate the projector and the camera according to each of the relative installation positions to obtain calibration parameters, and determine a target relative installation position of the projector and the camera according to the calibration parameters; An image feature triangulation module is configured to obtain a normal rotation angle of the projector and a camera image, and triangulate image features of the camera image to obtain target calibration parameters and a three-dimensional coordinate set; A plane fitting module is configured to perform plane fitting on a screen according to the three-dimensional coordinate set to obtain a screen plane, a vertical rotation angle, and a horizontal rotation angle; A first reverse projection module is configured to extract obstacle information in the camera image, reversely project the obstacle information to the screen plane to obtain an obstacle coordinate set; A second reverse projection module is configured to construct a virtual image according to the obstacle coordinate set, reversely project the obstacle information to the virtual image to obtain an obstacle-free display area; A first projection module is configured to project the obstacle-free display area to a camera image plane according to the normal rotation angle, the vertical rotation angle, the horizontal rotation angle, and the target calibration parameters to obtain a camera projection area; A second projection module is configured to project the camera projection area to a projector image plane to obtain a target obstacle avoidance area. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8. The processor executes the computer program to implement the steps of the omnidirectional correction-based projector automatic obstacle avoidance method of any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the omnidirectional correction-based projector automatic obstacle avoidance method of any one of claims 1 to 7.

Images