Projector automatic obstacle avoidance method and device combining adaptive picture cutting and deep learning

Abstract

The invention relates to a projector automatic obstacle avoidance method and device combining adaptive picture cutting and deep learning, and the method comprises the steps: enabling a projector to project a white background picture, capturing a projection picture through a camera, calculating a difference image based on a projection region background image and a projection picture image, carrying out the adaptive cutting of the difference image, and obtaining a projection region image, the difference image can highlight the difference between the projection area and the background area, the projection area image can be found out more conveniently, then a deep learning model is used for detecting an obstacle in the cut image and recording position information, the maximum projection area is calculated, the position information of an obstacle-free area is obtained and then converted to a projector coordinate system, and the position information of the obstacle-free area is obtained. And finally projecting a final picture. Therefore, the automatic obstacle avoidance of the projector is realized, the clear and complete projection picture is ensured, the problem of poor projection effect in a complex environment due to traditional manual processing and the prior art can be effectively solved, and the projection quality of the projector in a complex scene is improved.

Description

Technical Field

[0001] The present invention relates to the technical field related to projection imaging, and in particular to a method and device for automatic obstacle avoidance of a projector by combining adaptive cropping of images with deep learning. Background Art

[0002] With the continuous advancement of projection technology, projectors have been widely used in many fields such as education, entertainment, and business, creating diverse and rich visual enjoyment for people. However, in actual application scenarios, many factors have adversely affected the projection effect. For example, obstacles such as dirt on the wall, various hanging objects, and randomly placed furniture often cause the projection image to be blurred, missing content, and even cast shadows on the image, greatly weakening the clarity and visual beauty during viewing and display.

[0003] Traditionally, these projection obstacles are handled mainly manually. After observing the abnormality of the projected image with the naked eye, the operator manually adjusts the projector's angle, position and other parameters, or simply tidies up the surrounding environment. However, manual operation is limited by the accuracy of the human eye, and it is difficult to accurately judge subtle changes in the image and complex environmental factors. When faced with complex scene layouts and a combination of multiple obstacles, it is difficult to efficiently and accurately achieve the ideal projection effect, and it consumes a lot of manpower and time.

[0004] Although modern technology has developed, it is still lacking in the processing of complex environments. In particular, there are deficiencies in accurately identifying the projection background. It is difficult to accurately distinguish between curtain backgrounds of different materials, colors and textures. In an environment with complex light and shadow and large brightness differences, it is impossible to accurately analyze the relationship between the background and obstacles, making it difficult to intelligently adjust the projection parameters in a targeted manner. As a result, it is difficult to achieve high standards for projection effects in complex environments and cannot fully meet users' projection quality requirements in diverse scenarios. Summary of the invention

[0005] The purpose of the present invention is to solve at least one of the deficiencies of the prior art and to provide a method and device for automatic obstacle avoidance of a projector by combining adaptive cropping of images with deep learning.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions: Specifically, a projector automatic obstacle avoidance method combining adaptive cropping and deep learning is proposed, including the following: Obtain the background image of the projection area, and control the projector to transmit the white background image to obtain the projection screen image at this time; Calculating a differential image based on the projection area background image and the projection screen image, and adaptively cropping the differential image to obtain a projection area image; Input the image of the projection area into a pre-trained deep learning model to output the image of the projection area with the position information of the obstacles recorded; Based on the image of the projection area with the position information of the obstacles recorded, calculate the maximum projection area and then obtain the position information of the obstacle-free area; Convert the position information of the obstacle-free area to the projector coordinate system and project the actual projection screen accordingly.

[0007] Further, specifically, calculate the difference image based on the background image of the projection area and the projection screen image, and perform adaptive cropping on the difference image to obtain the image of the projection area, including, Subtract the projection screen from the background image of the projection area and take the absolute value to obtain the difference image; Perform grayscale processing on the difference image to obtain a grayscale image; Process the grayscale image through a pre-selected edge detection operator to obtain an edge image; Perform connected component analysis on the edge image to find multiple connected components and mark them; Take the connected component with the largest area among all connected components as the preliminary projection area; Crop the preliminary projection area from the grayscale image to obtain a preliminary image; Perform adaptive cropping on the preliminary image to obtain the projection area; Crop the projection area from the difference image and mark the position of the connected components contained therein. The projection area is recorded as the position of the largest connected component contained therein.

[0008] Further, specifically, perform adaptive cropping on the preliminary image to obtain the projection area, including, Divide the preliminary image into four regions: upper left, lower left, upper right, and lower right along the horizontal and vertical central axes; For any of the four regions, calculate its average grayscale value, and obtain the dynamic threshold of the region based on its average grayscale value combined with a preset scale factor. The dynamic threshold is the product of the average grayscale value and the scale factor; Taking the upper left region as an example, start traversing along the left and upper edges of the image with its corresponding dynamic threshold. If the grayscale value in the image of the region is greater than the dynamic threshold, it is considered that the boundary of the region is found and the region is updated accordingly; The other regions are the same as the upper left region. Then, find the updated regions of all four regions. The updated regions of the four regions are the projection area.

[0009] Further, specifically, perform connected component analysis on the edge image to find multiple connected components and mark them, including, Perform connected component analysis on the edge image through the connectedComponentsWithStats function of OpenCV to find multiple connected components and label them, create a stats array, and store the area information of each connected component in the stats array; Traverse the stats array, extract the area of each connected component, and calculate the average area of the remaining connected components after removing the connected component with the largest area among all connected components at this time to obtain the first value; Remove the connected components with an average area less than the first value to obtain updated connected components; Then calculate the average area of the other connected components after removing the connected component with the largest area in the updated connected components to obtain the second value; Preset a scale threshold, and subtract the scale threshold from the second value to obtain the third value; Remove the connected components with an area less than the third value in the updated connected components to obtain the final connected components, and use this as the result of the connected component analysis for marking.

[0010] Furthermore, specifically, input the projection area image into a pre-trained deep learning model, and output a projection area image recording the obstacle position information, including, Train the Faster R-CNN deep learning model based on the method of transfer learning; Convert the projection area image into Tensor format and input it into the trained Faster R-CNN deep learning model; Extract the confidence score of each detection box from the trained Faster R-CNN model; According to the preset confidence threshold, filter out the detection boxes with a confidence higher than the confidence threshold, and record these detection boxes as obstacles; Record the position information of the detection boxes recorded as obstacles.

[0011] Furthermore, specifically, train the Faster R-CNN deep learning model based on the method of transfer learning, including, Load the pre-trained model and modify the model. Specifically, load a Faster R-CNN pre-trained model that has been trained on a large dataset, modify the classifier at the end of the model, and adjust the output category of the last layer to 2; Prepare the dataset and data loader. Specifically, mark the rectangular box position and category of the obstacle on each picture, and customize a CustomDataset class, which inherits from Dataset and is used to load the training data; Train the model and save the model. Specifically, use the SGD (Stochastic Gradient Descent) optimizer to optimize the parameters of the Faster R-CNN model, and use the StepLR step learning rate scheduler to dynamically adjust the learning rate. Use the mAP (mean Average Precision) as the evaluation metric to evaluate the model, find the optimal model, and save the optimal model selected through evaluation.

[0012] Furthermore, specifically, based on the projection area image recording the obstacle position information, calculate the maximum projection area to obtain the obstacle-free area position information, including: Assume the size of the projection area image is W×H, and the center point is ( , ), where , ”. The initial rectangular boundary is = , = , = , = , and the aspect ratio of the rectangle satisfies . In the projection area image, starting from the center of the image, search in the four directions of up, down, left, and right. Traverse each column in the up and down directions respectively until an obstacle is encountered and update and ; Traverse each row in the left and right directions until an obstacle is encountered and update and . During the search process, if the boundary expansion changes the aspect ratio, adjust the longer boundary of the width or height to keep the rectangle ratio unchanged; finally, calculate the maximum rectangle area as , and use the maximum rectangle area as the obstacle-free area position information.

[0013] Furthermore, the method further includes: Judge whether the obstacle-free area position information is greater than the preset area threshold. If it is, do not process it; if not, give an alarm reminder to inform the relevant staff to conduct a verification and processing.

[0014] Furthermore, specifically, convert the obstacle-free area position information to the projector coordinate system for actual projection screen projection, including: Obtain the spatial relationship between the camera and the projector, and the spatial relationship includes the relative position and direction; According to the spatial relationship between the camera and the projector and the relative information of the obstacle-free area position information, convert the obtained obstacle-free area position information into two-dimensional coordinates in the projector coordinate system; Adjust the projection angle, focal length, and zoom ratio of the projector according to the position and size information of the obstacle-free area in the two-dimensional coordinates; Use the projector with adjusted parameters to project the actual projection screen.

[0015] Furthermore, the method further includes, Record the positions of the removed connected regions, and fill the content of the positions of the connected regions by filling with the background color to update the content of the corresponding positions when projecting the actual projection screen.

[0016] The present invention also proposes a device for automatic obstacle avoidance of a projector by combining adaptive cropping of the screen and deep learning, including the following: A data acquisition module, configured to acquire the background image of the projection area, and control the projector to transmit a white background image to acquire the projection screen image at this time; A projection area image cropping module, configured to calculate a difference image based on the background image of the projection area and the projection screen image, and perform adaptive cropping on the difference image to obtain the projection area image; An obstacle position determination module, configured to input the projection area image into a pre-trained deep learning model, and output a projection area image recording the obstacle position information; An obstacle-free area position determination module, configured to calculate the maximum projection area based on the projection area image recording the obstacle position information, and further obtain the obstacle-free area position information; A projection screen projection module, configured to convert the obstacle-free area position information into the projector coordinate system, and project the actual projection screen accordingly.

[0017] The beneficial effects of the present invention are as follows: The present invention proposes a method and device for automatic obstacle avoidance of a projector by combining adaptive cropping of the screen and deep learning. First, the projector projects a white background screen, then uses a camera to capture the projection screen, calculates a difference image based on the background image of the projection area and the projection screen image, performs adaptive cropping on the difference image to obtain the projection area image. The difference image can highlight the difference between the projection area and the background area, making it more convenient to find the projection area image. Then, use a deep learning model to detect the obstacles in the cropped image and record the position information, calculate the maximum projection area, obtain the obstacle-free area position information, convert it into the projector coordinate system, and finally project the final screen. In this way, automatic obstacle avoidance of the projector is realized, ensuring that the projection screen is clear and complete, effectively solving the problems of poor projection effects of traditional manual processing and existing technologies in complex environments, and improving the projection quality of the projector in complex scenarios; In addition, before inputting the cropped image into the deep learning model, the positions of the connected regions (i.e., suspected obstacles) are pre-marked through image processing, and the interference noise (eliminated according to the first value) and the connected regions that are not sufficient to affect the projection of the projector (eliminated according to the third value) are removed, which can improve the processing efficiency of the deep learning model and the accuracy of the processing results. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] By describing the embodiments shown in the accompanying drawings in detail, the above and other features of the present disclosure will become more apparent. The same reference numerals in the drawings of the present disclosure denote the same or similar elements. Obviously, the drawings in the following description are only some embodiments of the present disclosure. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings. In the drawings: Figure 1 The flowchart of the projector automatic obstacle avoidance method of the present invention combining adaptive cropped screen and deep learning is shown; Figure 2 In (a) of, it is the first complex environment image based on aperture interference in the embodiment of the present invention, and (b) is the second complex environment image based on aperture interference in the embodiment of the present invention; Figure 3 In (a) of, it is the image after adaptive cropping of the first complex environment image in the embodiment of the present invention, and (b) is the image after adaptive cropping of the second complex environment image in the embodiment of the present invention; Figure 4 In (a) of, it is the image of obstacle detection for the first complex environment image after adaptive cropping in the embodiment of the present invention, and (b) is the image of obstacle detection for the second complex environment image without adaptive cropping in the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0019] The following will clearly and completely describe the concept, specific structure and technical effects generated by the present invention in combination with the embodiments and the drawings, so as to fully understand the purpose, solution and effects of the present invention. It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The same reference numerals used throughout the drawings denote the same or similar parts.

[0020] Embodiment 1, referring to Figure 1 , the present invention proposes an automatic obstacle avoidance method for a projector combining adaptive cropped screen and deep learning, including the following: Obtain the background image of the projection area, and control the projector to transmit a white background image to obtain the projection screen image at this time; Calculate a difference image based on the background image of the projection area and the projection screen image, and perform adaptive cropping on the difference image to obtain a projection area image; Input the projection area image into a pre-trained deep learning model, and output a projection area image with obstacle position information recorded; Based on the projection area image with obstacle position information recorded, calculate the maximum projection area to obtain the position information of the obstacle-free area; Convert the position information of the obstacle-free area to the projector coordinate system, and project the actual projection screen accordingly.

[0021] In Embodiment 1, first, let the projector project a white background image, then use a camera to capture the projection screen. Calculate a difference image based on the background image of the projection area and the projection screen image, and perform adaptive cropping on the difference image to obtain a projection area image. The difference image can highlight the difference between the projection area and the background area, making it easier to find the projection area image. Then, use a deep learning model to detect obstacles in the cropped image and record the position information, calculate the maximum projection area, obtain the position information of the obstacle-free area, convert it to the projector coordinate system, and finally project the final image. In this way, the projector can automatically avoid obstacles, ensure the clarity and integrity of the projection screen, effectively solve the problems of traditional manual processing and poor projection effects of existing technologies in complex environments, and improve the projection quality of the projector in complex scenarios.

[0022] As a preferred embodiment of the present invention, specifically, calculating a difference image based on the background image of the projection area and the projection screen image, and performing adaptive cropping on the difference image to obtain a projection area image includes, Subtract the projection screen image from the background image of the projection area and take the absolute value to obtain a difference image; Perform grayscale processing on the difference image to obtain a grayscale image; Process the grayscale image through a pre-selected edge detection operator to obtain an edge image; Perform connected component analysis on the edge image to find multiple connected components and mark them; Take the connected component with the largest area among all connected components as the preliminary projection area; Crop the preliminary projection area from the grayscale image to obtain a preliminary image; Perform adaptive cropping on the preliminary image to obtain a projection area; Crop the projection area from the difference image, and mark the position of the connected component contained therein. The projection area is recorded as the position of the largest connected component contained therein.

[0023] In this preferred embodiment, considering the problem of aperture interference, the largest connected region cannot be used as the projection area image. Therefore, an area with less aperture interference is found from the largest connected domain, i.e., the preliminary projection area, through adaptive cropping to obtain a projection area with less aperture interference.

[0024] As a preferred embodiment of the present invention, specifically, the projection area is obtained by performing adaptive cropping on the preliminary image, including: The preliminary image is divided into four regions: upper left, lower left, upper right, and lower right along two central axes, horizontal and vertical; For any of the four regions, calculate its average gray value, and obtain the dynamic threshold of the region based on its average gray value in combination with a preset scale factor. The dynamic threshold is the product of the average gray value and the scale factor; Taking the upper left region as an example, start traversing along the left and upper edges of the image with its corresponding dynamic threshold. If the gray value in the region image is greater than the dynamic threshold, it is considered that the boundary of the region is found, and the region is updated accordingly; The other regions are the same as the upper left region. Then, the updated regions of all four regions are found, and the updated regions of the four regions are the projection area.

[0025] Refer to Figure 2 b of Figure 3 b of

[0026] As a preferred embodiment of the present invention, specifically, connected component analysis is performed on the edge image to find multiple connected components and mark them, including: Perform connected component analysis on the edge image through the connectedComponentsWithStats function of OpenCV to find multiple connected components and mark them, and create a stats array. The area information of each connected component is stored in the stats array; Traverse the stats array, extract the area of each connected component, and calculate the average area of the remaining connected components after removing the connected component with the largest area among all connected components at this time to obtain the first value; Remove the connected components with an average area less than the first value to obtain updated connected components; Then calculate the average area of the other connected components after removing the connected component with the largest area from the updated connected components to obtain the second value; Preset a scale threshold, and subtract the scale threshold from the second value to obtain the third value; Remove the connected components with an area less than the third value from the updated connected components to obtain the final connected components, and mark them as the result of the connected component analysis.

[0027] In this preferred embodiment, before inputting the cropped image into the deep learning model, the positions of connected regions (i.e., suspected obstacles) are pre-marked through image processing, and interference noises (eliminated according to the first value) and connected regions that are not sufficient to affect the projection of the projector (eliminated according to the third value) are removed, which can improve the processing efficiency of the deep learning model and the accuracy of the processing results.

[0028] As a preferred embodiment of the present invention, specifically, the projection area image is input into a pre-trained deep learning model, and a projection area image recording the position information of obstacles is output, including, Training the FasterR-CNN deep learning model based on the method of transfer learning; Converting the projection area image into Tensor format and inputting it into the trained FasterR-CNN deep learning model; Extracting the confidence score of each detection box from the trained FasterR-CNN model; According to a preset confidence threshold, screening out the detection boxes with a confidence higher than the confidence threshold and marking these detection boxes as obstacles; Recording the position information of the detection boxes marked as obstacles.

[0029] In this preferred embodiment, referring to Figure 4 a of Figure 4 b of Figure 4 In this embodiment, the trained FasterR-CNN model is used to convert the cropped image into RGB format, normalize the pixel values of the picture, then input the image into the model, and detect the obstacles in the image by judging whether the confidence is greater than 0.6, and record their position information. b of

[0030] As a preferred embodiment of the present invention, specifically, training the FasterR-CNN deep learning model based on the method of transfer learning includes, Loading a pre-trained model and modifying the model, specifically, loading a FasterR-CNN pre-trained model that has been trained on a large dataset, modifying the classifier at the end of the model, and adjusting the output categories of the last layer to 2; Preparing a dataset and a data loader, specifically, marking the rectangular box positions and categories of obstacles on each picture, and customizing a CustomDataset class, which inherits from Dataset and is used to load training data; Train the model and save the model. Specifically, use the SGD (Stochastic Gradient Descent) optimizer to optimize the parameters of the Faster R-CNN model, and use the StepLR step learning rate scheduler to dynamically adjust the learning rate. Use the mAP (mean Average Precision) as the evaluation metric to evaluate the model, find the optimal model, and save the optimal model selected through evaluation.

[0031] In this preferred embodiment, each photo is labeled in the form of {"image": "image_1.jpg", "boxes": [[x_min, y_min, x_max, y_max]],"labels": [1]}. The CustomDataset (custom dataset) class loads the images from the image paths and converts them to the RGB format. The labeled data corresponding to each image contains bounding box and label information.

[0032] And during the training process, the objective function is optimized by the SGD optimizer to minimize the detection error. We use StepLR to dynamically adjust the learning rate, reducing the learning rate every certain number of training epochs, and then select the optimal model according to the model performance and save it.

[0033] As a preferred embodiment of the present invention, specifically, based on the projection area image recording the obstacle position information, calculate the maximum projection area to obtain the obstacle-free area position information, including, Set the size of the projection area image as W×H, and the center point is ( , ), where , ”, the initial rectangular boundary is = , = , = , = , and the aspect ratio of the rectangle satisfies . In the projection area image, starting from the center of the image, search in the four directions of up, down, left, and right. Traverse each column in the up and down directions respectively until an obstacle is encountered and update and ; traverse each row in the left and right directions until an obstacle is encountered and update and . During the search process, if the boundary expansion changes the aspect ratio, adjust the longer boundary of the width or height to keep the rectangle ratio unchanged; finally, calculate the maximum rectangle area as , and use the maximum rectangle area as the obstacle-free area position information.

[0034] Refer toFigure 2 a and Figure 3 a. In this preferred embodiment, the above method can accurately find the position information of the obstacle-free area with the largest area and aspect ratio. The algorithm is used to find the largest projected area and record the position information of the obstacle-free area at this time. Then, based on the spatial relationship between the camera and the projector and the relative position information before and after image cropping, the position information is converted into the projector coordinate system. Finally, the projector parameters are adjusted to complete the projection of the projector, effectively avoiding obstacles.

[0035] As a preferred embodiment of the present invention, the method further includes judging whether the position information of the obstacle-free area is greater than a preset area threshold. If it is, no processing is performed. If not, an alarm reminder is given to inform the relevant staff to conduct verification and processing.

[0036] As a preferred embodiment of the present invention, specifically, converting the position information of the obstacle-free area into the projector coordinate system to perform the actual projection screen projection, including obtaining the spatial relationship between the camera and the projector, where the spatial relationship includes relative position and direction; converting the obtained position information of the obstacle-free area into two-dimensional coordinates in the projector coordinate system according to the spatial relationship between the camera and the projector and the relative information of the position information of the obstacle-free area; adjusting the projection angle, focal length, and zoom ratio of the projector according to the position and size information of the obstacle-free area in the two-dimensional coordinates; using the projector with adjusted parameters to perform the actual projection screen projection.

[0037] As a preferred embodiment of the present invention, the method further includes recording the position of the removed connected domain, and filling the content of the connected domain position by filling with the background color to update the content of the corresponding position during the actual projection screen projection.

[0038] Embodiment 2. The present invention also proposes a projector automatic obstacle avoidance device combining adaptive cropping of the screen and deep learning, including the following: A data acquisition module, configured to acquire the background image of the projection area, control the projector to transmit a white background image, and acquire the projection screen image at this time; A projection area image cropping module, configured to calculate a difference image based on the background image of the projection area and the projection screen image, and perform adaptive cropping on the difference image to obtain a projection area image; An obstacle position determination module, configured to input the projection area image into a pre-trained deep learning model, and output a projection area image recording the obstacle position information; An obstacle-free area position determination module, configured to calculate the maximum projection area based on the projection area image recording the obstacle position information, so as to obtain the obstacle-free area position information; A projection screen projection module, configured to convert the obstacle-free area position information into the projector coordinate system, and project the actual projection screen accordingly.

[0039] In addition, in each embodiment of the present invention, each functional module can be integrated into a processing module, or each module can exist physically alone, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules.

[0040] If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above-described embodiment methods of the present invention can also be completed by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-described method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or system, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium, etc., that can carry the computer program code.

[0041] Although the description of the present invention has been quite detailed and particularly described several of the embodiments, it is not intended to be limited to any of these details or embodiments or any particular embodiment, but rather should be regarded as providing a broad interpretation of these claims in light of the prior art by reference to the appended claims, thereby effectively covering the intended scope of the present invention. In addition, the present invention has been described above with embodiments foreseeable by the inventors for the purpose of providing a useful description, and non-substantive modifications to the present invention that are not currently foreseeable may still represent equivalent modifications of the present invention.

[0042] The above is only the preferred embodiment of the present invention. The present invention is not limited to the above-described embodiments. As long as it achieves the technical effects of the present invention by the same means, it should fall within the protection scope of the present invention. Within the protection scope of the present invention, its technical solutions and / or embodiments can have various different modifications and changes.

Claims

1. An automatic obstacle avoidance method for a projector that combines adaptive cropping of the screen and deep learning, characterized in that, The following are included: Obtain the background image of the projection area, control the projector to transmit a white background image, and obtain the projection screen image at this time; Calculate the difference image based on the background image of the projection area and the projection screen image, and perform adaptive cropping on the difference image to obtain the projection area image; Input the projection area image into a pre-trained deep learning model, and output the projection area image with the obstacle position information recorded; Based on the projection area image with the obstacle position information recorded, calculate the maximum projection area and then obtain the position information of the obstacle-free area; Convert the position information of the obstacle-free area to the projector coordinate system, and project the actual projection screen accordingly.

2. The automatic obstacle avoidance method for a projector that combines adaptive cropping of the screen and deep learning according to claim 1, characterized in that, Specifically, calculating the difference image based on the background image of the projection area and the projection screen image, and performing adaptive cropping on the difference image to obtain the projection area image includes, Subtract the projection screen from the background image of the projection area and take the absolute value to obtain the difference image; Perform grayscale processing on the difference image to obtain a grayscale image; Process the grayscale image through a pre-selected edge detection operator to obtain an edge image; Perform connected component analysis on the edge image to find multiple connected components and mark them; Take the connected component with the largest area among all connected components as the preliminary projection area; Crop the preliminary projection area from the grayscale image to obtain a preliminary image; Perform adaptive cropping processing on the preliminary image to obtain the projection area; Crop the projection area from the difference image, and mark the position of the connected components contained therein. The projection area is recorded as the position of the largest connected component contained therein.

3. The automatic obstacle avoidance method for a projector that combines adaptive cropping of the screen and deep learning according to claim 2, characterized in that, Specifically, performing adaptive cropping processing on the preliminary image to obtain the projection area includes, Divide the preliminary image into four regions: upper left, lower left, upper right, and lower right along the horizontal and vertical central axes; For any of the four regions, calculate its average grayscale value, and obtain the dynamic threshold of the region based on its average grayscale value combined with a preset scale factor. The dynamic threshold is the product of the average grayscale value and the scale factor; Taking the upper left region as an example, start traversing along the left and upper edges of the image with its corresponding dynamic threshold. If the grayscale value in the region image is greater than the dynamic threshold, it is considered that the boundary of the region is found, and the region is updated accordingly; The other regions are the same as the upper left region. Then, find the updated regions of all four regions. The updated regions of the four regions are the projection area.

4. The automatic obstacle avoidance method for a projector combining adaptive cropping of a screen and deep learning according to claim 2, characterized in that, Specifically, performing connected component analysis on the edge image to find multiple connected components and mark them includes, Perform connected component analysis on the edge image through the connectedComponentsWithStats function of OpenCV to find multiple connected components and mark them, and create a stats array. The stats array stores the area information of each connected component; Traverse the stats array, extract the area of each connected component, and calculate the average area of the remaining connected components after removing the connected component with the largest area among all connected components at this time to obtain a first value; Remove the connected components with an average area less than the first value to obtain updated connected components; Calculate the average area of the other connected components in the updated connected components after removing the connected component with the largest area to obtain a second value; Preset a scale threshold, and subtract the scale threshold from the second value to obtain a third value; Remove the connected components in the updated connected components with an area smaller than the third value to obtain the final connected components, and mark them as the result of the connected component analysis.

5. The automatic obstacle avoidance method for a projector that combines adaptive cropping of the screen and deep learning according to claim 1, characterized in that, Specifically, input the projection area image into a pre-trained deep learning model, and output a projection area image with obstacle position information recorded, including: Train the Faster R-CNN deep learning model based on the method of transfer learning; Convert the projection area image into the Tensor format and input it into the trained Faster R-CNN deep learning model; Extract the confidence score of each detection box from the trained Faster R-CNN model; According to a preset confidence threshold, screen out the detection boxes with a confidence higher than the confidence threshold, and mark these detection boxes as obstacles; Record the position information of the detection boxes marked as obstacles.

6. The projector automatic obstacle avoidance method combining adaptive cropping of the screen and deep learning according to claim 5, characterized in that, Specifically, training the Faster R-CNN deep learning model based on the method of transfer learning includes: Load a pre-trained model and modify the model. Specifically, load a Faster R-CNN pre-trained model that has been trained on a large dataset, modify the classifier at the end of the model, and adjust the output category of the last layer to 2; Prepare a dataset and a data loader. Specifically, mark the rectangular box position and category of the obstacle on each picture, and customize a CustomDataset class, which inherits from Dataset and is used to load training data; Train the model and save the model. Specifically, use the SGD stochastic gradient descent optimizer to optimize the parameters of the Faster R-CNN model, and use the StepLR step learning rate scheduler to dynamically adjust the learning rate. Use the mAP mean average precision as the evaluation index to evaluate the model, find the optimal model, and save the optimal model selected after evaluation.

7. The automatic obstacle avoidance method for a projector combining adaptive cropping of a screen and deep learning according to claim 1, characterized in that, Specifically, based on the projection area image with obstacle position information recorded, calculate the maximum projection area and then obtain the position information of the obstacle-free area, including: Let the image size of the projection area be W×H, and the center point be ( , ), where , ”. The initial rectangular boundary is = , = , = , = , and the aspect ratio of the rectangle satisfies . In the projection area image, starting from the image center, search in the four directions of up, down, left, and right. Traverse each column in the up and down directions respectively until an obstacle is encountered and update and ; Traverse each row in the left and right directions until an obstacle is encountered and update and , during the search process, if the aspect ratio is changed by the boundary expansion, adjust the longer boundary of the width or height to keep the rectangular ratio unchanged; finally, calculate the maximum rectangular area as , and use the maximum rectangular area as the position information of the obstacle-free area.

8. The automatic obstacle avoidance method of the projector combining adaptive cropping of the screen and deep learning according to claim 7, characterized in that, The method further includes: Judge whether the position information of the obstacle-free area is greater than a preset area threshold. If it is, no processing is performed. If not, an alarm reminder is given to inform the relevant staff to conduct a verification and processing.

9. The automatic obstacle avoidance method of a projector combining adaptive cropping of a screen and deep learning according to claim 1, characterized in that, Convert the position information of the obstacle-free area to the projector coordinate system, and project the actual projection screen accordingly, including: Obtain the spatial relationship between the camera and the projector, and the spatial relationship includes the relative position and direction; According to the spatial relationship between the camera and the projector and the relative information of the position information of the obstacle-free area, convert the obtained position information of the obstacle-free area into two-dimensional coordinates in the projector coordinate system; According to the position and size information of the obstacle-free area in the two-dimensional coordinates, adjust the projection angle, focal length and zoom ratio of the projector; Project the actual projection screen with the projector after adjusting the parameters.

10. An apparatus for automatic obstacle avoidance of a projector that jointly adapts to crop the screen and deep learning, characterized in that, Including the following: A data acquisition module, which is used to acquire the background image of the projection area and control the projector to transmit a white background image to acquire the projection screen image at this time; The projection area image cropping module is used to calculate a difference image based on the projection area background image and the projection screen image, and perform adaptive cropping on the difference image to obtain the projection area image; The obstacle position determination module is used to input the projection area image into a pre-trained deep learning model and output the projection area image with obstacle position information recorded; The obstacle-free area position determination module is used to calculate the maximum projection area based on the projection area image with obstacle position information recorded, and then obtain the obstacle-free area position information; The projection screen projection module is used to convert the obstacle-free area position information into the projector coordinate system, and project the actual projection screen accordingly.

Images