Projector automatic obstacle avoidance method and device combining adaptive cropping and deep learning

Abstract

The present invention relates to a method and device for automatic obstacle avoidance for a projector that combines adaptive cropping of images with deep learning. First, the projector projects a white background image, then uses a camera to capture the projected image. A differential image is calculated based on the background image of the projection area and the image of the projection screen. The differential image is adaptively cropped to obtain the projection area image. The differential image can highlight the difference between the projection area and the background area, making it easier to find the projection area image. A deep learning model is then used to detect obstacles in the cropped image and record their position information. The maximum projected area is calculated, and the position information of the obstacle-free area is obtained. This information is then converted to the projector coordinate system, and the final image is projected. This method achieves automatic obstacle avoidance for the projector, ensuring a clear and complete projection image. It can effectively solve the problem of poor projection effects in complex environments caused by traditional manual processing and existing technologies, and improve the projection quality of the projector in complex scenes.

Description

Technical Field

[0001] The present invention relates to the technical field related to projection imaging, and in particular to a method and device for automatic obstacle avoidance of a projector by combining adaptive image cropping and deep learning. Background Art

[0002] With the continuous advancement of projection technology, projectors have become widely used in many fields, including education, entertainment, and business, creating a diverse and rich visual experience for people. However, in actual use, many factors can negatively impact the projection effect. Obstructions such as dirt on the wall, various hanging objects, and haphazardly placed furniture often cause the projected image to appear blurry, with missing content or even cast shadows on the screen, significantly reducing the clarity and visual beauty of viewing and presenting.

[0003] Traditionally, addressing these projection obstacles has relied primarily on manual methods. Operators visually observe any anomalies in the projected image and then manually adjust the projector's angle, position, and other parameters, or perform simple adjustments to the surrounding environment. However, manual operations are limited by the human eye's precision, making it difficult to accurately judge subtle image changes and complex environmental factors. This makes it difficult to efficiently and accurately achieve ideal projection effects when faced with complex scene layouts and a variety of obstacles, and it consumes a significant amount of manpower and time.

[0004] Despite recent advancements, modern technology still struggles to accurately identify projection backgrounds. It struggles to distinguish between screens of varying materials, colors, and textures. In environments with complex lighting and large variations in brightness, it's difficult to accurately analyze the relationship between background and obstacles, making it difficult to intelligently adjust projection parameters. This makes it difficult to achieve high projection quality standards in complex environments and fails to fully meet user projection quality requirements in diverse scenarios. Summary of the Invention

[0005] The purpose of the present invention is to solve at least one of the deficiencies of the prior art and to provide a method and device for automatic obstacle avoidance of a projector by combining adaptive cropping of images with deep learning.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] Specifically, a projector automatic obstacle avoidance method combining adaptive cropping and deep learning is proposed, including the following:

[0008] Obtain the background image of the projection area, and control the projector to transmit the white background image to obtain the projection screen image at this time;

[0009] Calculating a differential image based on the projection area background image and the projection screen image, and adaptively cropping the differential image to obtain a projection area image;

[0010] Input the projection area image into a pre-trained deep learning model, and output the projection area image with obstacle location information recorded;

[0011] Based on the projection area image with obstacle location information recorded, the maximum projection area is calculated to obtain the obstacle-free area location information;

[0012] The position information of the obstacle-free area is converted into the projector coordinate system for actual projection of the image.

[0013] Further, specifically, a differential image is calculated based on the projection area background image and the projection screen image, and the projection area image is obtained by adaptively cropping the differential image, including:

[0014] Subtracting the projection image from the background image of the projection area and taking the absolute value to obtain a differential image;

[0015] Performing grayscale processing on the differential image to obtain a grayscale image;

[0016] Processing the grayscale image using a pre-selected edge detection operator to obtain an edge image;

[0017] Performing a connected domain analysis on the edge image to find multiple connected domains and marking them;

[0018] The connected domain with the largest area among all connected domains is used as the preliminary projection area;

[0019] Cutting out the preliminary projection area from the grayscale image to obtain a preliminary image;

[0020] Obtaining a projection area by adaptively cropping the preliminary image;

[0021] The projection area is cut out from the differential image, and the positions of the connected domains contained therein are marked. The projection area is recorded as the position of the largest connected domain contained therein.

[0022] Furthermore, specifically, the projection area is obtained by adaptively cropping the preliminary image, including:

[0023] Divide the preliminary image into four areas: upper left, lower left, upper right, and lower right, according to two horizontal and vertical central axes;

[0024] For any area of ​​the four areas, calculate its average grayscale value, and obtain a dynamic threshold of the area based on the average grayscale value combined with a preset scale factor, where the dynamic threshold is the product of the average grayscale value and the scale factor;

[0025] Taking the upper left area as an example, traverse along the left and upper edges of the image using their corresponding dynamic thresholds. If the grayscale value in the image of this area is greater than the dynamic threshold, it is considered that the boundary of this area has been found, and the area is updated accordingly.

[0026] The other areas are processed in the same way as the upper left area, and then the updated areas of all four areas are found. The updated areas of the four areas are the projection areas.

[0027] Further, specifically, the edge image is subjected to a connected domain analysis to find multiple connected domains and mark them, including:

[0028] Performing a connected domain analysis on the edge image using the connectedComponentsWithStats function of OpenCV to find multiple connected domains and mark them, creating a stats array that stores the area information of each connected domain;

[0029] Traverse the stats array, extract the area of ​​each connected domain, and calculate the average area of ​​the remaining connected domains after removing the connected domain with the largest area from all connected domains to obtain the first value;

[0030] Eliminate the connected domains whose average area is smaller than the first value to obtain updated connected domains;

[0031] Then, the average area of ​​other connected domains in the updated connected domain excluding the connected domain with the largest area is calculated to obtain a second value;

[0032] A scale threshold is preset, and a third value is obtained by subtracting the scale threshold from the second value;

[0033] The connected domains with an area smaller than the third value in the updated connected domain are eliminated to obtain a final connected domain, and the final connected domain is marked as the result of the connected domain analysis.

[0034] Furthermore, specifically, the projection area image is input into a pre-trained deep learning model, and the projection area image with obstacle position information recorded is output, including:

[0035] Train the FasterR-CNN deep learning model based on transfer learning;

[0036] Convert the projection area image into Tensor format and input it into the trained FasterR-CNN deep learning model;

[0037] Extract the confidence score of each detection box from the trained FasterR-CNN model;

[0038] According to a preset confidence threshold, detection frames with confidence levels higher than the confidence threshold are screened out, and these detection frames are recorded as obstacles;

[0039] Record the position information of the detection frame recorded as an obstacle.

[0040] Furthermore, specifically, the FasterR-CNN deep learning model is trained based on transfer learning, including:

[0041] Load a pre-trained model and modify it. Specifically, load a FasterR-CNN pre-trained model that has been trained on a large dataset, modify the final classifier of the model, and adjust the output category of the last layer to 2.

[0042] Prepare the dataset and data loader. Specifically, mark the location and category of the obstacle rectangle on each image. Customize a CustomDataset class, which inherits from Dataset, to load training data.

[0043] Train the model and save the model. Specifically, use the SGD stochastic gradient descent optimizer to optimize the parameters of the FasterR-CNN model, and use the StepLR step size learning rate scheduler to dynamically adjust the learning rate. Use the mAP average precision mean as the evaluation metric to evaluate the model, find the optimal model, and save the optimal model selected after evaluation.

[0044] Further, specifically, based on the projection area image with obstacle location information recorded, the maximum projection area is calculated to obtain the obstacle-free area location information, including:

[0045] Assume that the image size of the projection area is W×H and the center point is ( , ),in , ", the initial rectangular boundary is = , = , = , = , and the aspect ratio of the rectangle satisfies In the projection area image, starting from the center of the image, search in the four directions of upward, downward, left, and right, and traverse each column in the upward and downward directions respectively until an obstacle is encountered and the update and ; Traverse each row left and right until an obstacle is encountered and update and During the search process, if the boundary expansion changes the aspect ratio, the width or height of the longer boundary is adjusted to keep the rectangle ratio unchanged; the final calculation of the maximum rectangle area is The maximum rectangular area is used as the obstacle-free area location information.

[0046] Furthermore, the method further comprises:

[0047] Determine whether the obstacle-free area location information is greater than the preset area threshold. If so, no processing is performed. If not, an alarm is issued to inform relevant staff to verify and handle it.

[0048] Furthermore, specifically, the position information of the obstacle-free area is converted into the projector coordinate system to perform actual projection of the image, including:

[0049] Acquire a spatial relationship between the camera and the projector, wherein the spatial relationship includes a relative position and a direction;

[0050] According to the spatial relationship between the camera and the projector and the relative information of the obstacle-free area position information, the obtained obstacle-free area position information is converted into two-dimensional coordinates in the projector coordinate system;

[0051] Adjust the projection angle, focal length and zoom ratio of the projector according to the position and size information of the obstacle-free area in the two-dimensional coordinates;

[0052] The actual projection image is projected using the projector with adjusted parameters.

[0053] Furthermore, the method further comprises:

[0054] The positions of the removed connected domains are recorded, and the connected domain positions are filled with content by filling with background color to update the content of the corresponding positions when the actual projection image is projected.

[0055] The present invention also proposes a device for automatic obstacle avoidance of a projector combining adaptive cropping and deep learning, comprising the following:

[0056] The data acquisition module is used to obtain the background image of the projection area and control the projector to transmit the white background image to obtain the projection screen image at this time;

[0057] A projection area image cropping module, configured to calculate a differential image based on the projection area background image and the projection screen image, and adaptively crop the differential image to obtain the projection area image;

[0058] An obstacle position determination module is configured to input the projection area image into a pre-trained deep learning model and output a projection area image having obstacle position information recorded therein;

[0059] An obstacle-free area location determination module is used to calculate the maximum projection area based on the projection area image that records the obstacle location information and thus obtain the obstacle-free area location information;

[0060] The projection screen projection module is used to convert the position information of the obstacle-free area into the projector coordinate system for actual projection of the projection screen.

[0061] The beneficial effects of the present invention are:

[0062] The present invention proposes a method and device for automatic obstacle avoidance for a projector that combines adaptive cropping of images with deep learning. First, the projector projects a white background image, then uses a camera to capture the projection image. A differential image is calculated based on the background image of the projection area and the projection image. The differential image is adaptively cropped to obtain the projection area image. The differential image can highlight the difference between the projection area and the background area, making it easier to find the projection area image. Then, a deep learning model is used to detect obstacles in the cropped image and record their position information. The maximum projection area is calculated to obtain the position information of the obstacle-free area, which is then converted to the projector coordinate system and finally the final image is projected. In this way, automatic obstacle avoidance of the projector is achieved, and the projection image is ensured to be clear and complete. This can effectively solve the problem of poor projection effects of traditional manual processing and existing technologies in complex environments, and improve the projection quality of the projector in complex scenes.

[0063] In addition, before the cropped image is input into the deep learning model, the connected domain positions (i.e., suspected obstacles) are pre-marked through image processing, and interference noise (eliminated according to the first numerical value) and connected domains that are not sufficient to affect the projector projection (eliminated according to the third numerical value) are eliminated, which can improve the processing efficiency of the deep learning model and the accuracy of the processing results. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] The above and other features of the present disclosure will become more apparent through a detailed description of the embodiments shown in conjunction with the accompanying drawings. The same reference numerals in the drawings of the present disclosure represent the same or similar elements. Obviously, the drawings described below are only some embodiments of the present disclosure. It is possible for a person skilled in the art to derive other drawings based on these drawings without inventive effort. In the drawings:

[0065] Figure 1 Shown is a flow chart of the projector automatic obstacle avoidance method combining adaptive cropping and deep learning of the present invention;

[0066] Figure 2 (a) shows a first complex environment image based on aperture interference in an embodiment of the present invention, and (b) shows a second complex environment image based on aperture interference in an embodiment of the present invention;

[0067] Figure 3 (a) is an image after adaptive cropping of the first complex environment image in an embodiment of the present invention, and (b) is an image after adaptive cropping of the second complex environment image in an embodiment of the present invention;

[0068] Figure 4 (a) is an image of the first complex environment image after adaptive cropping for obstacle detection in an embodiment of the present invention, and (b) is an image of the second complex environment image after non-adaptive cropping for obstacle detection in an embodiment of the present invention. DETAILED DESCRIPTION

[0069] The following will be combined with the embodiments and drawings to clearly and completely describe the concept, specific structure and technical effects of the present invention so as to fully understand the purpose, scheme and effect of the present invention. It should be noted that the embodiments and features in the embodiments of this application can be combined with each other unless there is a conflict. The same reference numerals used throughout the drawings indicate the same or similar parts.

[0070] Example 1, with reference to Figure 1 The present invention proposes a projector automatic obstacle avoidance method combining adaptive cropping and deep learning, including the following:

[0071] Obtain the background image of the projection area, and control the projector to transmit the white background image to obtain the projection screen image at this time;

[0072] Calculating a differential image based on the projection area background image and the projection screen image, and adaptively cropping the differential image to obtain a projection area image;

[0073] Input the projection area image into a pre-trained deep learning model, and output the projection area image with obstacle location information recorded;

[0074] Based on the projection area image with obstacle location information recorded, the maximum projection area is calculated to obtain the obstacle-free area location information;

[0075] The position information of the obstacle-free area is converted into the projector coordinate system for actual projection of the image.

[0076] In this first embodiment, a projector is first used to project a white background image. A camera is then used to capture the projected image. A differential image is calculated based on the background image of the projection area and the projection image. The differential image is then adaptively cropped to obtain the projection area image. The differential image highlights the difference between the projection area and the background area, making it easier to find the projection area image. A deep learning model is then used to detect obstacles in the cropped image and record their location information. The maximum projected area is calculated, and the location information of the obstacle-free area is obtained. This information is then converted to the projector coordinate system, and the final image is projected. This enables the projector to automatically avoid obstacles, ensuring a clear and complete projection image. This effectively solves the problem of poor projection effects in complex environments caused by traditional manual processing and existing technologies, and improves the projection quality of the projector in complex scenes.

[0077] As a preferred embodiment of the present invention, specifically, a differential image is calculated based on the projection area background image and the projection screen image, and the projection area image is obtained by adaptively cropping the differential image, including:

[0078] Subtracting the projection image from the background image of the projection area and taking the absolute value to obtain a differential image;

[0079] Performing grayscale processing on the differential image to obtain a grayscale image;

[0080] Processing the grayscale image using a pre-selected edge detection operator to obtain an edge image;

[0081] Performing a connected domain analysis on the edge image to find multiple connected domains and marking them;

[0082] The connected domain with the largest area among all connected domains is used as the preliminary projection area;

[0083] Cutting out the preliminary projection area from the grayscale image to obtain a preliminary image;

[0084] Obtaining a projection area by adaptively cropping the preliminary image;

[0085] The projection area is cut out from the differential image, and the positions of the connected domains contained therein are marked. The projection area is recorded as the position of the largest connected domain contained therein.

[0086] In this preferred embodiment, considering the problem of aperture interference, the largest connected area cannot be used as the projection area image. Therefore, the area with less aperture interference is found from the largest connected domain, i.e., the preliminary projection area, through adaptive cropping to obtain the projection area with less aperture interference.

[0087] As a preferred embodiment of the present invention, specifically, the projection area is obtained by adaptively cropping the preliminary image, including:

[0088] Divide the preliminary image into four areas: upper left, lower left, upper right, and lower right, according to two horizontal and vertical central axes;

[0089] For any area of ​​the four areas, calculate its average grayscale value, and obtain a dynamic threshold of the area based on the average grayscale value combined with a preset scale factor, where the dynamic threshold is the product of the average grayscale value and the scale factor;

[0090] Taking the upper left area as an example, traverse along the left and upper edges of the image using their corresponding dynamic thresholds. If the grayscale value in the image of this area is greater than the dynamic threshold, it is considered that the boundary of this area has been found, and the area is updated accordingly.

[0091] The other areas are processed in the same way as the upper left area, and then the updated areas of all four areas are found. The updated areas of the four areas are the projection areas.

[0092] Reference Figure 2 b and Figure 3 In the preferred embodiment, adaptive cropping is performed in the above manner to obtain a projection area with less aperture interference as much as possible.

[0093] As a preferred embodiment of the present invention, specifically, the connected domain analysis is performed on the edge image to find multiple connected domains and mark them, including:

[0094] Performing a connected domain analysis on the edge image using the connectedComponentsWithStats function of OpenCV to find multiple connected domains and mark them, creating a stats array that stores the area information of each connected domain;

[0095] Traverse the stats array, extract the area of ​​each connected domain, and calculate the average area of ​​the remaining connected domains after removing the connected domain with the largest area from all connected domains to obtain the first value;

[0096] Eliminate the connected domains whose average area is smaller than the first value to obtain updated connected domains;

[0097] Then, the average area of ​​other connected domains in the updated connected domain excluding the connected domain with the largest area is calculated to obtain a second value;

[0098] A scale threshold is preset, and a third value is obtained by subtracting the scale threshold from the second value;

[0099] The connected domains with an area smaller than the third value in the updated connected domain are eliminated to obtain a final connected domain, and the final connected domain is marked as the result of the connected domain analysis.

[0100] In this preferred embodiment, before the cropped image is input into the deep learning model, the connected domain position (i.e., suspected obstacles) is pre-marked through image processing, and the interference noise (eliminated according to the first numerical value) and the connected domain that is not sufficient to affect the projection of the projector (eliminated according to the third numerical value) are eliminated, which can improve the processing efficiency of the deep learning model and the accuracy of the processing results.

[0101] As a preferred embodiment of the present invention, specifically, the projection area image is input into a pre-trained deep learning model, and the projection area image with obstacle position information recorded is output, including:

[0102] Train the FasterR-CNN deep learning model based on transfer learning;

[0103] Convert the projection area image into Tensor format and input it into the trained FasterR-CNN deep learning model;

[0104] Extract the confidence score of each detection box from the trained FasterR-CNN model;

[0105] According to a preset confidence threshold, detection frames with confidence levels higher than the confidence threshold are screened out, and these detection frames are recorded as obstacles;

[0106] Record the position information of the detection frame recorded as an obstacle.

[0107] In this preferred embodiment, see Figure 4 a、 Figure 4 In this embodiment, the trained Faster R-CNN model is used to convert the cropped image into RGB format and normalize the pixel values ​​of the image. The image is then input into the model, and obstacles in the image are detected by judging whether the confidence level is greater than 0.6, and their location information is recorded. Figure 4 The b shows that when adaptive cropping is not performed, obstacles outside the screen interfere with the detection, thus highlighting the role and necessity of adaptive cropping.

[0108] As a preferred embodiment of the present invention, specifically, the FasterR-CNN deep learning model is trained based on transfer learning, including:

[0109] Load a pre-trained model and modify it. Specifically, load a FasterR-CNN pre-trained model that has been trained on a large dataset, modify the final classifier of the model, and adjust the output category of the last layer to 2.

[0110] Prepare the dataset and data loader. Specifically, mark the location and category of the obstacle rectangle on each image. Customize a CustomDataset class, which inherits from Dataset, to load training data.

[0111] Train the model and save the model. Specifically, use the SGD stochastic gradient descent optimizer to optimize the parameters of the FasterR-CNN model, and use the StepLR step size learning rate scheduler to dynamically adjust the learning rate. Use the mAP average precision mean as the evaluation metric to evaluate the model, find the optimal model, and save the optimal model selected after evaluation.

[0112] In this preferred embodiment, each photo is annotated in the form of {"image":"image_1.jpg","boxes":[[x_min,y_min,x_max,y_max]],"labels":[1]}. The CustomDataset class loads the image from the image path and converts it to RGB format. The annotation data corresponding to each image includes bounding box and label information.

[0113] The training process uses the SGD optimizer to optimize the objective function to minimize the detection error. We use StepLR to dynamically adjust the learning rate, reducing it after a certain number of training cycles. Then, based on the model performance, we select the optimal model and save it.

[0114] As a preferred embodiment of the present invention, specifically, based on the projection area image with obstacle position information recorded, calculating the maximum projection area and then obtaining the obstacle-free area position information includes:

[0115] Assume that the image size of the projection area is W×H and the center point is ( , ),in , ", the initial rectangular boundary is = , = , = , = , and the aspect ratio of the rectangle satisfies In the projection area image, starting from the center of the image, search in the four directions of upward, downward, left, and right, and traverse each column in the upward and downward directions respectively until an obstacle is encountered and the update and ; Traverse each row left and right until an obstacle is encountered and update and During the search process, if the boundary expansion changes the aspect ratio, the width or height of the longer boundary is adjusted to keep the rectangle ratio unchanged; the final calculation of the maximum rectangle area is The maximum rectangular area is used as the obstacle-free area location information.

[0116] Reference Figure 2 a and Figure 3In this preferred embodiment, the above method can accurately find the location information of the largest obstacle-free area that meets the aspect ratio. An algorithm is used to find the maximum projection area and record the location information of the obstacle-free area at that time. This location information is then converted to the projector coordinate system based on the spatial relationship between the camera and projector and the relative position information before and after image cropping. Finally, the projector parameters are adjusted to complete the projection, effectively avoiding obstacles.

[0117] As a preferred embodiment of the present invention, the method further comprises:

[0118] Determine whether the obstacle-free area location information is greater than the preset area threshold. If so, no processing is performed. If not, an alarm is issued to inform relevant staff to verify and handle it.

[0119] As a preferred embodiment of the present invention, specifically, the position information of the obstacle-free area is converted into a projector coordinate system to perform actual projection of the projection image, including:

[0120] Acquire a spatial relationship between the camera and the projector, wherein the spatial relationship includes a relative position and a direction;

[0121] According to the spatial relationship between the camera and the projector and the relative information of the obstacle-free area position information, the obtained obstacle-free area position information is converted into two-dimensional coordinates in the projector coordinate system;

[0122] Adjust the projection angle, focal length and zoom ratio of the projector according to the position and size information of the obstacle-free area in the two-dimensional coordinates;

[0123] The actual projection image is projected using the projector with adjusted parameters.

[0124] As a preferred embodiment of the present invention, the method further comprises:

[0125] The positions of the removed connected domains are recorded, and the connected domain positions are filled with content by filling with background color to update the content of the corresponding positions when the actual projection image is projected.

[0126] In Example 2, the present invention further proposes a device for automatic obstacle avoidance of a projector combining adaptive cropping and deep learning, comprising the following:

[0127] The data acquisition module is used to obtain the background image of the projection area and control the projector to transmit the white background image to obtain the projection screen image at this time;

[0128] A projection area image cropping module, configured to calculate a differential image based on the projection area background image and the projection screen image, and adaptively crop the differential image to obtain the projection area image;

[0129] An obstacle position determination module is configured to input the projection area image into a pre-trained deep learning model and output a projection area image having obstacle position information recorded therein;

[0130] An obstacle-free area location determination module is used to calculate the maximum projection area based on the projection area image that records the obstacle location information and thus obtain the obstacle-free area location information;

[0131] The projection screen projection module is used to convert the position information of the obstacle-free area into the projector coordinate system for actual projection of the projection screen.

[0132] In addition, the functional modules in various embodiments of the present invention may be integrated into a single processing module, or each module may exist physically separately, or two or more modules may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or software functional modules.

[0133] If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or system that can carry the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0134] Although the present invention has been described in considerable detail and with particularity with respect to several described embodiments, it is not intended to be limited to any of these details or embodiments or any particular embodiment, but rather should be construed as providing a broad possible interpretation of these claims in view of the prior art by reference to the appended claims, thereby effectively encompassing the intended scope of the invention. In addition, the invention has been described above in terms of embodiments foreseen by the inventors for the purpose of providing a useful description, and those insubstantial modifications of the invention that are not currently foreseen may still represent equivalent modifications of the invention.

[0135] The above description is merely a preferred embodiment of the present invention. The present invention is not limited to the above-described embodiments. As long as the technical effects of the present invention are achieved by the same means, they shall fall within the scope of protection of the present invention. Within the scope of protection of the present invention, various modifications and variations of the technical solutions and / or implementation methods may be made.

Claims

1. The projector automatic obstacle avoidance method based on adaptive cropping and deep learning is characterized by: These include: Obtain the background image of the projection area, and control the projector to transmit the white background image to obtain the projection screen image at this time; Calculating a differential image based on the projection area background image and the projection screen image, and adaptively cropping the differential image to obtain a projection area image; Input the projection area image into a pre-trained deep learning model, and output the projection area image with obstacle location information recorded; Based on the projection area image with obstacle location information recorded, the maximum projection area is calculated to obtain the obstacle-free area location information; Convert the position information of the obstacle-free area to the projector coordinate system for actual projection of the image; Specifically, a differential image is calculated based on the projection area background image and the projection screen image, and the projection area image is obtained by adaptively cropping the differential image, including: Subtracting the projection image from the background image of the projection area and taking the absolute value to obtain a differential image; Performing grayscale processing on the differential image to obtain a grayscale image; Processing the grayscale image using a pre-selected edge detection operator to obtain an edge image; Performing a connected domain analysis on the edge image to find multiple connected domains and marking them; The connected domain with the largest area among all connected domains is used as the preliminary projection area; Cutting out the preliminary projection area from the grayscale image to obtain a preliminary image; Obtaining a projection area by adaptively cropping the preliminary image; The projection area is cut out from the difference image, and the positions of the connected domains contained therein are marked, and the projection area is recorded as the position of the largest connected domain contained therein; Specifically, the projection area is obtained by adaptively cropping the preliminary image, including: Divide the preliminary image into four areas: upper left, lower left, upper right, and lower right, according to two horizontal and vertical central axes; For any area of ​​the four areas, calculate its average grayscale value, and obtain a dynamic threshold of the area based on the average grayscale value combined with a preset scale factor, where the dynamic threshold is the product of the average grayscale value and the scale factor; Taking the upper left area as an example, traverse along the left and upper edges of the image using their corresponding dynamic thresholds. If the grayscale value in the image of this area is greater than the dynamic threshold, it is considered that the boundary of this area has been found, and the area is updated accordingly. The other areas are the same as the upper left area, and then the updated areas of all four areas are found. The updated areas of the four areas are the projection areas. Specifically, the edge image is subjected to a connected domain analysis to find multiple connected domains and mark them, including: Performing a connected domain analysis on the edge image using the connectedComponentsWithStats function of OpenCV to find multiple connected domains and mark them, creating a stats array that stores the area information of each connected domain; Traverse the stats array, extract the area of ​​each connected domain, and calculate the average area of ​​the remaining connected domains after removing the connected domain with the largest area from all connected domains to obtain the first value; Eliminate the connected domains whose average area is smaller than the first value to obtain updated connected domains; Then, the average area of ​​other connected domains in the updated connected domain excluding the connected domain with the largest area is calculated to obtain a second value; A scale threshold is preset, and a third value is obtained by subtracting the scale threshold from the second value; The connected domains with an area smaller than the third value in the updated connected domain are eliminated to obtain a final connected domain, and the final connected domain is marked as the result of the connected domain analysis.

2. The projector automatic obstacle avoidance method based on combined adaptive cropping and deep learning according to claim 1 is characterized in that: Specifically, the projection area image is input into a pre-trained deep learning model, and the projection area image with obstacle position information recorded is output, including: Train the FasterR-CNN deep learning model based on transfer learning; Convert the projection area image into Tensor format and input it into the trained FasterR-CNN deep learning model; Extract the confidence score of each detection box from the trained FasterR-CNN model; According to a preset confidence threshold, detection frames with confidence levels higher than the confidence threshold are screened out, and these detection frames are recorded as obstacles; Record the position information of the detection frame recorded as an obstacle.

3. The projector automatic obstacle avoidance method based on combined adaptive cropping and deep learning according to claim 2 is characterized in that: Specifically, the FasterR-CNN deep learning model is trained based on transfer learning, including: Load a pre-trained model and modify it. Specifically, load a FasterR-CNN pre-trained model that has been trained on a large dataset, modify the final classifier of the model, and adjust the output category of the last layer to 2. Prepare the dataset and data loader. Specifically, mark the location and category of the obstacle rectangle on each image. Customize a CustomDataset class, which inherits from Dataset, to load training data. Train the model and save the model. Specifically, use the SGD stochastic gradient descent optimizer to optimize the parameters of the FasterR-CNN model, and use the StepLR step size learning rate scheduler to dynamically adjust the learning rate. Use the mAP average precision mean as the evaluation metric to evaluate the model, find the optimal model, and save the optimal model selected after evaluation.

4. The projector automatic obstacle avoidance method based on combined adaptive cropping and deep learning according to claim 1, characterized in that: Specifically, based on the projection area image with obstacle location information recorded, the maximum projection area is calculated to obtain the obstacle-free area location information, including: Assume that the image size of the projection area is W×H and the center point is ,in , the initial rectangular boundary is = , = , = , = , and the aspect ratio of the rectangle satisfies In the projection area image, starting from the center of the image, search in the four directions of upward, downward, left, and right, and traverse each column in the upward and downward directions respectively until an obstacle is encountered and the update and ; Traverse each row left and right until an obstacle is encountered and update and During the search process, if the boundary expansion changes the aspect ratio, the width or height of the longer boundary is adjusted to keep the rectangle ratio unchanged; the final calculation of the maximum rectangle area is The maximum rectangular area is used as the obstacle-free area location information.

5. The projector automatic obstacle avoidance method based on combined adaptive cropping and deep learning according to claim 4 is characterized in that: The method further comprises, Determine whether the obstacle-free area location information is greater than the preset area threshold. If so, no processing is performed. If not, an alarm is issued to inform relevant staff to verify and handle it.

6. The projector automatic obstacle avoidance method based on combined adaptive cropping and deep learning according to claim 1, characterized in that: The position information of the obstacle-free area is converted to the projector coordinate system for actual projection of the image, including: Acquire a spatial relationship between the camera and the projector, wherein the spatial relationship includes a relative position and a direction; According to the spatial relationship between the camera and the projector and the relative information of the obstacle-free area position information, the obtained obstacle-free area position information is converted into two-dimensional coordinates in the projector coordinate system; Adjust the projection angle, focal length and zoom ratio of the projector according to the position and size information of the obstacle-free area in the two-dimensional coordinates; The actual projection image is projected using the projector with adjusted parameters.

7. A device for automatic obstacle avoidance of a projector combining adaptive cropping and deep learning, characterized in that: The method for automatic obstacle avoidance of a projector by combining adaptive cropping and deep learning as described in any one of claims 1 to 6 is applied, and the device includes the following: The data acquisition module is used to obtain the background image of the projection area and control the projector to transmit the white background image to obtain the projection screen image at this time; A projection area image cropping module, configured to calculate a differential image based on the projection area background image and the projection screen image, and adaptively crop the differential image to obtain the projection area image; An obstacle position determination module is configured to input the projection area image into a pre-trained deep learning model and output a projection area image having obstacle position information recorded therein; An obstacle-free area location determination module is used to calculate the maximum projection area based on the projection area image that records the obstacle location information and thus obtain the obstacle-free area location information; The projection screen projection module is used to convert the position information of the obstacle-free area into the projector coordinate system for actual projection of the projection screen.

Images