Projection equipment and projection obstacle avoidance method

Abstract

The invention provides a projection device and a projection obstacle avoidance method, and the method can respond to an obstacle avoidance instruction, obtains a sampling image obtained by a camera when a light emitting assembly projects a pure color chart, extracts a projectable area in the sampling image, and obtains a first transformation matrix and a second transformation matrix, the first transformation matrix is a transformation matrix between a light-emitting assembly coordinate system and a projection surface world coordinate system, and the second transformation matrix is a transformation matrix between a camera coordinate system and the projection surface world coordinate system. And delimiting a target projection area in a projectable area under a projection surface world coordinate system according to the second transformation matrix, calculating a target projection area under a light emitting component coordinate system according to the first transformation matrix, and controlling the light emitting component to project the projection content to the target projection area so as to realize projection obstacle avoidance. According to the method, feature graph card shooting and pose calibration of a camera and an optical machine are not needed, the obstacle avoidance time is shortened, and the user experience is improved.

Description

Technical Field

[0001] The present application relates to the technical field of projection equipment, and in particular to a projection equipment and a projection obstacle avoidance method. Background Art

[0002] A projection device is a display device that can project images or videos onto a screen. The projection device can project laser light of a specific color into the projection area and form a specific image through the refraction of an optical lens assembly. Due to the portability of the projection device, users can move it during the projection process to project images or videos in different directions.

[0003] When the projection device projects media data, the wall or screen can be regarded as the projection area. When there are obstacles in the projection area, the projected media data will cause the image to fluctuate or be incomplete due to the existence of obstacles. Therefore, in order to improve the user's viewing experience, the projection device detects obstacles in the projection area, performs automatic obstacle avoidance, and re-determines the projection area.

[0004] However, when the projection device performs obstacle avoidance, it relies on the calibration data between the camera and the optical machine. When there is an error in the camera's internal parameter data, the obstacle avoidance result will be affected. In addition, when performing the pose calibration between the camera and the optical machine, the optical machine needs to project a white card and a checkerboard card in sequence, which makes the calibration time-consuming, thereby reducing the efficiency of automatic obstacle avoidance. Summary of the invention

[0005] The present application provides a projection device and a projection obstacle avoidance method to solve the problem of low efficiency of automatic obstacle avoidance.

[0006] In a first aspect, the present application provides a projection device, comprising a light emitting component, a camera, a distance sensor and a controller, wherein the light emitting component is configured to project projection content onto a projection surface; the camera is configured to capture a sample image; the distance sensor is configured to collect depth information between the distance sensor and the projection surface; and the controller is configured to execute the following program steps:

[0007] In response to the obstacle avoidance instruction, a sample image is acquired, where the sample image is an image captured by the camera when the light emitting component projects a pure color picture card;

[0008] Extracting a projectable area from the sampled image, where the projectable area is an area in the sampled image that does not contain obstacles;

[0009] Obtain a first transformation matrix and a second transformation matrix, wherein the first transformation matrix is ​​a transformation matrix between the light output component coordinate system and the projection surface world coordinate system, and the second transformation matrix is ​​a transformation matrix between the camera coordinate system and the projection surface world coordinate system, and the first transformation matrix and the second transformation matrix are calculated based on the depth information collected by the distance sensor;

[0010] According to the second transformation matrix, a target projection area is defined in the projectable area in the world coordinate system of the projection surface, wherein the target projection area is an inscribed rectangular area of ​​a maximum preset aspect ratio contained in the projectable area in the world coordinate system of the projection surface;

[0011] According to the first transformation matrix, a target projection area in the light emitting component coordinate system is calculated, and the light emitting component is controlled to project the projection content to the target projection area in the light emitting component coordinate system.

[0012] In a second aspect, the present application further provides a projection obstacle avoidance method, which is applied to the above-mentioned projection device, and the method comprises:

[0013] In response to the obstacle avoidance instruction, a sample image is acquired, where the sample image is an image captured by the camera when the light emitting component projects a pure color picture card;

[0014] Extracting a projectable area from the sampled image, where the projectable area is an area in the sampled image that does not contain obstacles;

[0015] Obtain a first transformation matrix and a second transformation matrix, wherein the first transformation matrix is ​​a transformation matrix between the light output component coordinate system and the projection surface world coordinate system, and the second transformation matrix is ​​a transformation matrix between the camera coordinate system and the projection surface world coordinate system, and the first transformation matrix and the second transformation matrix are calculated based on the depth information collected by the distance sensor;

[0016] According to the second transformation matrix, a target projection area is defined in the projectable area in the world coordinate system of the projection surface, wherein the target projection area is an inscribed rectangular area of ​​a maximum preset aspect ratio contained in the projectable area in the world coordinate system of the projection surface;

[0017] According to the first transformation matrix, a target projection area in the light emitting component coordinate system is calculated, and the light emitting component is controlled to project the projection content to the target projection area in the light emitting component coordinate system.

[0018] It can be seen from the above technical solutions that the present application provides a projection device and a projection obstacle avoidance method, which can respond to obstacle avoidance instructions, obtain a sampled image taken by the camera when the light-emitting component projects a solid color picture card, extract the projectable area in the sampled image, and then obtain a first transformation matrix and a second transformation matrix, wherein the first transformation matrix is ​​a transformation matrix between the light-emitting component coordinate system and the projection surface world coordinate system, and the second transformation matrix is ​​a transformation matrix between the camera coordinate system and the projection surface world coordinate system. According to the second transformation matrix, the target projection area is delineated in the projectable area under the projection surface world coordinate system, and according to the first transformation matrix, the target projection area under the light-emitting component coordinate system is calculated, and the light-emitting component is controlled to project the projection content to the target projection area, thereby achieving projection obstacle avoidance. The method can shorten the obstacle avoidance time and improve the user experience without the need for feature picture card shooting and camera and optical machine posture calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 This is a schematic diagram of the projection state of the projection device in the embodiment of the present application;

[0021] Figure 2 This is a schematic diagram of the structure of a projection device in an embodiment of the present application;

[0022] Figure 3 This is a schematic diagram of the optical-mechanical architecture of a projection device in an embodiment of the present application;

[0023] Figure 4 This is a schematic diagram of the optical path of the projection device in the embodiment of the present application;

[0024] Figure 5 This is a schematic diagram of the lens structure of the projection device in the embodiment of the present application;

[0025] Figure 6 This is a schematic diagram of the distance sensor and camera structure in an embodiment of the present application;

[0026] Figure 7 This is a schematic diagram of the system framework of the projection device in the embodiment of the present application;

[0027] Figure 8 This is a schematic diagram of the process of projection obstacle avoidance in an embodiment of the present application;

[0028] Fig. 9Schematic diagram of the process of the projection obstacle avoidance method in the embodiment of the present application;

[0029] Fig.10 A schematic diagram of a sampling image in an embodiment of the present application;

[0030] Fig.11 This is a schematic diagram of the white card area in the embodiment of the present application;

[0031] Fig.12 This is a schematic diagram of a projection area in an embodiment of the present application;

[0032] Fig.13 This is a schematic diagram of a process for calculating a target projection area according to a first transformation matrix and a second transformation matrix in an embodiment of the present application;

[0033] Fig.14 Schematic diagram of the target projection area in the embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make the purpose and implementation method of the present application clearer, the exemplary implementation method of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0035] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0036] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0037] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0038] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0039] The embodiments of the present application can be applied to various types of projection devices. The following will take a projector as an example to explain the projection device and the automatic focusing method.

[0040] A projector is a device that can project images or videos onto a screen. It can be connected to computers, broadcasting networks, the Internet, VCD (Video Compact Disc), DVD (Digital Versatile Disc Recordable), game consoles, DV, etc. through different interfaces to play corresponding video signals. Projectors are widely used in homes, offices, schools, and entertainment venues.

[0041] Figure 1 The figure shows a schematic diagram of the placement of a projection device according to an embodiment of the present application. Figure 2 A schematic diagram of the optical path of a projection device according to an embodiment of the present application is shown.

[0042] In some embodiments, reference Figure 1-2 A projection device provided in the present application includes a projection screen 1 and a projection device 2. The projection screen 1 is fixed at a first position, and the projection device 2 is placed at a second position, so that the picture projected by it coincides with the projection screen 1. The projection device includes a laser light source 100, an optical machine 200, a lens 300, and a projection medium 400. Among them, the laser light source 100 provides illumination for the optical machine 200, the optical machine 200 modulates the light source beam, and outputs it to the lens 300 for imaging, and projects it to the projection medium 400 to form a projection picture. Since the laser light source 100, the optical machine 200, and the lens 300 are used together to emit projection light to project a projection picture, in some embodiments of the present application, the laser light source 100, the optical machine 200, and the lens 300 are collectively referred to as a light output component.

[0043] In some embodiments, the laser light source 100 of the projection device includes a laser assembly 110 and an optical lens assembly 120. The light beam emitted by the laser assembly 110 can pass through the optical lens assembly 120 to provide lighting for the optical machine. For example, the optical lens assembly 120 requires a higher level of environmental cleanliness and airtight sealing; while the chamber where the laser assembly is installed can be sealed with a lower level of dustproof sealing to reduce the sealing cost.

[0044] In some embodiments, the optical engine 200 of the projection device may be implemented as a blue optical engine, a green optical engine, a red optical engine, and may also include a heat dissipation system, a circuit control system, etc. It should be noted that in some embodiments, the light emitting component of the projector may also be implemented by an LED light source.

[0045] Figure 3A schematic diagram of the circuit architecture of a projection device according to an embodiment of the present application is shown. In some embodiments, the projection device may include a display control circuit 10, a laser light source 20, at least one laser driving component 30, and at least one brightness sensor 40. The laser light source 20 may include at least one laser corresponding to the at least one laser driving component 30. The at least one refers to one or more, and the multiple refers to two or more.

[0046] Based on the circuit architecture, the projection device can achieve adaptive adjustment. For example, by setting a brightness sensor 40 in the light output path of the laser light source 20, the brightness sensor 40 can detect a first brightness value of the laser light source and send the first brightness value to the display control circuit 10.

[0047] The display control circuit 10 can obtain the second brightness value corresponding to the driving current of each laser, and when it is determined that the difference between the second brightness value of the laser and the first brightness value of the laser is greater than the difference threshold, it is determined that a COD fault occurs in the laser; the display control circuit can adjust the current control signal of the laser driving component corresponding to the laser until the difference is less than or equal to the difference threshold, thereby eliminating the COD fault of the blue laser; the projection device can eliminate the COD fault of the laser in time, reduce the damage rate of the laser, and improve the image display effect of the projection device.

[0048] Figure 4 A schematic structural diagram of a projection device according to an embodiment of the present application is shown.

[0049] In some embodiments, the laser light source 20 in the projection device may include independently arranged blue laser 201, red laser 202 and green laser 203. The projection device may also be called a three-color projection device. The blue laser 201, the red laser 202 and the green laser 203 are all modular lightweight (Mirai Console Loader, MCL) packaged lasers, which are small in size and conducive to the compact arrangement of the optical path.

[0050] In some embodiments, the controller includes at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), RAM (Random Access Memory, RAM), ROM (Read-Only Memory, ROM), a first interface to an nth interface for input / output, a communication bus (Bus), etc.

[0051] In some embodiments, the projection device may be configured with a camera for cooperating with the projection device to achieve adjustment and control of the projection process. For example, the camera configured with the projection device may be specifically implemented as a 3D camera or a binocular camera; when the camera is implemented as a binocular camera, it specifically includes a left camera and a right camera; the binocular camera may obtain the image and playback content presented by the screen corresponding to the projection device, that is, the projection surface, and the image or playback content is projected by the built-in optical machine of the projection device.

[0052] When the projection device moves, its projection angle and distance to the projection surface change, which will cause the projected image to deform. The projected image will appear as a trapezoidal image or other deformed image. The projection device controller can achieve automatic trapezoidal correction based on the image taken by the camera by coupling the angle between the optical and mechanical projection surfaces and the correct display of the projection image.

[0053] The camera can be used to capture the image displayed on the projection surface, and can be a camera. The camera can include a lens assembly, in which a photosensitive element and a lens are provided. The lens refracts light through a plurality of lenses, so that the light of the image of the scene can be irradiated on the photosensitive element. The photosensitive element can be selected based on the detection principle of a charge coupled device or a complementary metal oxide semiconductor according to the specifications of the camera, and converts the light signal into an electrical signal through a photosensitive material, and outputs the converted electrical signal into image data.

[0054] Figure 5 Schematic diagram of the lens structure of the projection device 2 in some embodiments is shown. In order to support the automatic focusing process of the projection device 2, as shown in FIG. Figure 5 As shown, the lens 300 of the projection device 2 may further include an optical component 310 and a drive motor 320. The optical component 310 is a lens group composed of one or more lenses, which can refract the light emitted by the optical machine 200 so that the light emitted by the optical machine 200 can be transmitted to the projection surface to form a transmission content image.

[0055] The optical assembly 310 may include a lens barrel and a plurality of lenses disposed in the lens barrel. Depending on whether the position of the lens can be moved, the lens in the optical assembly 310 may be divided into a movable lens 311 and a fixed lens 312. By changing the position of the movable lens 311 and adjusting the distance between the movable lens 311 and the fixed lens 312, the overall focal length of the optical assembly 310 may be changed. Therefore, the driving motor 320 may drive the movable lens 311 to move its position by connecting to the movable lens 311 in the optical assembly 310, thereby realizing an automatic focusing function.

[0056] It should be noted that the focusing process described in some embodiments of the present application refers to changing the position of the movable lens 311 by driving the motor 320, thereby adjusting the distance between the movable lens 311 and the fixed lens 312, that is, adjusting the image plane position. Therefore, the imaging principle of the lens combination in the optical component 310, the adjustment of the focal length is actually adjusting the image distance, but in terms of the overall structure of the optical component 310, adjusting the position of the movable lens 311 is equivalent to adjusting the overall focal length of the optical component 310.

[0057] When the distance between the projection device 2 and the projection surface is different, the lens of the projection device 2 needs to adjust different focal lengths to project a clear image on the projection surface. During the projection process, the distance between the projection device 2 and the projection surface will require different focal lengths due to the different placement positions of the user. Therefore, in order to adapt to different usage scenarios, the projection device 2 needs to adjust the focal length of the optical component 310.

[0058] Figure 6 Schematic diagram of distance sensor and camera structure in some embodiments is shown. Figure 6 As shown, the projection device 2 may also have a built-in or external camera 700, and the camera 700 may capture the image projected by the projection device 2 to obtain the projection content image. The projection device 2 then performs a clarity detection on the projection content image to determine whether the current lens focal length is appropriate, and adjusts the focal length if it is not appropriate. When automatically focusing based on the projection content image captured by the camera 700, the projection device 2 may continuously adjust the lens position and take pictures, and find the focus position by comparing the clarity of the front and back position pictures, thereby adjusting the movable lens 311 in the optical component to a suitable position. For example, the controller 500 may first control the drive motor 320 to gradually move the focus starting position of the movable lens 311 to the focus end position, and continuously obtain the projection content image through the camera 700 during this period. Then, by performing a clarity detection on multiple projection content images, the position with the highest clarity is determined, and finally the drive motor 320 is controlled to adjust the movable lens 311 from the focus terminal to the position with the highest clarity to complete the automatic focusing.

[0059] Figure 7 A schematic diagram of a system framework for realizing display control of a projection device according to an embodiment of the present application is shown.

[0060] In some embodiments, the projection device 2 has the characteristics of a telephoto micro-projector, and its controller can control the display of the projection light image through a preset algorithm to achieve functions such as automatic trapezoidal correction of the display screen, automatic screen entry, automatic obstacle avoidance, automatic focusing, and anti-eye-shooting.

[0061] In some embodiments, the projection device 2 is configured with a gyroscope sensor; when the device is moving, the gyroscope sensor can sense the position movement and actively collect movement data; then the collected data is sent to the application service layer through the system framework layer to support the application data required during the user interface interaction and application interaction process. The collected data can also be used for data calls by the controller in the algorithm service implementation.

[0062] In some embodiments, the projection device 2 is equipped with a time-of-flight (TOF) sensor, which is a sensor that uses the principle of time-of-flight to measure the depth of an object. The distance between the target object and the sensor is measured by sending and receiving infrared light, thereby obtaining the depth information of the target object. After the time-of-flight sensor collects the corresponding data, the data will be sent to the corresponding time-of-flight service of the service layer; after the above-mentioned time-of-flight service obtains the data, the collected data will be sent to the application service layer through the process communication framework, and the data will be used for interactive use such as controller data calls, user interfaces, and program applications.

[0063] There are many types of time-of-flight sensors, such as single-point sensors, multi-area TOF sensors, and 3D TOF sensors. All of the above sensors use the flight time of light to calculate the distance between the object and the sensor. That is, the distance is calculated by measuring the time difference between the light being emitted and the light being reflected back and received by the receiver.

[0064] Among them, the single-point sensor has a receiver and a transmitter, which can perform single-point measurement. The multi-area TOF sensor can measure multiple points at the same time by dividing the entire field of view into multiple small areas and performing independent ranging for each area. Each area has a transmitter and a receiver, which can measure the distance independently. The 3DTOF sensor can achieve depth reconstruction of the three-dimensional scene by independently measuring the distance for each pixel. Compared with the multi-area TOF sensor, the 3DTOF sensor does not need to divide the field of view into multiple small areas, but directly measures the depth of the entire field of view.

[0065] In some embodiments, the camera 700 configured for the projection device 2 may be a binocular camera, a depth camera, or a 3D camera, etc.; the data collected by the camera 700 will be sent to the camera service, and then the camera service will send the collected image data to the process communication framework and / or the projection device correction service; the projection device correction service can receive the camera collection data sent by the camera service, and the controller can call the corresponding control algorithm in the algorithm library according to the different functions to be implemented.

[0066] In some embodiments, data is interacted with the application service through the process communication framework, and then the calculation results are fed back to the correction service through the process communication framework; the correction service sends the obtained calculation results to the projection device 2 operating system to generate control signals, and sends the control signals to the optical machine 200 control drive to control the working condition of the optical machine 200 and realize automatic correction of the displayed image.

[0067] In some embodiments, when an image correction instruction is detected, the projection device 2 can correct the projected image. For the correction of the projected image, an association relationship between the distance, the horizontal angle, and the offset angle can be created in advance. Then the controller in the projection device 2 obtains the current distance from the optical machine 200 to the projection surface, and determines the angle between the optical machine 200 and the projection surface at this moment in combination with the associated relationship to achieve projection image correction. The angle is specifically implemented as the angle between the central axis of the optical machine 200 and the projection surface.

[0068] In some embodiments, the projection device 2 automatically completes the calibration and refocuses, and the controller will detect whether the automatic focus function is turned on; when the automatic focus function is not turned on, the controller will end the automatic focus service; when the automatic focus function is turned on, the projection device 2 will obtain the detection distance of the time-of-flight sensor through the middleware for calculation.

[0069] The controller queries the preset mapping table according to the acquired distance to obtain the focal length of the projection device 2; then the middleware sets the acquired focal length to the optical machine 200 of the projection device 2; wherein the middleware is a series of applications for the focus control process. After the optical machine 200 emits laser light at the above focal length, the camera will execute the photo command; the controller determines whether the focus process of the projection device 2 is completed according to the acquired captured image and the evaluation function.

[0070] If the judgment result meets the preset completion conditions, the control of the automatic focusing process ends; if the judgment result does not meet the preset completion conditions, the middleware will fine-tune the focal length parameters of the projection device 2 optical machine 200, for example, the focal length can be gradually fine-tuned with a preset step length, and the adjusted focal length parameters will be set to the optical machine 200 again; thereby achieving repeated photo taking and clarity evaluation steps, and finally finding the optimal focal length through clarity comparison to complete automatic focusing.

[0071] In some embodiments, when the user turns on the projection device 2, the projection device 2 can project the content preset by the user onto a projection surface, which can be a wall or a screen, and the projection image can be displayed on the projection surface for the user to watch.

[0072] In some embodiments, the projection device 2 supports an automatic obstacle avoidance function. When projecting media data onto the projection surface, the projection device 2 can detect obstacles on the projection surface. When an obstacle is detected on the projection surface, the projection surface is re-divided into a target projection area where no obstacles exist, and the media data is projected onto the target projection area.

[0073] When the projection device 2 performs obstacle avoidance, the camera 700 captures an image of the projection surface 400 to determine the position of the obstacle. Then the light emitting component automatically avoids the obstacle based on the position of the obstacle and the projection area required by the projected media data. Among them, in the image captured by the camera 700, the obstacle is in the camera coordinate system, and its coordinates are confirmed according to the camera coordinate system. When the projection device 2 projects media data onto the projection surface 400, the media data is projected through the light emitting component, and the coordinates of the distribution of the media data on the projection surface 400 are confirmed according to the light emitting component coordinate system.

[0074] Therefore, the calculation of the target projection area also needs to involve the coordinate conversion between the camera coordinate system and the light output component coordinate system, wherein the calculation basis of the coordinate conversion relies on the homography matrix between the camera 700 and the light output component.

[0075] For example, Figure 8 As shown, it is a schematic diagram of the process of projection obstacle avoidance provided by an embodiment of the present application. The projection device 2 projects a pure color card (such as a white card) onto the projection surface 400 through the light emitting component, and controls the camera 700 to take a photo of the pure color card displayed on the projection surface when the light emitting component projects the pure color card, so as to obtain a first sampling image. Then, a feature card (such as a checkerboard picture) is projected onto the projection surface 400 through the light emitting component, and controls the camera 700 to take a photo of the pure color card displayed on the projection surface when the light emitting component projects the feature card, so as to obtain a second sampling image.

[0076] The projection device 2 calculates the transformation relationship Hwp between the coordinate system of the light-emitting component and the world coordinate system of the projection surface under the current projection state based on the coordinates of the corner points of the feature map in the second sampling image taken by the camera 700, the coordinates of the corner points of the projected feature map under the light-emitting component, the internal parameters calibrated by the camera 700, and the external parameters of the camera 700 and the light-emitting component (rotation matrix R and translation vector T).

[0077] And according to the corner coordinates of the feature map card in the second sampled image taken by the camera 700 in the camera coordinate system and the corner coordinates of the feature map card in the light output component coordinate system, the homography matrix Hpc between the camera coordinate system and the light output component coordinate system is calculated.

[0078] Then, according to the first sampling image taken by the camera 700, the largest projectable area in the projection area is calculated, and the projectable area is converted from the camera coordinate system to the light emitting component coordinate system through the homography matrix Hpc, and then the light emitting component coordinate system is converted to the world coordinate system through the conversion relationship Hwp, and the largest inscribed rectangular area is selected in the projectable area in the world coordinate system to obtain the four corner point coordinates Wi (i=1~3) of the largest inscribed rectangular area in the world coordinate system. Finally, through the conversion relationship Hwp, Wi is converted from the world coordinate system to the light emitting component coordinate system to obtain the four corner point coordinates Pi of the target projection area in the light emitting component coordinate system, and Pi is passed to the light emitting component so that the light emitting component projects the media data according to Pi to achieve projection obstacle avoidance.

[0079] It can be seen that in the above-mentioned implementation, the obstacle avoidance result depends on the calibration data between the camera 700 and the light-emitting component. The calibration requires data such as the camera internal parameters, the optical machine internal parameters, and the external parameters of the camera and the optical machine, and the calibration is complicated. When there is an error in a certain data, it will affect the obstacle avoidance result. In addition, when calibrating between the camera 700 and the light-emitting component, the light-emitting component needs to project a pure color image card and a feature image card in sequence. The light-emitting component needs a certain conversion time when switching between projecting media data, projecting a pure color image card, and projecting a feature image card. The camera 700 also needs a certain shooting time to shoot the image card, which makes the calibration operation between the camera 700 and the light-emitting component complicated and time-consuming, reducing the efficiency of automatic obstacle avoidance.

[0080] To this end, some embodiments of the present application provide a projection obstacle avoidance method, which can be applied to the projection device 2 provided in the above embodiment. In order to meet the implementation of the projection obstacle avoidance method, the projection device 2 should at least include a light emitting component, a camera 700, a distance sensor and a controller. Among them, the light emitting component is configured to project the projection content onto the projection surface, and the projection surface can be a wall or a curtain. The camera 700 is configured to capture a sampled image. The distance sensor is configured to collect depth information between the distance sensor and the projection surface, and can be a multi-area TOF sensor or a 3DTOF sensor. The controller is configured to execute the program steps corresponding to the projection obstacle avoidance method, such as Fig. 9 As shown, it is a flow chart of the projection obstacle avoidance method provided in the embodiment of the present application, which specifically includes the following contents:

[0081] S100, acquiring a sample image in response to an obstacle avoidance instruction.

[0082] The projection device 2 can automatically avoid obstacles in the projection surface 400 in response to the obstacle avoidance instruction. The obstacle avoidance instruction refers to a control instruction used to trigger the projection device 2 to automatically perform the obstacle avoidance process. The obstacle avoidance instruction can be automatically triggered and generated. For example, the projection device 2 receives a projection instruction, and in response to the projection instruction, triggers the generation of an obstacle avoidance instruction to turn on the automatic obstacle avoidance function. That is, before projecting the projection image onto the projection surface 400, the projection device 2 can automatically detect obstacles on the projection surface. The obstacle avoidance instruction can also be actively input by the user. For example, the user can press the automatic obstacle avoidance switch pre-set in the projection device 2, or the automatic obstacle avoidance button on the remote control of the projection device 2 to input the obstacle avoidance instruction so that the projection device 2 starts the automatic obstacle avoidance function.

[0083] In response to the obstacle avoidance instruction, the projection device 2 controls the light emitting component to project a preset pure color card to the projection surface 400, and controls the camera 700 to take a picture of the pure color card displayed on the projection surface 400 when the light emitting component projects the pure color card to obtain a sampled image.

[0084] Projecting a pure color card means projecting a pure color projection screen. When the light emitting component projects the pure color card onto the projection surface 400, part of the light in the pure color card will illuminate the obstacle between the light emitting component and the projection surface 400. A shadow area formed by the occlusion of the obstacle will appear at the corresponding position of the projection surface 400. The controller of the projection device 2 can identify the shadow area in the pure color card in the captured sample image to identify the position of the obstacle, and perform subsequent obstacle avoidance functions according to the position of the obstacle.

[0085] In order to facilitate the subsequent calculation of the target projection area that does not include obstacles, the size of the pure color image card projected by the light emitting component should be the maximum projection size of the light emitting component, that is, the projection device 2 controls the light emitting component to project the pure color image card with the maximum projection size, and after projecting the pure color image card, controls the camera 700 to take a picture of the pure color image card displayed on the projection surface 400, so that the pure color image card area in the sampled image obtained is the maximum projection area under the current posture of the projection device 2.

[0086] It should be noted that in order to more clearly identify the location of obstacles, the solid color picture card should be a light color picture card, such as light yellow, light blue, white, gray, etc. The embodiment of the present application does not impose any specific restrictions on the color of the solid color picture card.

[0087] S200: extracting a projectable area from the sampled image.

[0088] After acquiring the sampled image, the projection device 2 can perform obstacle detection on the sampled image to identify the location of the obstacle. Since the image area of ​​the projection surface image captured by the camera 700 is larger than the image area of ​​the projection area (pure color card area), in order to reduce the resource consumption of obstacle detection in the non-projection area, before performing obstacle detection on the sampled image, the projection device 2 can first extract the card area where the pure color card is located in the sampled image, and then perform obstacle detection on the card area to identify the obstacles in the card area. Then, according to the identified obstacle position, the area that does not contain obstacles is extracted in the card area to obtain the projectable area.

[0089] For example, in response to the obstacle avoidance instruction, the projection device 2 controls the light emitting component to project a white card onto the projection surface 400, and controls the camera 700 to take a photo of the pure white card displayed on the projection surface 400, so as to obtain Fig.10 After acquiring the sample image, the projection device 2 extracts the image card area where the white image card is located in the sample image, that is, Fig.11 The white card area (excluding the shaded area) is shown in the figure. Then, obstacle detection is performed on the white card area, and the largest blank area that does not contain obstacles is extracted from the card area according to the obstacle detection result, as shown in the figure. Fig.12 The projectable area shown (the area within the dotted box).

[0090] S300: Obtain a first transformation matrix and a second transformation matrix.

[0091] The first transformation matrix is ​​a transformation matrix between the light output component coordinate system and the projection surface world coordinate system, and the second transformation matrix is ​​a transformation matrix between the camera coordinate system and the projection surface world coordinate system. The first transformation matrix and the second transformation matrix are calculated based on the depth information collected by the time-of-flight sensor.

[0092] The first transformation matrix can be calculated based on the depth information collected by the distance sensor, the projection parameters (internal parameters) of the light-emitting component, and the posture parameters (external parameters) between the distance sensor and the light-emitting component. The projection device 2 can obtain the depth information collected by the distance sensor, construct point cloud data based on the depth information, and calculate the first world coordinate system, wherein the first world coordinate system is a world coordinate system with the optical center of the distance sensor as the origin.

[0093] Then, the posture parameters between the light emitting component and the distance sensor are obtained, wherein the posture parameters include the rotation matrix R and the translation vector T. According to the posture parameters and the first world coordinate system, the second world coordinate system is calculated, wherein the second world coordinate system is a world coordinate system with the light emitting component as the origin. Then, according to the second world coordinate system, the projection surface 400 is fitted to obtain the normal vector of the projection surface 400. The projection parameters of the light emitting component are obtained, and the first transformation matrix is ​​calculated according to the normal vector and the projection parameters.

[0094] For example, Fig.13 As shown, the time-of-flight sensor is a multi-region TOF sensor, which divides the projection area of ​​the projection device 2 in the current projection state into multiple areas through the multi-region TOF sensor. For example, the multi-region TOF sensor can arrange the infrared light source into a 4x4 or 8x8 grid shape, and measure the time from emission to reflection of each grid, so as to obtain the depth information of multiple areas (4x4 area or 8x8 area). The depth information of the multiple areas obtained is converted into a point cloud Fi (Xti, Yti, Zti), that is, the world coordinates of the multiple areas in the projection area in the current projection state with the optical center of the multi-region TOF sensor as the origin are obtained.

[0095] Then, according to the pose parameter R between the light-emitting component and the multi-region TOF sensor tp , calculate the world coordinates Pi (Xpi, Ypi, Zpi) of multiple areas in the projection area with the optical center of the optical machine 200 as the origin = R tp *Fi(Xti, Yti, Zti).

[0096] Then fit the projection surface 400 and calculate the normal vector n of the projection surface 400. And according to the optical machine internal parameter M p , calculate the conversion relationship H between the optical-mechanical coordinate system and the projection surface world coordinate system wP , which is the first transformation matrix.

[0097] For the second transformation matrix, the standard projection area in the world coordinate system of the projection surface can be calculated according to the first transformation matrix, wherein the standard projection area is the projection area of ​​the maximum size projected by the preset light-emitting component. Then the image card area in the camera coordinate system is calculated, wherein the image card area is the area where the pure color image card in the sampled image is located. Finally, the second transformation matrix is ​​calculated according to the standard projection area in the world coordinate system of the projection surface and the image card area in the camera coordinate system.

[0098] The second transformation matrix can be calculated according to the corner point coordinates of the standard projection area and the corner point coordinates of the chart area. The projection device 2 can obtain the projection corner point coordinates of the standard projection area and the chart area corner point coordinates, wherein the projection corner point coordinates are the world coordinates of the corner point of the standard projection area, and the chart area corner point coordinates are the pixel coordinates of the corner point of the chart area.

[0099] The projection plane 400 is taken as a plane, and the vertical axis coordinate value in the three-dimensional coordinates of the projection corner point coordinates is deleted to obtain the projection plane corner point coordinates. Finally, according to the projection plane corner point coordinates and the chart corner point coordinates, the homography matrix between the camera coordinate system and the projection plane world coordinate system is calculated to obtain the second transformation matrix.

[0100] For example, Fig.13 As shown, the above embodiment is used, according to the first transformation matrix H wP , calculate the world coordinates Wi(Xi, Yi, Zi) of the four corner points of the projection area with the maximum size under the current projection state of the optical component, take the projection surface 400 as a plane, delete the vertical axis coordinate value Zi in the world coordinates Wi(Xi, Yi, Zi) of the four corner points, and obtain the coordinates Wi(Xi, Yi) of the four corner points of the projection area on the projection surface.

[0101] Then calculate the pixel coordinates Ci(Xi, Yi) of the four corner points of the white card area photographed by camera 400 in the sampled image, and finally calculate the homography matrix H between the projection surface and the camera image coordinates based on the coordinate point pairs Wi(Xi, Yi) and Ci(Xi, Yi) wC , which is the second transformation matrix.

[0102] It can be understood that in the above example, the optical machine internal parameter is M p , the position of the optical machine 200 and the multi-region TOF sensor is: R tp , that is, the world coordinates of the midpoint O in the multi-region TOF sensor coordinate system are F(Xt, Yt, Zt), and the world coordinates in the optical machine coordinate system are P(Xp, Yp, Zp), and the conversion relationship satisfies F T =R tp *p T 、p T =R p t*FT.

[0103] The conversion relationship between the image coordinates of the optical machine 200 and the world coordinates is: wP , that is, the coordinates of the point O in the world coordinate system are W(Xt, Yt, Zt), and the coordinates in the optical image are P(Xp, Yp), and the transformation relationship satisfies W T =H wp *P T 、p T =H wP *W T .

[0104] The conversion relationship between the camera 700 image coordinates and the world coordinates is: wc, that is, the coordinates of the point O in the world coordinate system are W(Xt, Yt, Zt), and the coordinates in the camera image are C(Xc, Yc), and the transformation relationship satisfies W T =H wc *C T , C T =H wC *W T .

[0105] S400: Delimiting a target projection area in a projectable area in a world coordinate system of the projection surface according to a second transformation matrix.

[0106] The target projection area is an inscribed rectangular area with a maximum preset aspect ratio contained in the projectable area under the world coordinates of the projection surface. The projection device 2 can fit the area contour of the projectable area, and according to the contour ratio of the area contour and the second transformation matrix, transform the projectable area from the camera coordinate system to the world coordinate system of the projection surface to obtain the spatial projection area, and extract the inscribed rectangular area with a maximum preset aspect ratio in the spatial projection area to obtain the target projection area.

[0107] For example, following the above embodiment, the contour of the projectable area (such as Fig.12 The area outline box shown in the figure is converted according to the second transformation matrix H according to the outline ratio. wC The projectable area is transformed from the camera coordinate system to the projection surface world coordinate system, and the following is obtained: Fig.14 The spatial projection area (shadow area and white area) shown is Fig.12 The area outline of the projectable area shown corresponds to the area outline of the maximum projection area (the area within the dotted box) of the optical engine 200. Fig.14 Extract the inscribed rectangular area with the maximum preset aspect ratio from the spatial projection area shown in the figure ( Fig.14 White area) to obtain the target projection area.

[0108] S500: Calculate a target projection area in a light emitting component coordinate system according to a first transformation matrix, and control the light emitting component to project projection contents to the target projection area in the light emitting component coordinate system.

[0109] After the projection device 2 obtains the target projection area in the projection surface world coordinate system, it needs to convert the target projection area from the projection surface world coordinate system to the light emitting component coordinate system, and control the light emitting component to project the projection content to the target projection area in the light emitting component coordinate system, thereby achieving projection obstacle avoidance.

[0110] The projection device 2 can obtain the plane corner coordinates of the target projection area, add the vertical axis coordinate value in the three-dimensional coordinates to the plane corner coordinates to obtain the world corner coordinates. Then, according to the first transformation matrix, the world corner coordinates are converted from the projection surface world coordinate system to the light output component coordinate system to obtain the target projection area in the light output component coordinate system.

[0111] For example, Fig.13 As shown, the above embodiment is used, the coordinates of the four corner points of the target projection area are Wj (Xj, Yj), and the vertical axis coordinate value Zj is added to the four corner point coordinates Wj (Xj, Yj) to obtain the coordinates Wi (Xi, Yi, Zj) of the four corner points of the target projection area in the world coordinates of the projection surface. Then according to the first transformation matrix H wP , the coordinates of the four corner points of the target projection area are converted from the projection surface world coordinate system to the light output component coordinate system, and the coordinates of the four corner points of the target projection area in the light output component coordinate system are obtained as P(Xp, Yp, Zp)=H wP *Wj T Finally, the light-emitting component is controlled to project the media data according to the four corner point coordinates P (Xp, Yp, Zp) to achieve projection obstacle avoidance.

[0112] It can be understood that the embodiment of the present application only requires a pure color card image taken by the camera, the internal parameters of the optical machine, and the external parameters of the distance sensor and the optical machine to achieve obstacle avoidance. There is no need to shoot a feature card and calibrate the posture of the camera and the optical machine, which eliminates the influence of the posture error of the camera and the optical machine on obstacle avoidance. Compared with the external parameter calibration between the camera and the light output component, the external parameter calibration of the distance sensor and the optical machine is simple and time-saving, which shortens the obstacle avoidance time, improves the obstacle avoidance efficiency, and improves the user experience.

[0113] Based on the projection obstacle avoidance method provided in the above embodiments, some embodiments of the present application further provide a projection device 2, wherein the projection device 2 includes a light emitting component, a camera 700, a distance sensor, and a controller. The light emitting component is configured to project the projection content onto the projection surface. The camera 700 is configured to capture a sampled image. The distance sensor is configured to collect depth information between the distance sensor and the projection surface. The controller is configured to execute the following program steps:

[0114] In response to the obstacle avoidance instruction, a sample image is acquired.

[0115] The sampled image is an image captured by the camera when the light-emitting component projects a pure color picture card.

[0116] Extract the projectable area in the sampled image.

[0117] The projectable area is the area in the sampled image that does not contain obstacles.

[0118] Get the first transformation matrix and the second transformation matrix.

[0119] Among them, the first transformation matrix is ​​the transformation matrix between the light output component coordinate system and the projection surface world coordinate system, and the second transformation matrix is ​​the transformation matrix between the camera coordinate system and the projection surface world coordinate system. The first transformation matrix and the second transformation matrix are calculated based on the depth information collected by the distance sensor.

[0120] According to the second transformation matrix, a target projection area is defined in the projectable area in the world coordinate system of the projection surface.

[0121] The target projection area is an inscribed rectangular area with a maximum preset aspect ratio contained in the projectable area in the world coordinate system of the projection surface.

[0122] According to the first transformation matrix, a target projection area in the light emitting component coordinate system is calculated, and the light emitting component is controlled to project the projection content to the target projection area in the light emitting component coordinate system.

[0123] It can be seen from the above technical solutions that the projection device and projection obstacle avoidance method provided in the above embodiments can respond to obstacle avoidance instructions, obtain a sampled image taken by the camera when the light-emitting component projects a solid color picture card, extract the projectable area in the sampled image, and then obtain the first transformation matrix and the second transformation matrix, wherein the first transformation matrix is ​​the transformation matrix between the light-emitting component coordinate system and the projection surface world coordinate system, and the second transformation matrix is ​​the transformation matrix between the camera coordinate system and the projection surface world coordinate system. According to the second transformation matrix, the target projection area is delineated in the projectable area in the projection surface world coordinate system, and according to the first transformation matrix, the target projection area in the light-emitting component coordinate system is calculated, and the light-emitting component is controlled to project the projection content to the target projection area, thereby realizing projection obstacle avoidance. The method can eliminate the need for feature picture card shooting and camera and optical machine posture calibration, shorten obstacle avoidance time, and improve user experience.

[0124] The same and similar parts between the various embodiments in this specification can be referenced to each other and will not be described again here.

[0125] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of various embodiments of the present invention or certain parts of the embodiments.

[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0127] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

Claims

1. A projection device, It is characterized in that include: A light emitting component is configured to project projection content onto a projection surface; a camera configured to capture a sample image; a distance sensor configured to collect depth information between the distance sensor and the projection surface; The controller is configured as: In response to the obstacle avoidance instruction, a sample image is acquired, where the sample image is an image captured by the camera when the light emitting component projects a pure color picture card; Extracting a projectable area from the sampled image, where the projectable area is an area in the sampled image that does not contain obstacles; Obtain a first transformation matrix and a second transformation matrix, wherein the first transformation matrix is ​​a transformation matrix between the light output component coordinate system and the projection surface world coordinate system, and the second transformation matrix is ​​a transformation matrix between the camera coordinate system and the projection surface world coordinate system, and the first transformation matrix and the second transformation matrix are calculated based on the depth information collected by the distance sensor; According to the second transformation matrix, a target projection area is defined in the projectable area in the world coordinate system of the projection surface, wherein the target projection area is an inscribed rectangular area of ​​a maximum preset aspect ratio contained in the projectable area in the world coordinate system of the projection surface; According to the first transformation matrix, a target projection area in the light emitting component coordinate system is calculated, and the light emitting component is controlled to project the projection content to the target projection area in the light emitting component coordinate system.

2. The projection device according to claim 1, It is characterized in that The controller executes acquisition of a sampled image and is further configured to: In response to the obstacle avoidance instruction, controlling the light emitting component to project a preset pure color picture card; When the light emitting component projects a pure color picture card, the camera is controlled to take a picture of the pure color picture card displayed on the projection surface to obtain a sampled image.

3. The projection device according to claim 1, It is characterized in that The controller is configured to extract the projectable area from the sampled image and further: Extracting the image card area where the pure color image card is located in the sampled image; Performing obstacle detection on the image card area to identify obstacles in the image card area; An area without obstacles is extracted from the image card area to obtain a projectable area.

4. The projection device according to claim 1, It is characterized in that The controller is also configured to: Acquiring depth information collected by the distance sensor; Acquire projection parameters of the light emitting component and posture parameters between the light emitting component and the distance sensor, wherein the posture parameters include a rotation matrix and a translation vector; The first transformation matrix is ​​calculated according to the depth information, the posture parameters and the projection parameters.

5. The projection device according to claim 4, It is characterized in that The controller calculates the first transformation matrix according to the depth information, the posture parameter and the projection parameter, and is further configured to: constructing point cloud data based on the depth information to calculate a first world coordinate system, wherein the first world coordinate system is a world coordinate system with the optical center of the distance sensor as the origin; Calculate a second world coordinate system according to the posture parameter and the first world coordinate system, where the second world coordinate system is a world coordinate system with the light emitting component as the origin; Fitting the projection surface according to the second world coordinate system to obtain a normal vector of the projection surface; The first transformation matrix is ​​calculated according to the normal vector and the projection parameters.

6. The projection device according to claim 5, It is characterized in that The controller executes acquiring the second transformation matrix and is further configured to: Calculate the standard projection area of ​​the projection surface in the world coordinate system according to the first transformation matrix, where the standard projection area is the maximum size projection area of ​​the light output component; Calculating a card area in the camera coordinate system, where the card area is a region where the pure color card is located in the sampled image; The second transformation matrix is ​​calculated according to the standard projection area in the projection surface world coordinate system and the image card area in the camera coordinate system.

7. The projection device according to claim 6, It is characterized in that The controller calculates the second transformation matrix according to the standard projection area in the projection surface world coordinate system and the image card area in the camera coordinate system, and is further configured as follows: Acquire the projection corner point coordinates of the standard projection area, where the projection corner point coordinates are the world coordinates of the corner point of the standard projection area; Deleting the vertical axis coordinate value in the three-dimensional coordinates of the projection corner point coordinates to obtain the projection plane corner point coordinates; Obtaining coordinates of a corner point of the card area, where the coordinates of the corner point of the card area are pixel coordinates of the corner point of the card area; According to the projection plane corner point coordinates and the chart card corner point coordinates, a homography matrix between the camera coordinate system and the projection surface world coordinate system is calculated to obtain the second transformation matrix.

8. The projection device according to claim 1, It is characterized in that The controller is configured to define a target projection area in the projectable area in the projection surface world coordinate system according to the second transformation matrix, and is further configured to: Fitting the area contour of the projectable area; According to the contour ratio of the area contour and the second transformation matrix, the projectable area is transformed from the camera coordinate system to the projection surface world coordinate system to obtain a spatial projection area; An inscribed rectangular area with a maximum preset aspect ratio is extracted from the spatial projection area to obtain a target projection area.

9. The projection device according to claim 8, It is characterized in that The controller calculates the target projection area in the light output component coordinate system according to the first transformation matrix, and is further configured as follows: Obtaining the coordinates of the plane corner points of the target projection area; Adding the vertical axis coordinate value in the three-dimensional coordinates to the plane corner point coordinates to obtain the world corner point coordinates; According to the first transformation matrix, the world corner point coordinates are transformed from the projection surface world coordinate system to the light output component coordinate system to obtain a target projection area in the light output component coordinate system.

10. A projection obstacle avoidance method, It is characterized in that Applied to a projection device, the projection device comprises a light emitting component, a camera, a distance sensor and a controller; the light emitting component is configured to project projection content onto a projection surface; the camera is configured to capture a sample image; The distance sensor is configured to collect depth information between the distance sensor and the projection surface; The projection obstacle avoidance method comprises: In response to the obstacle avoidance instruction, a sample image is acquired, where the sample image is an image captured by the camera when the light emitting component projects a pure color picture card; Extracting a projectable area from the sampled image, where the projectable area is an area in the sampled image that does not contain obstacles; Obtain a first transformation matrix and a second transformation matrix, wherein the first transformation matrix is ​​a transformation matrix between the light output component coordinate system and the projection surface world coordinate system, and the second transformation matrix is ​​a transformation matrix between the camera coordinate system and the projection surface world coordinate system, and the first transformation matrix and the second transformation matrix are calculated based on the depth information collected by the distance sensor; According to the second transformation matrix, a target projection area is defined in the projectable area in the world coordinate system of the projection surface, wherein the target projection area is an inscribed rectangular area of ​​a maximum preset aspect ratio contained in the projectable area in the world coordinate system of the projection surface; According to the first transformation matrix, a target projection area in the light emitting component coordinate system is calculated, and the light emitting component is controlled to project the projection content to the target projection area in the light emitting component coordinate system.

Images