Projection device and projection obstacle avoidance method

Abstract

This application provides a projection device and a projection obstacle avoidance method. The method, in response to an obstacle avoidance command, acquires a sampled image captured by a camera when the light-emitting component projects a solid-color image card. It then extracts a projectable region from the sampled image and acquires a first transformation matrix and a second transformation matrix. The first transformation matrix is ​​a transformation matrix between the coordinate system of the light-emitting component and the world coordinate system of the projection surface, and the second transformation matrix is ​​a transformation matrix between the camera coordinate system and the world coordinate system of the projection surface. Based on the second transformation matrix, a target projection region is defined within the projectable region in the world coordinate system of the projection surface. Based on the first transformation matrix, the target projection region in the coordinate system of the light-emitting component is calculated. The light-emitting component is then controlled to project the content onto this target projection region, thereby achieving projection obstacle avoidance. This method eliminates the need for feature card capture and camera and optical engine pose calibration, shortening obstacle avoidance time and improving user experience.

Description

Technical Field

[0001] This application relates to the field of projection equipment technology, and in particular to a projection device and a projection obstacle avoidance method. Background Technology

[0002] A projection device is a display device that projects images or videos onto a screen. It uses optical lenses to refract laser light of a specific color onto a projection area, forming a concrete image. Due to its portability, the user can move the projection device during the projection process to project the image or video in different directions.

[0003] When projecting media data, a wall or screen can be considered the projection area. If an obstacle appears in the projection area, the projected media data will cause the image to flicker or become incomplete. Therefore, to improve the user viewing experience, the projector detects obstacles in the projection area and performs automatic obstacle avoidance to redefine the projection area.

[0004] However, when projection devices perform obstacle avoidance, they rely on calibration data between the camera and the optical engine. Errors in the camera's intrinsic parameters can affect the obstacle avoidance results. Furthermore, the pose calibration between the camera and the optical engine requires the optical engine to project white maps and checkerboard maps sequentially, which makes the calibration time-consuming and reduces the efficiency of automatic obstacle avoidance. Summary of the Invention

[0005] This application provides a projection device and a projection obstacle avoidance method to solve the problem of low efficiency in automatic obstacle avoidance.

[0006] In a first aspect, this application provides a projection device, including a light-emitting component, a camera, a distance sensor, and a controller, wherein the light-emitting component is configured to project projection content onto a projection surface; the camera is configured to capture a sampled image; the distance sensor is configured to acquire depth information between the distance sensor and the projection surface; and the controller is configured to execute the following program steps:

[0007] In response to an obstacle avoidance command, a sampled image is acquired, wherein the sampled image is an image captured by the camera when the light-emitting component projects a solid color image card;

[0008] Extract a projectable region from the sampled image, wherein the projectable region is the area in the sampled image that does not contain obstacles;

[0009] Obtain a first transformation matrix and a second transformation matrix. The first transformation matrix is ​​the transformation matrix between the coordinate system of the light-emitting component and the world coordinate system of the projection surface. The second transformation matrix is ​​the transformation matrix between the camera coordinate system and the world coordinate system of the projection surface. The first transformation matrix and the second transformation matrix are calculated based on the depth information collected by the distance sensor.

[0010] According to the second transformation matrix, a target projection region is defined in the projectable region under the projection plane world coordinate system. The target projection region is the inscribed rectangular region with the largest preset aspect ratio that can be accommodated in the projectable region under the projection plane world coordinate system.

[0011] Based on the first transformation matrix, the target projection area in the coordinate system of the light-emitting component is calculated, and the light-emitting component is controlled to project the projection content onto the target projection area in the coordinate system of the light-emitting component.

[0012] Secondly, this application also provides a projection obstacle avoidance method applied to the above-mentioned projection device, the method comprising:

[0013] In response to an obstacle avoidance command, a sampled image is acquired, wherein the sampled image is an image captured by the camera when the light-emitting component projects a solid color image card;

[0014] Extract a projectable region from the sampled image, wherein the projectable region is the area in the sampled image that does not contain obstacles;

[0015] Obtain a first transformation matrix and a second transformation matrix. The first transformation matrix is ​​the transformation matrix between the coordinate system of the light-emitting component and the world coordinate system of the projection surface. The second transformation matrix is ​​the transformation matrix between the camera coordinate system and the world coordinate system of the projection surface. The first transformation matrix and the second transformation matrix are calculated based on the depth information collected by the distance sensor.

[0016] According to the second transformation matrix, a target projection region is defined in the projectable region under the projection plane world coordinate system. The target projection region is the inscribed rectangular region with the largest preset aspect ratio that can be accommodated in the projectable region under the projection plane world coordinate system.

[0017] Based on the first transformation matrix, the target projection area in the coordinate system of the light-emitting component is calculated, and the light-emitting component is controlled to project the projection content onto the target projection area in the coordinate system of the light-emitting component.

[0018] As can be seen from the above technical solutions, this application provides a projection device and a projection obstacle avoidance method. The method, in response to an obstacle avoidance command, acquires a sampled image captured by a camera when the light-emitting component projects a solid-color image card. It then extracts a projectable area from the sampled image and acquires a first transformation matrix and a second transformation matrix. The first transformation matrix is ​​the transformation matrix between the coordinate system of the light-emitting component and the world coordinate system of the projection surface, and the second transformation matrix is ​​the transformation matrix between the camera coordinate system and the world coordinate system of the projection surface. Based on the second transformation matrix, a target projection area is defined within the projectable area in the world coordinate system of the projection surface. Based on the first transformation matrix, the target projection area in the coordinate system of the light-emitting component is calculated. The light-emitting component is then controlled to project the content onto the target projection area, thereby achieving projection obstacle avoidance. This method eliminates the need for feature card capture and camera and optical engine pose calibration, shortening obstacle avoidance time and improving user experience. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of the projection state of the projection device in an embodiment of this application;

[0021] Figure 2 This is a schematic diagram of the projection device structure in an embodiment of this application;

[0022] Figure 3 This is a schematic diagram of the optical engine architecture of the projection device in the embodiments of this application;

[0023] Figure 4 This is a schematic diagram of the optical path of the projection device in the embodiments of this application;

[0024] Figure 5 This is a schematic diagram of the lens structure of the projection device in the embodiments of this application;

[0025] Figure 6 This is a schematic diagram of the distance sensor and camera structure in the embodiments of this application;

[0026] Figure 7 This is a schematic diagram of the system framework of the projection device in the embodiments of this application;

[0027] Figure 8 This is a schematic diagram of the projection obstacle avoidance process in an embodiment of this application;

[0028] Figure 9This is a flowchart illustrating the projection obstacle avoidance method in an embodiment of this application;

[0029] Figure 10 This is a schematic diagram of the sampled images in the embodiments of this application;

[0030] Figure 11 This is a schematic diagram of the white card area in the embodiments of this application;

[0031] Figure 12 This is a schematic diagram of the projectable area in an embodiment of this application;

[0032] Figure 13 This is a schematic diagram of the process for calculating the target projection region based on the first transformation matrix and the second transformation matrix in an embodiment of this application;

[0033] Figure 14 This is a schematic diagram of the target projection area in an embodiment of this application. Detailed Implementation

[0034] To make the objectives and implementation methods of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the exemplary embodiments described are only some embodiments of this application, and not all embodiments.

[0035] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0036] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0037] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0038] The term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0039] The embodiments of this application can be applied to various types of projection devices. The following description will use a projector as an example to illustrate the projection device and the automatic focusing method.

[0040] A projector is a device that projects images or videos onto a screen. Projectors can connect to computers, cable TV networks, the internet, VCD (Video Compact Disc), DVD (Digital Versatile Disc Recordable), game consoles, DV camcorders, and other devices via various interfaces to play corresponding video signals. Projectors are widely used in homes, offices, schools, and entertainment venues.

[0041] Figure 1 A schematic diagram of the placement of a projection device according to an embodiment of this application is shown. Figure 2 A schematic diagram of the optical path of a projection device according to an embodiment of this application is shown.

[0042] In some embodiments, reference Figure 1-2 This application provides a projection device including a projection screen 1 and a projection device 2. The projection screen 1 is fixed in a first position, and the projection device 2 is placed in a second position so that the image projected by the device matches the projection screen 1. The projection device includes a laser light source 100, an optical engine 200, a lens 300, and a projection medium 400. The laser light source 100 provides illumination for the optical engine 200, which modulates the light beam and outputs it to the lens 300 for imaging, projecting it onto the projection medium 400 to form a projected image. Since the laser light source 100, the optical engine 200, and the lens 300 are used together to emit projection light to project the image, in some embodiments of this application, the laser light source 100, the optical engine 200, and the lens 300 are collectively referred to as the light-emitting assembly.

[0043] In some embodiments, the laser source 100 of the projection device includes a laser assembly 110 and an optical lens assembly 120. The light beam emitted by the laser assembly 110 can pass through the optical lens assembly 120 to provide illumination for the optical engine. For example, the optical lens assembly 120 requires a high level of environmental cleanliness and airtightness; while the chamber in which the laser assembly is installed can be sealed with a lower level of dustproof sealing to reduce sealing costs.

[0044] In some embodiments, the optical engine 200 of the projection device may include a blue optical engine, a green optical engine, and a red optical engine, and may also include a heat dissipation system, a circuit control system, etc. It should be noted that in some embodiments, the light-emitting component of the projector may also be implemented using an LED light source.

[0045] Figure 3A schematic diagram of the circuit architecture of a projection device according to an embodiment of this application is shown. In some embodiments, the projection device may include a display control circuit 10, a laser light source 20, at least one laser driving component 30, and at least one brightness sensor 40. The laser light source 20 may include at least one laser corresponding to at least one laser driving component 30. Here, "at least one" refers to one or more, and "more than one" refers to two or more.

[0046] Based on this circuit architecture, the projection device can achieve adaptive adjustment. For example, by setting a brightness sensor 40 in the light output path of the laser light source 20, the brightness sensor 40 can detect the first brightness value of the laser light source and send the first brightness value to the display control circuit 10.

[0047] The display control circuit 10 can acquire the second brightness value corresponding to the driving current of each laser, and determine that the laser has a COD fault when the difference between the second brightness value and the first brightness value of the laser is greater than the difference threshold. Then the display control circuit can adjust the current control signal of the corresponding laser driving component until the difference is less than or equal to the difference threshold, thereby eliminating the COD fault of the blue laser. The projection device can eliminate the COD fault of the laser in a timely manner, reduce the damage rate of the laser, and improve the image display effect of the projection device.

[0048] Figure 4 A schematic diagram of the structure of a projection device according to an embodiment of this application is shown.

[0049] In some embodiments, the laser light source 20 in the projection device may include independently configured blue laser 201, red laser 202 and green laser 203. The projection device may also be called a three-color projection device. The blue laser 201, red laser 202 and green laser 203 are all modular lightweight (Mirai Console Loader, MCL) packaged lasers, which are small in size and facilitate compact arrangement of the optical path.

[0050] In some embodiments, the controller includes at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), a RAM (RandomAccess Memory), a ROM (Read-Only Memory), a first to an nth interface for input / output, a communication bus, etc.

[0051] In some embodiments, the projection device may be configured with a camera for working in conjunction with the projection device to adjust and control the projection process. For example, the camera configured with the projection device may be specifically implemented as a 3D camera or a binocular camera; when the camera is implemented as a binocular camera, it specifically includes a left camera and a right camera; the binocular camera can acquire the image and playback content presented on the screen corresponding to the projection device, i.e., the projection surface, which is projected by the optical engine built into the projection device.

[0052] When the projection device moves, its projection angle and distance to the projection surface change, which will cause the projected image to be distorted, and the projected image will be displayed as a trapezoidal image or other distorted image; the projection device controller can achieve automatic trapezoidal correction based on the image captured by the camera, by coupling the angle between the optical engine and the projection surface and the correct display of the projected image.

[0053] The camera, which can be used to capture images displayed on the projection surface, can be a webcam. The webcam may include a lens assembly containing a photosensitive element and a lens. The lens, through the refraction of light by multiple lenses, allows light from the scene to illuminate the photosensitive element. The photosensitive element, depending on the camera's specifications, may be based on a detection principle using charge-coupled devices or complementary metal-oxide-semiconductor (CMOS), converting light signals into electrical signals through a photosensitive material, and outputting the converted electrical signals as image data.

[0054] Figure 5 A schematic diagram of the lens structure of the projection device 2 in some embodiments is shown. To support the automatic focusing process of the projection device 2, such as... Figure 5 As shown, the lens 300 of the projection device 2 may also include an optical component 310 and a drive motor 320. The optical component 310 is a lens group consisting of one or more lenses, which can refract the light emitted by the optical engine 200, so that the light emitted by the optical engine 200 can be transmitted onto the projection surface to form a transmitted content image.

[0055] The optical assembly 310 may include a lens barrel and multiple lenses disposed within the lens barrel. Depending on whether the lenses are movable, the lenses in the optical assembly 310 can be divided into movable lenses 311 and fixed lenses 312. By changing the position of the movable lens 311, the distance between the movable lens 311 and the fixed lens 312 is adjusted, thereby changing the overall focal length of the optical assembly 310. Therefore, the drive motor 320 can connect to the movable lens 311 in the optical assembly 310, driving the movable lens 311 to move its position, thus achieving an automatic focusing function.

[0056] It should be noted that the focusing process described in some embodiments of this application refers to changing the position of the moving lens 311 by driving the motor 320, thereby adjusting the distance between the moving lens 311 and the fixed lens 312, that is, adjusting the image plane position. Therefore, in the imaging principle of the lens combination in the optical component 310, the adjustment of focal length is actually the adjustment of image distance. However, in terms of the overall structure of the optical component 310, adjusting the position of the moving lens 311 is equivalent to adjusting the overall focal length of the optical component 310.

[0057] When the distance between the projection device 2 and the projection surface varies, the lens of the projection device 2 needs to be adjusted to different focal lengths to project a clear image onto the projection surface. During projection, the distance between the projection device 2 and the projection surface will vary depending on the user's placement, requiring different focal lengths. Therefore, to adapt to different usage scenarios, the projection device 2 needs to adjust the focal length of the optical component 310.

[0058] Figure 6 A schematic diagram of the distance sensor and camera structure is shown in some embodiments. For example... Figure 6 As shown, the projection device 2 can also have a built-in or external camera 700. The camera 700 can capture images of the projected content from the projection device 2. The projection device 2 then performs a sharpness test on the projected content image to determine if the current lens focal length is suitable, and adjusts the focal length if it is not suitable. When automatically focusing based on the projected content image captured by the camera 700, the projection device 2 can continuously adjust the lens position and take pictures, and find the focus position by comparing the sharpness of the images before and after, thereby adjusting the moving lens 311 in the optical component to the appropriate position. For example, the controller 500 can first control the drive motor 320 to gradually move the moving lens 311 from the focusing starting position to the focusing ending position, and continuously acquire projected content images through the camera 700 during this period. Then, by performing a sharpness test on multiple projected content images, the position with the highest sharpness is determined, and finally the drive motor 320 is controlled to adjust the moving lens 311 from the focusing end position to the position with the highest sharpness, completing the automatic focusing.

[0059] Figure 7 A schematic diagram of the system framework for display control of a projection device according to an embodiment of this application is shown.

[0060] In some embodiments, the projection device 2 has the characteristics of a long-throw micro-projector, and its controller can control the display of the projected light image through a preset algorithm to achieve functions such as automatic keystone correction, automatic screen entry, automatic obstacle avoidance, automatic focus adjustment, and anti-glare.

[0061] In some embodiments, the projection device 2 is equipped with a gyroscope sensor; during the movement of the device, the gyroscope sensor can sense the position movement and actively collect movement data; then the collected data is sent to the application service layer through the system framework layer to support the application data required during user interface interaction and application interaction. The collected data can also be used by the controller for data calls in the algorithm service implementation.

[0062] In some embodiments, the projection device 2 is equipped with a Time-of-Flight (TOF) sensor, which is a sensor that uses the time-of-flight principle to measure the depth of an object. It measures the distance between the target object and the sensor by sending and receiving infrared light, thereby obtaining the depth information of the target object. After the time-of-flight sensor collects the corresponding data, the data is sent to the corresponding time-of-flight service in the service layer. After acquiring the data, the time-of-flight service sends the collected data to the application service layer through a process communication framework. The data will be used for data calls from the controller, user interface, program applications, and other interactive applications.

[0063] Time-of-flight (TOF) sensors come in various types, such as single-point sensors, multi-area TOF sensors, and 3D TOF sensors. All of these sensors calculate the distance between an object and the sensor by utilizing the time of flight of light. That is, they calculate the distance by measuring the time difference between the emission of light and its reflection back to the receiver.

[0064] Single-point sensors have a receiver and a transmitter, enabling single-point measurements. Multi-region TOF sensors can simultaneously measure multiple points by dividing the entire field of view into smaller regions and performing independent ranging on each region. Each region has its own transmitter and receiver for independent ranging. 3D TOF sensors can reconstruct the depth of a 3D scene by performing independent ranging on each pixel. Compared to multi-region TOF sensors, 3D TOF sensors do not require dividing the field of view into smaller regions; instead, they directly measure the depth of the entire field of view.

[0065] In some embodiments, the camera 700 configured in the projection device 2 may be a binocular camera, a depth camera, or a 3D camera, etc.; the data acquired by the camera 700 will be sent to the camera service, and then the camera service will send the acquired image data to the process communication framework and / or the projection device calibration service; the projection device calibration service can receive the camera acquisition data sent by the camera service, and the controller can call the corresponding control algorithm in the algorithm library for different functions to be implemented.

[0066] In some embodiments, data interaction is performed with the application service through a process communication framework, and then the calculation results are fed back to the correction service through the process communication framework. The correction service sends the obtained calculation results to the operating system of the projection device 2 to generate control signaling, and sends the control signaling to the optical engine 200 control driver to control the operating status of the optical engine 200 and realize automatic correction of the displayed image.

[0067] In some embodiments, when an image correction command is detected, the projection device 2 can correct the projected image. For projected image correction, a correlation between distance, horizontal angle, and offset angle can be pre-established. Then, the controller in the projection device 2 obtains the current distance between the optical engine 200 and the projection surface, and determines the angle between the optical engine 200 and the projection surface at that moment based on the corresponding correlation, thereby achieving projected image correction. Specifically, the angle is the angle between the central axis of the optical engine 200 and the projection surface.

[0068] In some embodiments, after the projection device 2 automatically completes the calibration and refocuses, the controller will detect whether the autofocus function is enabled; when the autofocus function is not enabled, the controller will terminate the autofocus operation; when the autofocus function is enabled, the projection device 2 will obtain the detection distance of the time-of-flight sensor through the middleware for calculation.

[0069] The controller queries a preset mapping table based on the acquired distance to obtain the focal length of the projection device 2; then the middleware sets the acquired focal length to the optical engine 200 of the projection device 2; wherein, the middleware is a series of applications related to the focusing control process. After the optical engine 200 emits a laser at the aforementioned focal length, the camera executes a photo-taking command; the controller determines whether the focusing process of the projection device 2 is complete based on the acquired image and evaluation function.

[0070] If the judgment result meets the preset completion conditions, the automatic focus adjustment process ends; if the judgment result does not meet the preset completion conditions, the middleware will fine-tune the focal length parameters of the optical engine 200 of the projection device 2. For example, the focal length can be finely adjusted gradually by preset step size, and the adjusted focal length parameters are set back to the optical engine 200; thereby realizing the repeated shooting and sharpness evaluation steps, and finally finding the optimal focal length through sharpness comparison to complete the automatic focus adjustment.

[0071] In some embodiments, when a user turns on the projection device 2, the projection device 2 can project content preset by the user onto a projection surface, which can be a wall or a screen, and the projected image can be displayed on the projection surface for the user to view.

[0072] In some embodiments, the projection device 2 supports an automatic obstacle avoidance function. When projecting media data onto the projection surface, the projection device 2 can detect obstacles on the projection surface. When an obstacle is detected on the projection surface, the target projection area without obstacles is re-divided on the projection surface, and the media data is projected onto the target projection area.

[0073] When projection device 2 performs obstacle avoidance, it captures an image of the projection surface 400 using camera 700 to determine the position of obstacles. Then, the light-emitting component automatically avoids obstacles based on the obstacle's position and the required projection area for the projected media data. In the image captured by camera 700, the obstacle is positioned in the camera coordinate system, and its coordinates are determined according to the camera coordinate system. When projection device 2 projects media data onto the projection surface 400, the light-emitting component projects the media data, and the coordinates of the media data's distribution on the projection surface 400 are determined according to the light-emitting component's coordinate system.

[0074] Therefore, calculating the target projection area also requires coordinate transformation between the camera coordinate system and the light-emitting component coordinate system. This coordinate transformation is based on the homography matrix between the camera (700) and the light-emitting component.

[0075] For example, such as Figure 8 The diagram shown is a schematic representation of the projection obstacle avoidance process provided in this embodiment. The projection device 2 projects a solid-color image (such as a white image) onto the projection surface 400 via the light-emitting component. The camera 700 takes a picture of the solid-color image displayed on the projection surface while the light-emitting component is projecting the image, thus obtaining a first sampled image. Then, a feature image (such as a checkerboard pattern) is projected onto the projection surface 400 via the light-emitting component. The camera 700 takes a picture of the solid-color image displayed on the projection surface while the light-emitting component is projecting the feature image, thus obtaining a second sampled image.

[0076] Based on the corner coordinates of the feature map card in the camera coordinate system, the corner coordinates of the projected feature map card in the light-emitting component, the intrinsic parameters calibrated by the camera 700, and the extrinsic parameters (rotation matrix R and translation vector T) of the camera 700 and the light-emitting component, the projection device 2 calculates the transformation relationship Hwp between the coordinate system of the light-emitting component and the world coordinate system of the projection surface under the current projection state.

[0077] Based on the corner coordinates of the feature map card in the camera coordinate system and the corner coordinates of the feature map card in the light output component coordinate system in the second sampled image captured by camera 700, the homography matrix Hpc between the camera coordinate system and the light output component coordinate system is calculated.

[0078] Then, based on the first sampled image captured by camera 700, the largest projectable area in the projection region is calculated. This projectable area is then transformed from the camera coordinate system to the light-emitting component coordinate system using the homography matrix Hpc. Next, the light-emitting component coordinate system is transformed to the world coordinate system using the transformation relation Hwp. The largest inscribed rectangle region is selected from the projectable area in the world coordinate system, and the coordinates of the four corner points Wi (i = 1 to 3) of the largest inscribed rectangle region in the world coordinate system are obtained. Finally, Wi is transformed from the world coordinate system to the light-emitting component coordinate system using the transformation relation Hwp, and the coordinates of the four corner points Pi of the target projection area in the light-emitting component coordinate system are obtained. Pi is then passed to the light-emitting component so that the light-emitting component projects the media data according to Pi, thereby achieving obstacle avoidance during projection.

[0079] As can be seen, in the above embodiments, the obstacle avoidance result relies on the calibration data between the camera 700 and the light-emitting component. Calibration requires data such as camera intrinsic parameters, optical engine intrinsic parameters, and extrinsic parameters of both the camera and the optical engine, making the calibration process complex. Errors in any data point will affect the obstacle avoidance result. Furthermore, the calibration between the camera 700 and the light-emitting component requires the light-emitting component to sequentially project solid color maps and feature maps. The light-emitting component needs a certain amount of conversion time when switching between projected media data, solid color maps, and feature maps. The camera 700 also needs time to capture images of the maps, making the calibration operation between the camera 700 and the light-emitting component complex, time-consuming, and reducing the efficiency of automatic obstacle avoidance.

[0080] Therefore, some embodiments of this application provide a projection obstacle avoidance method, which can be applied to the projection device 2 provided in the above embodiments. To implement the projection obstacle avoidance method, the projection device 2 should include at least a light-emitting component, a camera 700, a distance sensor, and a controller. The light-emitting component is configured to project projection content onto a projection surface, which can be a wall or a screen. The camera 700 is configured to capture sampled images. The distance sensor is configured to collect depth information between the distance sensor and the projection surface, and can be a multi-area TOF sensor or a 3D TOF sensor. The controller is configured to execute the program steps corresponding to the projection obstacle avoidance method, such as... Figure 9 The diagram shown is a flowchart illustrating the projection obstacle avoidance method provided in this application embodiment, which specifically includes the following:

[0081] S100, in response to obstacle avoidance commands, acquires sampled images.

[0082] The projection device 2 can automatically avoid obstacles on the projection surface 400 in response to obstacle avoidance commands. An obstacle avoidance command is a control command used to trigger the automatic obstacle avoidance process of the projection device 2. Obstacle avoidance commands can be automatically generated; for example, when the projection device 2 receives a projection command, it generates an obstacle avoidance command in response to the projection command, activating the automatic obstacle avoidance function. That is, before projecting the image onto the projection surface 400, the projection device 2 can automatically detect obstacles on the projection surface. Obstacle avoidance commands can also be actively input by the user; for example, the user can press a pre-set automatic obstacle avoidance switch on the projection device 2, or the automatic obstacle avoidance button on the remote control of the projection device 2, to input an obstacle avoidance command and activate the automatic obstacle avoidance function of the projection device 2.

[0083] In response to an obstacle avoidance command, the projection device 2 controls the light-emitting component to project a preset solid color image card onto the projection surface 400, and controls the camera 700 to take a picture of the solid color image card displayed on the projection surface 400 when the light-emitting component projects the solid color image card, so as to obtain a sampled image.

[0084] Projecting a solid-color image card means projecting a solid-color image. When the light-emitting component projects the solid-color image card onto the projection surface 400, some of the light from the solid-color image card will illuminate obstacles located between the light-emitting component and the projection surface 400. A shadow area will appear on the corresponding position of the projection surface 400 due to the obstruction of the obstacle. The controller of the projection device 2 can identify the location of the obstacle by recognizing the shadow area in the solid-color image card in the captured sampled image, and then perform subsequent obstacle avoidance functions based on the obstacle's location.

[0085] To facilitate subsequent calculation of the target projection area that does not contain obstacles, the size of the solid color map card projected by the light-emitting component should be the maximum projection size of the light-emitting component. That is, the projection device 2 controls the light-emitting component to project the solid color map card at the maximum projection size, and after projecting the solid color map card, controls the camera 700 to take a picture of the solid color map card displayed on the projection surface 400, so that the solid color map card area in the sampled image is the maximum projection area of ​​the projection device 2 in the current posture.

[0086] It should be noted that, in order to more clearly identify the location of obstacles, the solid color card should be a light color card, such as light yellow, light blue, white, gray, etc. This application embodiment does not impose specific restrictions on the color of the solid color card.

[0087] S200: Extract the projectible region from the sampled image.

[0088] After acquiring the sampled image, the projection device 2 can perform obstacle detection on the sampled image to identify the location of obstacles. Since the image area of ​​the projection surface image captured by the camera 700 is larger than the image area of ​​the projection area (the solid color card area), in order to reduce the resource consumption of obstacle detection in non-projection areas, the projection device 2 can first extract the card area containing the solid color card in the sampled image before performing obstacle detection on the sampled image. Then, obstacle detection is performed on the card area to identify obstacles in the card area. Based on the identified obstacle locations, areas that do not contain obstacles are then extracted from the card area to obtain the projectable area.

[0089] For example, in response to an obstacle avoidance command, projection device 2 controls the light-emitting component to project a white image card onto projection surface 400, and controls camera 700 to take a picture of the pure white image card displayed on projection surface 400, so as to obtain... Figure 10 The sampled image is shown. After acquiring the sampled image, the projection device 2 extracts the area of ​​the white card located in the sampled image, as shown in the figure. Figure 11 The white map area shown is the area excluding the shaded areas. Obstacle detection is then performed on the white map area, and based on the obstacle detection results, the largest blank area without obstacles is extracted from the map area, resulting in the following: Figure 12 The projectable area shown is the area within the dashed box.

[0090] S300: Obtain the first transformation matrix and the second transformation matrix.

[0091] The first transformation matrix is ​​the transformation matrix between the light-emitting component coordinate system and the projection plane world coordinate system, and the second transformation matrix is ​​the transformation matrix between the camera coordinate system and the projection plane world coordinate system. Both the first and second transformation matrices are calculated based on depth information collected by the time-of-flight sensor.

[0092] The first transformation matrix can be calculated based on the depth information collected by the distance sensor, the projection parameters (intrinsic parameters) of the light-emitting component, and the pose parameters (extrinsic parameters) between the distance sensor and the light-emitting component. The projection device 2 can acquire the depth information collected by the distance sensor, construct point cloud data based on the depth information, and calculate the first world coordinate system, where the first world coordinate system is a world coordinate system with the optical center of the distance sensor as the origin.

[0093] Then, the pose parameters between the light-emitting component and the distance sensor are obtained, including the rotation matrix R and the translation vector T. Based on the pose parameters and the first world coordinate system, a second world coordinate system is calculated, where the second world coordinate system is a world coordinate system with the light-emitting component as the origin. Then, based on the second world coordinate system, the projection plane 400 is fitted to obtain the normal vector of the projection plane 400. The projection parameters of the light-emitting component are then obtained, and based on the normal vector and the projection parameters, the first transformation matrix is ​​calculated.

[0094] For example, such as Figure 13 As shown, the time-of-flight sensor is a multi-region TOF sensor. The multi-region TOF sensor divides the projection area of ​​the projection device 2 under its current projection state into multiple regions. For example, the multi-region TOF sensor can arrange infrared light sources into a 4x4 or 8x8 grid shape and measure the time from emission to reflection for each grid cell, thereby obtaining depth information for multiple regions (4x4 or 8x8 regions). The acquired depth information from multiple regions is converted into point cloud Fi(Xti, Yti, Zti), which gives the world coordinates of multiple regions in the projection area under the current projection state, with the optical center of the multi-region TOF sensor as the origin.

[0095] Then, based on the pose parameters R between the light-emitting component and the multi-region TOF sensor... tp Calculate the world coordinates Pi(Xpi, Ypi, Zpi) = R for multiple regions within the projected area, with the optical center at optical engine 200 as the origin. tp *Fi(Xti, Yti, Zti).

[0096] Then, the normal vector n of the projection plane 400 is calculated by fitting the projection plane 400. And based on the optomechanical intrinsic parameter M... p The transformation relationship H between the optical-mechanical coordinate system and the projection world coordinate system is calculated. wP That is, the first transformation matrix.

[0097] For the second transformation matrix, the standard projection region in the world coordinate system of the projection plane can be calculated based on the first transformation matrix. The standard projection region is the projection region of the maximum size projected by the preset light-emitting component. Then, the map region in the camera coordinate system can be calculated, where the map region is the area containing the solid-color map in the sampled image. Finally, the second transformation matrix is ​​calculated based on the standard projection region in the world coordinate system of the projection plane and the map region in the camera coordinate system.

[0098] The second transformation matrix can be calculated based on the corner coordinates of the standard projection area and the corner coordinates of the map area. The projection device 2 can obtain the projection corner coordinates of the standard projection area and the map corner coordinates of the map area, where the projection corner coordinates are the world coordinates of the corners of the standard projection area and the map corner coordinates are the pixel coordinates of the corners of the map area.

[0099] Using the projection plane 400 as a plane, the vertical axis coordinate value in the three-dimensional coordinates of the projection corner point coordinates is deleted to obtain the projection plane corner point coordinates. Finally, based on the projection plane corner point coordinates and the map corner point coordinates, the homography matrix between the camera coordinate system and the projection plane world coordinate system is calculated to obtain the second transformation matrix.

[0100] For example, such as Figure 13 As shown, following the above embodiment, based on the first transformation matrix H wP Calculate the world coordinates Wi(Xi, Yi, Zi) of the four corner points of the projection area with the maximum projection size under the current projection state of the light component. Take the projection surface 400 as a plane and delete the vertical axis coordinate value Zi in the world coordinates Wi(Xi, Yi, Zi) of the four corner points to obtain the coordinates Wi(Xi, Yi) of the four corner points on the projection surface.

[0101] Next, calculate the pixel coordinates Ci(Xi, Yi) of the four corner points of the white map area captured by camera 400 in the sampled image. Finally, based on the coordinate pairs of Wi(Xi, Yi) and Ci(Xi, Yi), calculate the homography matrix H between the projection plane and the camera image coordinates. wC That is, the second transformation matrix.

[0102] It is understandable that, in the above example, the optomechanical intrinsic parameter is M. p The poses of the optomechanical system 200 and the multi-region TOF sensor are: R tp That is, the world coordinates of point O in space in the multi-region TOF sensor coordinate system are F(Xt, Yt, Zt), and the world coordinates in the optomechanical coordinate system are P(Xp, Yp, Zp). The transformation relationship satisfies F T =R tp *p T p T =R p t*FT.

[0103] The transformation relationship between the optical-mechanical system 200 image coordinates and world coordinates is: H wP That is, the coordinates of point O in space are W(Xt, Yt, Zt) in the world coordinate system and P(Xp, Yp) in the optomechanical image coordinate system. The transformation relationship satisfies W T =H wp *P T p T =H wP *W T .

[0104] The transformation relationship between camera 700 image coordinates and world coordinates is: H wcThat is, the coordinates of point O in space in the world coordinate system are W(Xt, Yt, Zt), and the coordinates in the camera image are C(Xc, Yc). The transformation relationship satisfies W T =H wc *C T C T =H wC *W T .

[0105] S400: Delineate the target projection area in the projectable region under the world coordinate system of the projection plane according to the second transformation matrix.

[0106] The target projection area is the inscribed rectangular region with the largest preset aspect ratio that can be accommodated within the projectable area in the world coordinate system of the projection surface. The projection device 2 can fit the region contour of the projectable area, and transform the projectable area from the camera coordinate system to the world coordinate system of the projection surface according to the contour ratio and the second transformation matrix to obtain the spatial projection area. Then, it extracts the inscribed rectangular region with the largest preset aspect ratio from the spatial projection area to obtain the target projection area.

[0107] For example, following the above embodiments, the region contour of the projectible area is fitted (e.g.) Figure 12 The region outline shown is calculated according to the outline proportions and the second transformation matrix H. wC Transform the projectable region from the camera coordinate system to the world coordinate system of the projection surface to obtain, as follows: Figure 14 The spatial projection area shown (shaded area and white area), i.e. Figure 12 The outline of the projectable area shown corresponds to the outline of the maximum projection area of ​​the optical engine 200 (the area within the dashed box). Finally, in Figure 14 Extract the inscribed rectangular region with the maximum preset aspect ratio from the spatial projection area shown. Figure 14 (white area) to obtain the target projection area.

[0108] S500: Calculate the target projection area in the coordinate system of the light component according to the first transformation matrix, and control the light output component to project the projection content onto the target projection area in the coordinate system of the light output component.

[0109] After the projection device 2 obtains the target projection area in the world coordinate system of the projection surface, it needs to transform the target projection area from the world coordinate system of the projection surface to the coordinate system of the light output component, and control the light output component to project the projection content onto the target projection area in the coordinate system of the light output component, thereby achieving obstacle avoidance in projection.

[0110] Projection device 2 can obtain the coordinates of the planar corner points of the target projection area, add the vertical axis coordinate value of the three-dimensional coordinate system to the planar corner point coordinates to obtain the world corner point coordinates. Then, according to the first transformation matrix, the world corner point coordinates are transformed from the projection surface world coordinate system to the light output component coordinate system to obtain the target projection area in the light output component coordinate system.

[0111] For example, such as Figure 13 As shown, following the above embodiment, the coordinates of the four corner points of the target projection area are Wj(Xj, Yj). Adding the vertical axis coordinate value Zj to the four corner point coordinates Wj(Xj, Yj) yields the coordinates Wi(Xi, Yi, Zj) of the four corner points in the world coordinates of the projection plane. Then, according to the first transformation matrix H... wP The coordinates of the four corner points of the target projection area are transformed from the world coordinate system of the projection surface to the coordinate system of the light output component, resulting in the coordinates of the four corner points of the target projection area in the coordinate system of the light output component as P(Xp, Yp, Zp) = H. wP *Wj T Finally, the light-emitting component is controlled to project media data according to the coordinates P(Xp, Yp, Zp) of the four corner points to achieve obstacle avoidance during projection.

[0112] It is understood that the embodiments of this application only require a solid color image of the card captured by the camera, the intrinsic parameters of the optomechanical system, and the extrinsic parameters of the distance sensor and the optomechanical system to achieve obstacle avoidance. There is no need for feature card capture or pose calibration steps for the camera and optomechanical system, thus eliminating the influence of pose errors of the camera and optomechanical system on obstacle avoidance. Furthermore, compared to the extrinsic parameter calibration between the camera and the light-emitting components, the extrinsic parameter calibration of the distance sensor and the optomechanical system is simpler and less time-consuming, shortening obstacle avoidance time, improving obstacle avoidance efficiency, and enhancing user experience.

[0113] Based on the projection obstacle avoidance method provided in the above embodiments, some embodiments of this application also provide a projection device 2, which includes a light-emitting component, a camera 700, a distance sensor, and a controller. The light-emitting component is configured to project projection content onto a projection surface. The camera 700 is configured to capture sampled images. The distance sensor is configured to collect depth information between the distance sensor and the projection surface. The controller is configured to execute the following program steps:

[0114] In response to obstacle avoidance commands, acquire sampled images.

[0115] The sampled images are those captured by the camera when it projects a solid color image onto the light-emitting component.

[0116] Extract the projectible region from the sampled image.

[0117] The projectable region is the area in the sampled image that does not contain obstacles.

[0118] Obtain the first transformation matrix and the second transformation matrix.

[0119] The first transformation matrix is ​​the transformation matrix between the light-emitting component coordinate system and the projection surface world coordinate system, and the second transformation matrix is ​​the transformation matrix between the camera coordinate system and the projection surface world coordinate system. The first and second transformation matrices are calculated based on the depth information collected by the distance sensor.

[0120] Based on the second transformation matrix, the target projection area is delineated in the projectable region under the world coordinate system of the projection surface.

[0121] The target projection area is the inscribed rectangular area with the largest preset aspect ratio that can be accommodated in the projectable area under the projection plane world coordinate system.

[0122] Based on the first transformation matrix, the target projection area in the coordinate system of the light component is calculated, and the light output component is controlled to project the projection content onto the target projection area in the coordinate system of the light output component.

[0123] As can be seen from the above technical solutions, the projection device and projection obstacle avoidance method provided in the above embodiments can, in response to obstacle avoidance commands, acquire a sampled image captured by the camera when the light-emitting component projects a solid-color image card, extract a projectable area from the sampled image, and then acquire a first transformation matrix and a second transformation matrix. The first transformation matrix is ​​the transformation matrix between the coordinate system of the light-emitting component and the world coordinate system of the projection surface, and the second transformation matrix is ​​the transformation matrix between the camera coordinate system and the world coordinate system of the projection surface. Based on the second transformation matrix, a target projection area is defined within the projectable area in the world coordinate system of the projection surface, and based on the first transformation matrix, the target projection area in the coordinate system of the light-emitting component is calculated. The light-emitting component is then controlled to project the content onto the target projection area, thereby achieving projection obstacle avoidance. This method eliminates the need for feature card capture and camera and optical engine pose calibration, shortening obstacle avoidance time and improving user experience.

[0124] The same or similar parts among the various embodiments in this specification can be referred to mutually, and will not be repeated here.

[0125] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or certain parts of the embodiments of the present invention.

[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0127] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the described embodiments and various different variations of embodiments suitable for specific use considerations.

Claims

1. A projection device, characterized by include: The light-emitting component is configured to project the content onto the projection surface; The camera is configured to capture sampled images; A distance sensor is configured to acquire depth information between the distance sensor and the projection surface; The controller is configured as follows: In response to an obstacle avoidance command, a sampled image is acquired, wherein the sampled image is an image captured by the camera when the light-emitting component projects a solid color image card; Extract a projectable region from the sampled image, wherein the projectable region is the area in the sampled image that does not contain obstacles; Based on the depth information, a first world coordinate system is determined with the optical center of the distance sensor as the origin; Based on the pose parameters between the light-emitting component and the distance sensor, the first world coordinate system is converted into a second world coordinate system with the light-emitting component as the origin; In the second world coordinate system, the normal vector of the projection surface is fitted, and the world coordinate system of the projection surface is determined based on the normal vector; Determine the first transformation matrix between the projection surface world coordinate system and the light-emitting component coordinate system, and the second transformation matrix between the projection surface world coordinate system and the camera coordinate system; According to the second transformation matrix, the projectable region is transformed from the camera coordinate system to the projection plane world coordinate system, and a target projection region is defined in the projectable region under the projection plane world coordinate system; the target projection region is the inscribed rectangular region with the largest preset aspect ratio that can be accommodated in the projectable region under the projection plane world coordinate system. According to the first transformation matrix, the target projection area is transformed from the projection surface world coordinate system to the light-emitting component coordinate system, and the light-emitting component is controlled to project the projection content onto the target projection area in the light-emitting component coordinate system.

2. The projection device according to claim 1, characterized in that, The controller is configured to acquire sampled images and is also configured to: In response to an obstacle avoidance command, the light-emitting component is controlled to project a preset solid color image card; When the camera is controlled to project a solid color image onto the projection surface of the solid color image card by the light-emitting component, a sampled image is obtained.

3. The projection device according to claim 1, characterized in that, The controller performs the extraction of a projectable region from the sampled image and is further configured to: Extract the region of the solid color card located in the sampled image; Obstacle detection is performed on the map area to identify obstacles in the map area; Extract the area that does not contain obstacles from the map area to obtain the projectable area.

4. The projection device according to claim 1, characterized in that, The pose parameters include rotation matrix and translation vector.

5. The projection device according to claim 1, characterized in that, The controller executes a first transformation matrix to determine the world coordinate system of the projection surface and the coordinate system of the light-emitting component, and is further configured to: Obtain the projection parameters of the light-emitting component; The first transformation matrix is ​​calculated based on the normal vector and the projection parameters.

6. The projection device according to claim 1, characterized in that, The controller executes the determination of a second transformation matrix between the projection plane world coordinate system and the camera coordinate system, and is further configured to: Based on the first transformation matrix, the standard projection area in the world coordinate system of the projection surface is calculated, and the standard projection area is the projection area of ​​the maximum size projected by the light-emitting component; Calculate the map region in the camera coordinate system, where the map region is the area where the solid color map is located in the sampled image; The second transformation matrix is ​​calculated based on the standard projection area in the world coordinate system of the projection surface and the map area in the camera coordinate system.

7. The projection device according to claim 6, characterized in that, The controller performs the calculation of the second transformation matrix based on the standard projection area in the world coordinate system of the projection surface and the map area in the camera coordinate system, and is further configured to: Obtain the projection corner coordinates of the standard projection area, where the projection corner coordinates are the world coordinates of the corner points of the standard projection area; Delete the vertical axis coordinate value from the three-dimensional coordinates of the projected corner point coordinates to obtain the corner point coordinates of the projection plane; Obtain the coordinates of the corner points of the image card area, where the corner points are the pixel coordinates of the corner points of the image card area; Based on the corner coordinates of the projection plane and the corner coordinates of the map, the homography matrix between the camera coordinate system and the world coordinate system of the projection plane is calculated to obtain the second transformation matrix.

8. The projection device according to claim 1, characterized in that, The controller performs a transformation based on the second transformation matrix, converting the projectable region from the camera coordinate system to the projection plane world coordinate system, and delineates the target projection region within the projectable region in the projection plane world coordinate system. It is also configured to: Fit the region contour of the projectible area; Based on the contour ratio of the region and the second transformation matrix, the projectable region is transformed from the camera coordinate system to the projection surface world coordinate system to obtain the spatial projection region. Extract the inscribed rectangular region with the largest preset aspect ratio from the spatial projection area to obtain the target projection area.

9. The projection device according to claim 8, characterized in that, The controller performs a transformation based on the first transformation matrix, converting the target projection region from the projection surface world coordinate system to the light-emitting component coordinate system, and is further configured to: Obtain the coordinates of the planar corner points of the target projection area; Add the vertical axis coordinate value from the three-dimensional coordinate system to the coordinates of the planar corner point to obtain the world corner point coordinates; Based on the first transformation matrix, the world corner coordinates are transformed from the projection surface world coordinate system to the light-emitting component coordinate system to obtain the target projection area in the light-emitting component coordinate system.

10. A projection obstacle avoidance method, characterized in that, This invention is applied to a projection device, which includes a light-emitting component, a camera, a distance sensor, and a controller; the light-emitting component is configured to project content onto a projection surface; and the camera is configured to capture sampled images. The distance sensor is configured to acquire depth information between the distance sensor and the projection surface; The projection obstacle avoidance method includes: In response to an obstacle avoidance command, a sampled image is acquired, wherein the sampled image is an image captured by the camera when the light-emitting component projects a solid color image card; Extract a projectable region from the sampled image, wherein the projectable region is the area in the sampled image that does not contain obstacles; Based on the depth information, a first world coordinate system is determined with the optical center of the distance sensor as the origin; Based on the pose parameters between the light-emitting component and the distance sensor, the first world coordinate system is converted into a second world coordinate system with the light-emitting component as the origin; In the second world coordinate system, the normal vector of the projection surface is fitted, and the world coordinate system of the projection surface is determined based on the normal vector; Determine the first transformation matrix between the projection surface world coordinate system and the light-emitting component coordinate system, and the second transformation matrix between the projection surface world coordinate system and the camera coordinate system; According to the second transformation matrix, the projectable region is transformed from the camera coordinate system to the projection plane world coordinate system, and a target projection region is defined in the projectable region under the projection plane world coordinate system; the target projection region is the inscribed rectangular region with the largest preset aspect ratio that can be accommodated in the projectable region under the projection plane world coordinate system. According to the first transformation matrix, the target projection area is transformed from the projection surface world coordinate system to the light-emitting component coordinate system, and the light-emitting component is controlled to project the projection content onto the target projection area in the light-emitting component coordinate system.

Images