Projection device and projection obstacle avoidance method

Abstract

The application provides a projection device and an obstacle avoidance method. In the case that an obstacle is identified in a projection area, the projection device further acquires an instruction input by a user for specifying a first obstacle target, to confirm whether obstacle avoidance processing is needed in a projection process. Whether the first obstacle target exists and its position when existing are confirmed by shooting a sampling image. The coordinates of the sampling image are converted to a light emitting assembly coordinate system through a homography matrix between a camera and the light emitting assembly. The projection device re-divides the projection area according to the position of the first obstacle target when projecting media data, and projects the media data to the re-divided projection area. When the projection device identifies an obstacle, the automatic obstacle avoidance function is not executed when the obstacle does not affect the projection effect, the projection effect is enhanced, and the user experience is improved.

Description

Technical Field

[0001] This application relates to the field of projection equipment technology, and in particular to a projection device and a projection obstacle avoidance method. Background Technology

[0002] Projection equipment can project media data onto a wall or screen through a projector, thereby presenting the media data to the user at a larger scale and enhancing the user's viewing experience.

[0003] When a projector projects media data, the wall or screen can be considered the projection area. If an obstacle appears in the projection area, the projected media data will be affected, causing the image to flicker or become incomplete, impacting the user's viewing experience. For movable obstacles in the projection area, such as hooks, the user can remove the hooks to ensure a smooth wall surface. For immovable obstacles in the projection area, such as light switches, the projector will automatically avoid these obstacles and re-evaluate the projection area.

[0004] However, in some cases, obstacles that are difficult to move will not affect the projection effect because their flat structure and color are similar to the projection area. In order to avoid the obstacle, the projection device will still redivide the projection area. Obviously, the redivised projection area is inferior to the original projection area in terms of projection position and projection scale, thus affecting the projection effect. Summary of the Invention

[0005] This application provides a projection device and a projection obstacle avoidance method to solve the problem that the projection effect is poor when the projection device avoids obstacles during the automatic obstacle avoidance process.

[0006] On one hand, this application provides a projection device, including a light-emitting component, a camera, and a controller. The light-emitting component is configured to project projection content onto a projection surface. The camera is configured to capture sampled images. The controller is configured to:

[0007] In response to an obstacle avoidance command, a transformation matrix is ​​acquired, as well as an adjustment command input by the user. The transformation matrix is ​​a homography matrix of the coordinates between the camera and the light-emitting component. The adjustment command includes a first obstacle target specified by the user.

[0008] The first obstacle target is identified in the sampled image, which is an image captured by the camera when the light-emitting component projects a corrected image.

[0009] Based on the first obstacle target, a projectable area is defined in the sampled image. The projectable area is a rectangular area with the largest preset aspect ratio that can be accommodated by the area in the sampled image other than the first obstacle target.

[0010] The target projection area is determined according to the projectable area and the transformation matrix, and the light-emitting component is controlled to project the content onto the target projection area.

[0011] On the other hand, this application also provides a projection obstacle avoidance method, including:

[0012] In response to an obstacle avoidance command, a transformation matrix is ​​acquired, as well as an adjustment command input by the user. The transformation matrix is ​​a homography matrix of the coordinates between the camera and the light-emitting component. The adjustment command includes a first obstacle target specified by the user.

[0013] The first obstacle target is identified in the sampled image, which is an image captured by the camera when the light-emitting component projects a corrected image.

[0014] Based on the first obstacle target, a projectable area is defined in the sampled image. The projectable area is a rectangular area with the largest preset aspect ratio that can be accommodated by the area in the sampled image other than the first obstacle target.

[0015] The target projection area is determined according to the projectable area and the transformation matrix, and the light-emitting component is controlled to project the content onto the target projection area.

[0016] As can be seen from the above technical solution, when the projection device of this application detects an obstacle in the projection area, it also obtains the user's input instruction to specify a first obstacle target to confirm whether obstacle avoidance processing is required during projection. By capturing a sample image, the existence and location of the first obstacle target are confirmed. Specifically, the coordinates of the sample image are transformed to the coordinate system of the light-emitting component using the homography matrix between the camera and the light-emitting component. When projecting media data, the projection device re-divides the projection area according to the location of the first obstacle target and projects the media data into the re-divised projection area. When the projection device detects an obstacle, it combines the user's obstacle avoidance instruction and does not execute the automatic obstacle avoidance function if the obstacle does not affect the projection effect, thus ensuring the projection effect and improving the user experience. Attached Figure Description

[0017] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the projection state of the projection device in an embodiment of this application;

[0019] Figure 2 This is a schematic diagram of the projection device structure in an embodiment of this application;

[0020] Figure 3 This is a schematic diagram of the optical engine architecture of the projection device in the embodiments of this application;

[0021] Figure 4 This is a schematic diagram of the optical path of the projection device in the embodiments of this application;

[0022] Figure 5 This is a schematic diagram of the system framework of the projection device in the embodiments of this application;

[0023] Figure 6 This is a schematic diagram illustrating the automatic obstacle avoidance performed by the projection device in an embodiment of this application;

[0024] Figure 7 This is a schematic diagram illustrating the automatic obstacle avoidance performed by the projection device according to user instructions in an embodiment of this application;

[0025] Figure 8 This is a schematic diagram illustrating obstacle avoidance based on user instructions in an embodiment of this application;

[0026] Figure 9 This is a schematic diagram of a projection device projecting a solid color image card and a feature image card in an embodiment of this application;

[0027] Figure 10 This is a schematic diagram of the projection device dividing the projectable area according to the first obstacle target in an embodiment of this application;

[0028] Figure 11 This is a schematic diagram of the projection device inserting a graphic card into the media asset data stream according to an embodiment of this application;

[0029] Figure 12 This is a schematic diagram illustrating the steps performed by the projection device after inserting a graphics card into the media asset data stream in an embodiment of this application.

[0030] Figure 13 This is a schematic diagram illustrating the projection device recognizing user-inputted voice commands in an embodiment of this application;

[0031] Figure 14 This is a schematic diagram illustrating how the projection device determines whether the adjustment command input by the user conforms to the standard range in an embodiment of this application;

[0032] Figure 15 This is a schematic diagram of the projection device identifying the second faulty target in an embodiment of this application. Detailed Implementation

[0033] To make the objectives and implementation methods of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the exemplary embodiments described are only some embodiments of this application, and not all embodiments.

[0034] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0035] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0036] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0037] The term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0038] The embodiments of this application can be applied to various types of projection devices. The following description will use a projector as an example to illustrate the projection device and the automatic focusing method.

[0039] A projector is a device that projects images or videos onto a screen. Projectors can connect to computers, cable TV networks, the internet, VCD (Video Compact Disc), DVD (Digital Video Disc Recordable), game consoles, DV camcorders, and other devices via various interfaces to play corresponding video signals. Projectors are widely used in homes, offices, schools, and entertainment venues.

[0040] Figure 1 A schematic diagram of the placement of a projection device according to an embodiment of this application is shown. Figure 2 A schematic diagram of the optical path of a projection device according to an embodiment of this application is shown.

[0041] In some embodiments, reference Figure 1-2This application provides a projection device including a projection screen 1 and a projection device 2. The projection screen 1 is fixed in a first position, and the projection device 2 is placed in a second position so that the image projected by the device matches the projection screen 1. The projection device includes a laser light source 100, an optical engine 200, a lens 300, and a projection surface 400. The laser light source 100 provides illumination for the optical engine 200, which modulates the light beam and outputs it to the lens 300 for imaging, projecting it onto the projection surface 400 to form a projected image. Since the laser light source 100, the optical engine 200, and the lens 300 are used together to emit projection light to project the image, in some embodiments of this application, the laser light source 100, the optical engine 200, and the lens 300 are collectively referred to as the light-emitting assembly.

[0042] In some embodiments, the laser source 100 of the projection device includes a laser assembly 110 and an optical lens assembly 120. The light beam emitted by the laser assembly 110 can pass through the optical lens assembly 120 to provide illumination for the optical engine. For example, the optical lens assembly 120 requires a high level of environmental cleanliness and airtightness; while the chamber in which the laser assembly is installed can be sealed with a lower level of dustproof sealing to reduce sealing costs.

[0043] In some embodiments, the optical engine 200 of the projection device may include a blue optical engine, a green optical engine, and a red optical engine, and may also include a heat dissipation system, a circuit control system, etc. It should be noted that in some embodiments, the light-emitting component of the projector may also be implemented using an LED light source.

[0044] Figure 3 A schematic diagram of the circuit architecture of a projection device according to an embodiment of this application is shown. In some embodiments, the projection device may include a display control circuit 10, a laser light source 20, at least one laser driving component 30, and at least one brightness sensor 40. The laser light source 20 may include at least one laser corresponding to at least one laser driving component 30. Here, "at least one" refers to one or more, and "more than one" refers to two or more.

[0045] Based on this circuit architecture, the projection device can achieve adaptive adjustment. For example, by setting a brightness sensor 40 in the light output path of the laser light source 20, the brightness sensor 40 can detect the first brightness value of the laser light source and send the first brightness value to the display control circuit 10.

[0046] The display control circuit 10 can acquire the second brightness value corresponding to the driving current of each laser, and determine that the laser has a COD fault when the difference between the second brightness value and the first brightness value of the laser is greater than the difference threshold. Then the display control circuit can adjust the current control signal of the corresponding laser driving component until the difference is less than or equal to the difference threshold, thereby eliminating the COD fault of the blue laser. The projection device can eliminate the COD fault of the laser in a timely manner, reduce the damage rate of the laser, and improve the image display effect of the projection device.

[0047] Figure 4 A schematic diagram of the structure of a projection device according to an embodiment of this application is shown.

[0048] In some embodiments, the laser light source 20 in the projection device may include independently configured blue laser 201, red laser 202 and green laser 203. The projection device may also be called a three-color projection device. The blue laser 201, red laser 202 and green laser 203 are all modular lightweight (Mirai Console Loader, MCL) packaged lasers, which are small in size and facilitate compact arrangement of the optical path.

[0049] In some embodiments, the controller includes at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), RAM (random access memory), ROM (read-only memory), a first to an nth interface for input / output, a communication bus, etc.

[0050] In some embodiments, the projection device may be configured with a camera for working in conjunction with the projection device to adjust and control the projection process. For example, the camera configured with the projection device may be specifically implemented as a 3D camera or a binocular camera; when the camera is implemented as a binocular camera, it specifically includes a left camera and a right camera; the binocular camera can acquire the image and playback content presented on the screen corresponding to the projection device, i.e., the projection surface, which is projected by the optical engine built into the projection device.

[0051] When the projection device moves, its projection angle and distance to the projection surface change, which will cause the projected image to be distorted, and the projected image will be displayed as a trapezoidal image or other distorted image; the projection device controller can achieve automatic trapezoidal correction based on the image captured by the camera, by coupling the angle between the optical engine and the projection surface and the correct display of the projected image.

[0052] Figure 5A schematic diagram of the system framework for display control of a projection device according to an embodiment of this application is shown.

[0053] In some embodiments, the projection device has the characteristics of a long-throw micro-projector, and its controller can control the display of the projected light image through a preset algorithm to achieve functions such as automatic keystone correction, automatic screen entry, automatic obstacle avoidance, automatic focus adjustment, and eye protection.

[0054] In some embodiments, the projection device is equipped with a gyroscope sensor; during the movement of the device, the gyroscope sensor can sense the position movement and actively collect movement data; then the collected data is sent to the application service layer through the system framework layer to support the application data required during user interface interaction and application interaction. The collected data can also be used by the controller for data calls in the algorithm service implementation.

[0055] In some embodiments, the projection device is equipped with a time-of-flight sensor. After the time-of-flight sensor collects the corresponding data, the data will be sent to the time-of-flight service corresponding to the service layer. After the time-of-flight service obtains the data, it will send the collected data to the application service layer through a process communication framework. The data will be used for data calls, user interfaces, program applications, and other interactive applications of the controller.

[0056] In some embodiments, the projection device is configured with a camera for acquiring images, which may be a binocular camera, a depth camera, or a 3D camera, etc. The camera acquisition data is sent to a camera service, and then the camera service sends the acquired image data to a process communication framework and / or a projection device calibration service. The projection device calibration service can receive the camera acquisition data sent by the camera service, and the controller can call the corresponding control algorithm in the algorithm library for different functions to be implemented.

[0057] In some embodiments, data interaction is performed with the application service through a process communication framework, and the calculation results are then fed back to the correction service through the process communication framework. The correction service sends the obtained calculation results to the projection device operating system to generate control signaling, and sends the control signaling to the light output component control driver to control the operating conditions of the light output component and realize automatic correction of the displayed image.

[0058] In some embodiments, the projection device uses an autofocus algorithm and its configured laser rangefinder to obtain the current object distance, calculate the initial focal length and search range, and then drives the camera to take pictures and uses the corresponding algorithm to evaluate the sharpness.

[0059] Within the aforementioned search range, the projection device uses a search algorithm to find the optimal focal length, then repeats the steps of taking photos and evaluating sharpness. Finally, it finds the optimal focal length through sharpness comparison and completes automatic focusing.

[0060] For example, after the projection device is turned on, the user moves the device; after the projection device automatically completes the calibration and refocuses, the controller will detect whether the autofocus function is enabled; when the autofocus function is not enabled, the controller will end the autofocus operation; when the autofocus function is enabled, the projection device will obtain the detection distance of the time-of-flight sensor through the middleware for calculation.

[0061] The controller queries a preset mapping table based on the acquired distance to obtain the focal length of the projection device; then the middleware sets the acquired focal length to the light-emitting component of the projection device; after the light-emitting component emits laser light at the aforementioned focal length, the camera executes the image capture command; the controller determines whether the projection device has completed focusing based on the acquired image and evaluation function.

[0062] If the judgment result meets the preset completion conditions, the automatic focus adjustment process ends; if the judgment result does not meet the preset completion conditions, the middleware will fine-tune the focal length parameters of the light output component of the projection device. For example, the focal length can be finely adjusted gradually by preset step size, and the adjusted focal length parameters are set back to the light output component. This achieves repeated photo taking and sharpness evaluation steps, and finally finds the optimal focal length through sharpness comparison to complete the automatic focus adjustment.

[0063] like Figure 6 As shown, in some embodiments, when the projection device projects media data onto a wall, it detects a control switch on the wall. Upon detecting the control switch, the projection device considers it an obstacle that will affect the projection effect, and therefore activates the automatic obstacle avoidance function. After the automatic obstacle avoidance function is executed, the projection device avoids the control switch, re-divides the projection area on the wall, and projects the media data into the projection area. However, it can be seen that the area of ​​the re-divised projection area is smaller than the original projection area.

[0064] When projecting media data, projection devices can select multiple projection ratios based on the predefined projection area to achieve the best projection effect, and these ratios can also meet various user needs. However, because automatic obstacle avoidance reduces the projection area, the range of projection ratios available for media data is directly reduced, thus failing to achieve the optimal projection effect.

[0065] like Figure 7 As shown, some embodiments of this application provide a projection obstacle avoidance method. When an obstacle is detected in the projection area, the method also obtains user-inputted instructions regarding the obstacle, and determines whether to avoid the obstacle in the projection area based on the user-inputted instructions. Figure 8 As shown, the projection obstacle avoidance method includes the following steps:

[0066] S100: In response to obstacle avoidance commands, obtain the transformation matrix and obtain adjustment commands input by the user.

[0067] The transformation matrix is ​​the homography matrix between the camera and the light-emitting component, used for coordinate transformation of a target at the same location in the projection area between the camera coordinate system and the light-emitting component's coordinate system. When an obstacle appears, the camera first needs to capture an image of the projection area to confirm the obstacle's position. Then, the light-emitting component automatically avoids the obstacle based on its position and the required projection area according to the projected media data.

[0068] In images captured by a camera, obstacles are positioned in the camera's coordinate system, and their coordinates are determined according to this system. When a projection device projects media data onto a projection area, the light-emitting component projects the data, and the coordinates of the media data's distribution on the projection area are determined according to the light-emitting component's coordinate system. Therefore, the coordinates of obstacles in the projection area, defined in the camera's coordinate system, need to be converted to the light-emitting component's coordinate system. The controller identifies the presence of obstacles within the projection area in the light-emitting component's coordinate system to achieve automatic obstacle avoidance. The calculation for this coordinate transformation relies on the homography matrix between the camera and the optical engine.

[0069] The adjustment command includes the first obstacle target within the projection area specified by the user. Since multiple obstacle targets may exist within the projection area, obstacle avoidance needs to be confirmed and executed for each target. The first obstacle target can refer to multiple similar obstacle targets, such as multiple control switches on a wall. It can also refer to a single obstacle target. The controller performs obstacle avoidance processing based on the actual situation of the projection area. The first obstacle target is the obstacle that the user wants the projection device to avoid during operation; that is, the first obstacle target will affect the projection effect.

[0070] There is no strict order between when the user inputs the adjustment command and when the projection device responds to the obstacle avoidance command. The user can input the adjustment command to identify the obstacle to avoid while the projection device is detecting the projection area; the user can input the adjustment command immediately after the projection device is turned on; or the user can input the adjustment command after the media data projection begins and discovers that an obstacle not selected for avoidance before projection is actually affecting the projection effect, so as to avoid the obstacle.

[0071] In some embodiments, after the projection device is turned on, obstacle detection begins in the projection area according to a preset program. Simultaneously, the user outputs a voice command to the projection device, "Avoid the hooks on the wall," which the controller receives via an audio receiver. At the same time, a transformation matrix is ​​acquired for the conversion between the camera coordinate system and the light-emitting component coordinate system, providing a basis for subsequent coordinate transformation of obstacles.

[0072] Adjustment commands can be output via voice input, or via a remote control device such as a remote controller, or through the control interface on the projection device. However, voice input is the easiest method for users and enhances the interaction between the user and the projection device, thus improving the user experience. Therefore, in this embodiment, the primary method for users to output adjustment commands is through voice input; other methods can be used to assist users in outputting adjustment commands in specific scenarios. For example... Figure 10 As shown, the controller delineates a projectable area in the sampled image based on the first obstacle target.

[0073] S200: Identify the first obstacle target in the sampled image, and delineate a projectable area in the sampled image based on the first obstacle target.

[0074] The sampled image is the image captured by the camera during the optical engine's projection and correction of the image. The optical engine's projection and correction of the image aims to acquire image information of the entire projection area and obstacles indicated by the user that need to be avoided, and to establish a coordinate system for the sampled image based on the optical engine's coordinate system. With the aid of the corrected image, the presence of obstacles can be identified more accurately, and the position of the obstacle in the sampled image can be obtained through the coordinate system. The controller can then re-divide the projection area based on the obstacle's position in the sampled image, transform the divided projection area according to the homography matrix, and then project media data into the divided projection area via the optical engine.

[0075] Following the above embodiment, after receiving the adjustment command output by the user, the projection device confirms that the hook is an obstacle, that is, it determines that an obstacle exists in the projection area. After determining that an obstacle exists in the projection area, the controller controls the optical engine to project a corrected image onto the wall of the projection area. While the corrected image is projected onto the wall, the controller also controls the camera to capture a corrected image (i.e., capture the projection area). The controller confirms the position of the hook on the sampled image and, based on the sampled image after removing the hook, re-divides a new projectable area.

[0076] When dividing the projectable area, it is defined as a rectangular region with the largest preset aspect ratio that can be accommodated within the sampled image, excluding the first obstacle target. Since the primary reason for using a projection device is to achieve a higher aspect ratio for media data playback, it is essential to ensure the projectable area has the largest possible aspect ratio. Furthermore, currently only the first obstacle target is being detected; other obstacles may exist within the projection area, but the user has not yet provided relevant commands. To ensure successful projection of media data, a larger projection area needs to be reserved to accommodate further division of the projection area.

[0077] In the process of identifying obstacles using corrected images, projecting only one corrected image or projecting an identical corrected image cannot accurately identify obstacles. Therefore, the corrected image consists of two different images. By comparing the information in the two images, the obstacle and its location can be confirmed. Figure 9 As shown, the corrected image in this embodiment includes a solid color image card and a feature image card. The controller also performs the following operations in response to an obstacle avoidance command:

[0078] In response to obstacle avoidance commands, the light-emitting components are controlled to project corrected images.

[0079] like Figure 9 As shown, the corrected image includes a solid color card, which is generally a white card, while the feature card contains a feature image, the specific form of which is not limited. Comparing the feature image with the white card image helps to identify obstacles in the projection area.

[0080] When the camera is controlling the light-emitting component to project a solid color image card and a feature image card, it takes pictures of the corrected images displayed on the projection surface to obtain a first sampled image and a second sampled image.

[0081] While projecting white and feature maps using the light-emitting component, the controller cannot directly determine obstacles based on the projection effect. Therefore, a camera is needed to capture and record the state of the projection area when the white and feature maps are projected, generating a first sampled image and a second sampled image, respectively. The controller then compares the first and second sampled images to obtain relevant information about the obstacles.

[0082] The first sampled image generated by the optical engine projecting a white map will not show any content on the wall when the camera is shooting, even if there are no obstacles. It may appear as a white image that is slightly different from the wall's true color. However, if there are obstacles, the camera will also capture the obstacles on the wall. But relying solely on the content of the first sampled image is insufficient to determine the location of obstacles on the wall. When redefining the projectable area, it is difficult to perform coordinate transformations because obstacles and the areas they occupy need to be removed.

[0083] Furthermore, with sufficiently high recognition accuracy, some insects on the wall might be identified as obstacles. However, these insects are likely to move away after the white feature map is projected due to the influence of the projected light. If the controller then determines that an obstacle exists, it will unnecessarily execute the obstacle avoidance function, resulting in a deterioration in the projection effect. Therefore, it is necessary to project the feature map again to further confirm the obstacle. Moreover, the feature images on the feature map are also helpful for establishing a coordinate system. Establishing a coordinate system facilitates describing the location of obstacles and also facilitates the reconstruction of the coordinate system after the obstacle is removed, as well as the transformation to the coordinate system of the light-emitting component.

[0084] When the projection device has started projecting media data and received obstacle avoidance instructions, the light output component requires a certain transition time when switching between projecting media data and projecting solid color cards or feature cards. This increases the user's waiting time and affects the user experience. To solve this problem, the controller performs the following steps:

[0085] The system detects the moment when an obstacle avoidance command is input, as well as the projection status of the light-emitting component.

[0086] The projection device needs to execute the obstacle avoidance function immediately upon receiving the obstacle avoidance command. The obstacle avoidance function can be activated either when media data projection has already begun or before projection has started. The specific activation method can be determined by detecting the projection status of the light component. If media data projection has not yet begun, solid color images and feature images can be projected sequentially, eliminating the time-consuming process of switching between media data and solid color / feature images.

[0087] like Figure 11 As shown, if the light-emitting component is projecting media assets, a solid color image card and a feature image card are inserted into the media asset data stream to be projected.

[0088] For situations where media asset data has already started to be projected, feature cards of solid color cards can be inserted into the media asset data stream. This allows the projection process of solid color cards and feature cards to follow the playback time of the media asset data, resulting in a natural projection without the need to switch projection sources. Therefore, users do not need to wait for too long.

[0089] Based on the insertion positions of the solid color card and feature card in the media asset data stream, the shooting time is calculated, and the camera is controlled to capture the first sampled image and the second sampled image according to the shooting time.

[0090] To ensure the camera accurately captures the first and second sampled images corresponding to the solid color and feature maps, the time of receiving the obstacle avoidance command needs to be recorded to obtain the initial time. Combined with the time when the maps are subsequently inserted into the media asset data, the camera's capture time can be calculated. When the two capture times are reached, the camera executes the capture function, obtaining the first and second sampled images.

[0091] like Figure 12As shown, in some embodiments, the projection device is playing a ten-minute video. At the fifth minute, the controller receives an obstacle avoidance command input by the user and records the time of the command as five minutes. It also adds a solid color image card to the video data stream at the tenth second of the fifth minute and a feature image card at the fifteenth second of the fifth minute. Based on the time the image cards are inserted into the data stream, the controller calculates that the camera will begin its first capture ten seconds later and its second capture fifteen seconds later. At the tenth second of the fifth minute, the light-emitting component projects the solid color image card, and the camera captures a first sampled image. At the fifteenth second of the fifth minute, the light-emitting component projects the feature image card, and the camera captures a second sampled image.

[0092] As can be seen, adding image cards to the media asset data stream avoids switching of the projection source, making the process of obtaining sampled images more natural, reducing user waiting time, and improving the efficiency of obstacle avoidance. After acquiring the first and second sampled images, the first obstacle target can be identified and obstacle avoidance can be performed.

[0093] The first obstacle target is identified in the first sampled image, and a transformation matrix is ​​established based on the shape of the feature map card in the second sampled image.

[0094] By comparing the first and second sampled images, the location and coordinates of the first obstacle target can be identified. The first sampled image, free from interference from feature points on the feature image, allows for the complete identification of the first obstacle target's outline and the acquisition of its coordinates. The feature points in the second sampled image possess characteristics such as symmetry and uniform distribution, thus facilitating the establishment of a coordinate system and the acquisition of coordinates. After obtaining the coordinates, a transformation matrix between the camera coordinate system and the light-emitting component coordinate system can be further established. During the process of establishing the transformation matrix using feature points from the second image, the controller executes the following steps:

[0095] Iterate through the color values ​​of pixels in the second sampled image, and identify feature points in the second sampled image based on the color values.

[0096] An image is composed of numerous pixels, and its specific color is influenced by the color values ​​of these pixels. The color of the feature points differs from the solid color chart in the first sampled image. Since the solid color chart mostly uses white, the feature points in the feature chart can use colors that contrast sharply with white. By identifying the color values ​​of the pixels in the second sampled image—that is, by identifying color values ​​that differ from white—the feature points can be identified.

[0097] Extract the feature distribution information from the feature map corresponding to the second sampled image, and establish a homography matrix based on the feature points and feature distribution information.

[0098] Multiple feature points, with varying distributions, can yield feature patterns. Furthermore, multiple feature points facilitate the establishment of a coordinate system suitable for the current feature pattern, and further, facilitate the creation of a homography matrix for transformation between the camera coordinate system and the light-emitting component coordinate system. After extracting feature points and their distribution information, the homography matrix can be calculated using the following formula. The coordinates of the feature points in the second sampled image captured by the camera can be represented by a matrix as follows:

[0099]

[0100] (x, y, 1) are the coordinates of the feature point in the camera coordinate system.

[0101]

[0102] (x1, y1, 1) are the coordinates of the feature point in the coordinate system of the light output component.

[0103] The coordinates in the camera coordinate system are converted to the coordinates in the light-emitting component coordinate system, which can be regarded as:

[0104]

[0105] H is the homography matrix, which can be calculated using the known camera coordinates and light-emitting component coordinates. The calculation process is a general method, so it will not be explained further.

[0106] Store the homography matrix to obtain the transformation matrix.

[0107] After calculating the homography matrix H, H is stored in the storage space to form a transformation matrix for camera-light-emitting component coordinate transformation, ready for later use. The specific values ​​contained in the homography matrix may vary depending on the way the feature map coordinate system is established, but this has no specific impact on the camera-light-emitting component coordinate transformation. The actual content of the homography matrix is ​​the transformation rule from camera coordinates to light-emitting component coordinates; the transformation only needs to be implemented accordingly.

[0108] S300: Determines the projection area according to the projectable area and the transformation matrix, and controls the light output component to project the content onto the target projection area.

[0109] The projected content, or media asset data, is first divided on a sampled image captured by a camera during the process of defining the projectable area. Projecting the media asset data also requires the light-emitting component to perform related actions; therefore, the coordinates of the newly defined projectable area on the sampled image need to be converted to coordinates suitable for projection by the light-emitting component. During this conversion, a transformation matrix is ​​used to convert the coordinates on the sampled image one by one into coordinates in the coordinate system of the light-emitting component, thus forming the projection area.

[0110] Continuing with the above embodiment, after the projection device re-divides the projectable area, it immediately calls the already acquired transformation matrix, i.e., the homography matrix used for coordinate transformation. The controller converts the coordinates of the re-divided projectable area on the sampled image from the camera coordinate system to the light-emitting component coordinate system. After obtaining the projectable area coordinates in the light-emitting component coordinate system, the controller projects the media data onto the projectable area—the wall—through the light-emitting component.

[0111] When dividing the projectable area based on the first obstacle target, since a command specifying the first obstacle target is received from the user, it is necessary to identify the user-specified first obstacle target. For example... Figure 13 As shown, the controller performs corresponding actions based on the identification result of the first obstacle target:

[0112] The recognition model is invoked based on the type of the first obstacle target specified by the user, and the sampled image is input into the recognition model.

[0113] The recognition model is a neural network model trained on sample image data. The controller parses the user-inputted first obstacle target and determines its type based on keywords and other information. Then, based on the type, the recognition model is invoked to determine whether the first obstacle target exists in the sampled image. The determination of the first obstacle target does not have to be a precise value; it can be represented by a probability such as similarity. For example, after receiving the sampled image, the recognition model determines the probability that the sampled image contains the first obstacle target.

[0114] If the recognition result contains a first obstacle target, perform the step of dividing the projectable region according to the position of the first obstacle target in the sampled image.

[0115] In some embodiments, the user inputs a first obstacle target as "a hook on the wall." The controller identifies the first obstacle target by parsing the keywords "on the wall" and "hook," and then calling the "household ornament" recognition model. If the probability that the sampled image contains a hook is 90%, it can be determined that the first obstacle target—a hook—exists in the sampled image. The projectable area is then redefined based on the coordinates of the hook.

[0116] If the identification result does not contain the first obstacle target, generate the first prompt message, and control the light-emitting component to project the first prompt message.

[0117] In other embodiments, the recognition model determines that the probability of the sampled image containing the hook is 10%, thus indicating that there is no first obstacle target in the sampled image. It then generates a first prompt message to alert the user that no first obstacle target has been identified in the current projection area. This first prompt message can be presented via projection or played as audio. After receiving the prompt message, the user can re-enter the command specifying the first obstacle target, and the controller will then re-perform obstacle avoidance processing.

[0118] In addition to the first obstacle target, user-inputted adjustment commands can also include projection parameters. Projection parameters are used to define the target's projection area. For example... Figure 14 As shown, when the controller receives an adjustment command including projection parameters, it performs the following steps based on the projection parameters:

[0119] Based on the transformation matrix, the projectable region is fitted to the effective projection range.

[0120] The projectable area includes the target projection area required for projecting media data, as well as some reserved blank areas. These areas are the optimal areas calculated by the controller for the current situation. However, due to different usage scenarios and user needs, users will provide projection parameters according to actual requirements to adjust the actual projection area.

[0121] Extract projection parameters from the adjustment instructions, and delineate the target projection area within the effective projection range based on the projection parameters.

[0122] Projection parameters can directly control the aspect ratio of the image, such as "play media data in a format with a length of 2 meters and a width of 1 meter." Based on the projection parameters, the controller further adjusts the actual projection area within the effective projection range to meet the user's needs, thus obtaining the target projection area. In some cases, the projection parameters input by the user may be unclear or exceed the effective projection range. In such cases, the controller needs to determine whether the projection parameters conform to the effective projection range and perform the following steps:

[0123] Obtain the boundary dimensions of the effective projection range.

[0124] If the boundary size is greater than or equal to the specified screen size, the target projection area is defined within the effective projection range according to the specified screen size.

[0125] If the boundary size is smaller than the specified screen size, a second prompt message is generated, and the light-emitting component is controlled to project the second prompt message.

[0126] The boundary size of the effective projection range is the maximum size of the projection area. Exceeding this boundary size will affect the projection resolution and scaling, leading to distortion of the projected media data. When users input adjustment commands containing projection parameters, the parameters they input based solely on visual observation may exceed the boundary size. Therefore, when generating the target projection area based on the user-input projection parameters, the system compares the projection parameters with the boundary size to determine if a suitable target projection area can be defined according to the user's needs.

[0127] In some embodiments, the user-input adjustment command includes "the width of the projected media data is 2.56 meters and the height is 1.44 meters". After receiving the adjustment command, the controller judges the projection parameters according to the boundary dimensions of the effective projection range. If the distance projection parameters are found to be within the effective projection range, the target projection area is determined based on the projection parameters.

[0128] In other embodiments, the user-input adjustment command includes "the width of the projected media data is 3 meters and the height is 2 meters." After receiving the adjustment command, the controller judges the boundary dimensions based on the effective projection range. If it finds that the projection parameters have exceeded the effective projection range, it projects a second prompt message onto the wall via the light-emitting component: "Target projection area division failed, projection parameters out of range." After seeing the out-of-range prompt, the user can re-enter the projection parameters for further adjustment.

[0129] User-inputted adjustment commands can also include specified interval distances, such as "project media data five inches from the right side of the wardrobe" or "project media data three inches from the ground." Specifying an interval distance leaves blank areas within the effective projection range, thus further reducing the target projection area. At this point, the controller needs to add the specified interval distance to the width and height of the projected media data and then compare it with the boundary dimensions to determine whether the projection exceeds the boundary size.

[0130] Calculate the extreme value size based on the specified screen size and the specified interval distance.

[0131] Extreme dimensions include minimum width and minimum height. The minimum width is the sum of the specified width of the image within a given image size and the specified horizontal distance within a given spacing distance; the minimum height is the sum of the specified height of the image within a given image size and the specified vertical distance within a given spacing distance. Extreme dimensions are equivalent to the minimum width and minimum height that the effective projection area should have.

[0132] Obtain the effective width and effective height of the effective projection range, and then make a judgment.

[0133] If the effective width is less than the minimum width, and / or the effective height is less than the minimum height, a third prompt message is generated.

[0134] If the effective width is greater than or equal to the minimum width, and the effective height is greater than or equal to the minimum height, the target projection area is defined within the effective projection range according to the specified interval distance and the specified screen size.

[0135] In some embodiments, the controller calculates a minimum width of 4.56 meters based on the projection parameters of "projection media data width is 2.56 meters" and "horizontal spacing from the wardrobe is 2 meters" and a specified interval distance; and calculates a minimum height of 1.94 meters based on the projection parameters of "projection media data height is 1.44 meters" and "distance from the ground is 0.5 meters" and a specified interval distance. The current effective projection range has a width of 4 meters and a height of 2 meters. After the controller calls the parameters of the effective projection range, it determines that the width is out of range, the height is within range, and the boundary dimensions of the effective projection range do not conform to the user's input adjustment command. It then projects "Failed to adjust projection range according to adjustment command" through the light-emitting component to prompt the user to re-enter the adjustment command. Simultaneously, the controller will also play the message "Current maximum size is 4 meters wide and 2 meters high" to inform the user.

[0136] In other embodiments, the effective projection range has a width of 5 meters and a height of 2 meters. The controller determines that the width and height are within the range, and the boundary dimensions of the effective projection range conform to the adjustment instructions input by the user. The target projection area is then divided according to the adjustment instructions.

[0137] Enriching the adjustment instructions makes the projected media data more aligned with user needs and increases the human-computer interaction between the user and the projection device, thereby improving user satisfaction. When adjusting the target projection area according to the instructions, the image quality after the projection device projects the media data is ensured by judging the width and height. Media data is not projected if it does not meet the standard range, and is only projected if it does meet the standard range.

[0138] By identifying the first obstacle target, the projection device can project media data onto the target projection area when there are no other obstacles. However, when there are other obstacles, such as... Figure 15 As shown, the controller will also interact with the user to confirm whether other obstacles need to be avoided, and execute subsequent actions based on the user's input:

[0139] Identify the second obstacle target within the projectable area.

[0140] The projectable area at this point is the projection area redefined by the controller based on the first obstacle target. The second obstacle target is any target in the sampled image other than the first obstacle target. The category of the second obstacle target may differ from that of the first obstacle target, therefore the controller needs to continue to confirm with the user whether to execute the obstacle avoidance function. The second obstacle target can be identified autonomously by the controller or input by the user. The process of identifying the second obstacle target is the same as in the above embodiment and will not be repeated here.

[0141] If a second obstacle target exists in the projectable area, generate a fourth prompt message, and / or generate an inquiry instruction.

[0142] After confirming the presence of the second obstacle, the controller generates audio / projection information to alert the user. Simultaneously, it issues a query to the user to confirm whether obstacle avoidance is necessary.

[0143] Obtain the confirmation instruction input by the user based on the fourth prompt information and / or inquiry instructions, and redefine the projectable area according to the confirmation instruction.

[0144] The confirmation command is a user's instruction to the controller indicating whether to perform obstacle avoidance. After receiving the user's confirmation command, the controller re-divides the current projectable area into a new projectable area based on the second obstacle target, following the process described in the above embodiment.

[0145] In some embodiments, the controller, in conjunction with user-inputted adjustment commands, generates a projectable area after avoiding hooks on the wall. Simultaneously, it detects a control switch on the wall and immediately sends an audio message to the user asking, "Do you need to avoid the switch?" The user responds to the controller via voice input, indicating "I need to avoid the switch." Upon receiving this instruction, the controller divides the projectable area according to the previous method, further combining user needs and projection standards to form a target projection area, and then projects the media data onto that target projection area.

[0146] Identifying the second obstacle ensures projection quality within the target projection area. Furthermore, user interaction enhances the user experience. Understandably, users can also input a voice command to "avoid the switch," at which point the controller can directly project media data onto the target projection area.

[0147] like Figure 8 As shown, this application also provides a projection obstacle avoidance method applied to the above-mentioned projection device, which includes a light-emitting component, a camera, and a controller. The method includes:

[0148] In response to an obstacle avoidance command, a transformation matrix is ​​acquired, as well as an adjustment command input by the user. The transformation matrix is ​​a homography matrix of the coordinates between the camera and the light-emitting component. The adjustment command includes a first obstacle target specified by the user.

[0149] The first obstacle target is identified in the sampled image, which is an image captured by the camera when the light-emitting component projects a corrected image.

[0150] Based on the first obstacle target, a projectable area is defined in the sampled image. The projectable area is a rectangular area with the largest preset aspect ratio that can be accommodated by the area in the sampled image other than the first obstacle target.

[0151] The target projection area is determined according to the projectable area and the transformation matrix, and the light-emitting component is controlled to project the content onto the target projection area.

[0152] As can be seen from the above technical solution, when the projection device of this application detects an obstacle in the projection area, it also obtains the user's input instruction to specify a first obstacle target to confirm whether obstacle avoidance processing is required during projection. By capturing a sample image, the existence and location of the first obstacle target are confirmed. Specifically, the coordinates of the sample image are transformed to the coordinate system of the light-emitting component using the homography matrix between the camera and the light-emitting component. When projecting media data, the projection device re-divides the projection area according to the location of the first obstacle target and projects the media data into the re-divised projection area. When the projection device detects an obstacle, it combines the user's obstacle avoidance instruction and does not execute the automatic obstacle avoidance function if the obstacle does not affect the projection effect, thus ensuring the projection effect and improving the user experience.

[0153] Similar parts between the embodiments provided in this application can be referred to mutually. The specific implementation methods provided above are only a few examples under the overall concept of this application and do not constitute a limitation on the scope of protection of this application. For those skilled in the art, any other implementation methods extended from the solution of this application without creative effort shall fall within the scope of protection of this application.

Claims

1. A projection device, characterized in that, include: The light-emitting component is configured to project the content onto the projection surface; The camera is configured to capture sampled images; The controller is configured as follows: In response to an obstacle avoidance command, a transformation matrix is ​​acquired, as well as an adjustment command input by the user. The transformation matrix is ​​a homography matrix of the coordinates between the camera and the light-emitting component. The adjustment command includes a first obstacle target specified by the user, which is an obstacle specified by the user that the projection device needs to avoid during operation. The first obstacle target is identified in the sampled image, which is an image captured by the camera when the light-emitting component projects a corrected image. Based on the first obstacle target, a projectable area is defined in the sampled image. The projectable area is a rectangular area with the largest preset aspect ratio that can be accommodated by the area in the sampled image other than the first obstacle target. The target projection area is determined according to the projectable area and the transformation matrix, and the light-emitting component is controlled to project the content onto the target projection area. The controller is also configured to: In response to an obstacle avoidance command, the light-emitting component is controlled to project a corrected image, which includes a solid color image card and a feature image card; When the camera is controlled to project the solid color image card and the feature image card onto the light-emitting component, it takes pictures of the corrected image displayed on the projection surface to obtain a first sampled image and a second sampled image. Identify the first obstacle target in the first sampled image; The transformation matrix is ​​established based on the shape of the feature map in the second sampled image.

2. The projection device according to claim 1, characterized in that, The controller executes control over the light-emitting component to project the corrected image, and is further configured to: The projection state of the light-emitting component is detected at the moment the obstacle avoidance command is input; If the light-emitting component is projecting media assets, insert the solid color image card and the feature image card into the media asset data stream to be projected; The shooting time is calculated based on the insertion positions of the solid color image card and the feature image card in the media asset data stream; The shooting time is sent to the camera so that the camera can capture the first sampled image and the second sampled image according to the shooting time.

3. The projection device according to claim 2, characterized in that, The controller performs the acquisition of the transformation matrix and is also configured to: Iterate through the color values ​​of pixels in the second sampled image, and identify feature points in the second sampled image based on the color values; Extract the feature distribution information from the feature map corresponding to the second sampled image; Based on the identified feature points and feature distribution information, a homography matrix of coordinates between the camera and the light-emitting component is established; Store the homography matrix to obtain the transformation matrix.

4. The projection device according to claim 1, characterized in that, The controller performs the task of identifying the first obstacle target in the sampled image and is further configured to: The recognition model is invoked based on the type of the first obstacle target specified by the user. The recognition model is a neural network model trained based on sample image data. The sampled image is input into the recognition model; Obtain the recognition result output by the recognition model, wherein the recognition result is the classification probability that the sampled image contains the first obstacle target; If the recognition result contains the first obstacle target, perform the step of delineating a projectable region in the sampled image based on the first obstacle target; If the recognition result does not contain the first obstacle target, a first prompt message is generated, and the light-emitting component is controlled to project the first prompt message.

5. The projection device according to claim 1, characterized in that, The adjustment instruction also includes user-specified projection parameters. The controller executes the determination of the target projection area according to the projectable area and the transformation matrix, and is further configured to: Based on the transformation matrix, the projectable region is fitted to an effective projection range, where the effective projection range is the region of the projectable region mapped in the coordinate system of the light-emitting component. Extract the projection parameters from the adjustment instructions; The target projection area is defined within the effective projection range based on the projection parameters.

6. The projection device according to claim 5, characterized in that, The projection parameters include a specified image size. The controller, based on the projection parameters, delineates the target projection area within the effective projection range and is further configured to: Obtain the boundary dimensions of the effective projection range; If the boundary size is greater than or equal to the specified screen size, the target projection area is defined within the effective projection range according to the specified screen size; If the boundary size is smaller than the specified screen size, a second prompt message is generated, and the light-emitting component is controlled to project the second prompt message.

7. The projection device according to claim 6, characterized in that, The projection parameters include a specified interval distance. The controller, based on the projection parameters, delineates the target projection area within the effective projection range and is further configured to: The extreme size is calculated based on the specified screen size and the specified interval distance. The extreme size includes a minimum width and a minimum height. The minimum width is the sum of the specified screen width in the specified screen size and the specified horizontal distance in the specified interval distance. The minimum height is the sum of the specified screen height in the specified screen size and the specified vertical distance in the specified interval distance. Obtain the effective width and effective height of the effective projection range; If the effective width is less than the minimum width, and / or the effective height is less than the minimum height, a third prompt message is generated, and the light-emitting component is controlled to project the third prompt message; If the effective width is greater than or equal to the minimum width, and the effective height is greater than or equal to the minimum height, the target projection area is defined within the effective projection range according to the specified interval distance and the specified screen size.

8. The projection device according to claim 1, characterized in that, The controller performs the function of delineating a projectable region in the sampled image based on the first obstacle target, and is further configured to: A second obstacle target is identified in the projectable area, wherein the second obstacle target is a target in the sampled image other than the first obstacle target; If the second obstacle target exists in the projectable area, generate a fourth prompt message, and / or generate an inquiry command; Control the light-emitting component to project the fourth prompt information, and / or output the query command; Obtain the confirmation instruction input by the user based on the fourth prompt information and / or the inquiry instruction; In response to the confirmation command, the projectable area in the sampled image is redefined according to the second obstacle target.

9. A projection obstacle avoidance method, characterized in that, The method is applied to a projection device, which includes a light-emitting component, a camera, and a controller; the projection obstacle avoidance method includes: In response to an obstacle avoidance command, a transformation matrix is ​​acquired, as well as an adjustment command input by the user. The transformation matrix is ​​a homography matrix of the coordinates between the camera and the light-emitting component. The adjustment command includes a first obstacle target specified by the user, which is an obstacle specified by the user that the projection device needs to avoid during operation. The first obstacle target is identified in the sampled image, which is an image captured by the camera when the light-emitting component projects a corrected image. Based on the first obstacle target, a projectable area is defined in the sampled image. The projectable area is a rectangular area with the largest preset aspect ratio that can be accommodated by the area in the sampled image other than the first obstacle target. The target projection area is determined according to the projectable area and the transformation matrix, and the light-emitting component is controlled to project the content onto the target projection area. The method further includes: In response to the obstacle avoidance command, the light-emitting component is controlled to sequentially project a solid color image card and a feature image card as the corrected image; The camera is controlled to take a picture when the light-emitting component projects the solid color image card to obtain a first sampled image, and the camera is controlled to take a picture when the light-emitting component projects the feature image card to obtain a second sampled image; Identify the first obstacle target in the first sampled image; The transformation matrix is ​​established based on the shape of the feature map in the second sampled image.

Images