Projection equipment and projection picture processing method

Abstract

The invention discloses projection equipment and a projection picture processing method. The projection equipment comprises a light emitting assembly, a camera and a controller. The light emitting assembly is configured to project the projection content to the projection surface, the camera is configured to shoot the sampling image, the controller responds to different instructions to realize obstacle avoidance, correction, movement and other processing of the projection image, and the user experience is improved.

Description

Projection device and projection image processing method

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the China Patent Office on September 29, 2022, with application number 202211203032.2; this application claims priority to the Chinese patent application filed with the China Patent Office on September 29, 2022, with application number 202211212932.3; this application claims priority to the Chinese patent application filed with the China Patent Office on September 29, 2022, with application number 202211195978.9, the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the technical field of display devices, and in particular to a projection device and a projection image processing method. Background Art

[0004] A projector is a display device that projects images or videos onto a screen. It refracts laser light of a specific color through an optical lens assembly, creating a specific image. This allows the user to view the displayed image at a larger scale, enhancing the viewing experience.

[0005] During projection, the projector must maintain a certain distance from the screen so that the image formed on the screen fits within the focal length range of the optical lens assembly to achieve a clear image. Changes in the placement and angle of the projector, defects in the projection surface, changes in the relative position between the projection surface and the projector, or obstacles between the projector and the projection surface can all affect the projection effect and reduce the user experience.

[0006] Summary of the Invention

[0007] An embodiment of the present application provides a projection device, comprising: a light emitting component, configured to project projection content onto a projection surface; a camera, configured to capture a sample image; a controller, configured to: obtain a transformation matrix in response to an obstacle avoidance instruction, and obtain an adjustment instruction input by a user, wherein the transformation matrix is ​​a homography matrix of coordinates between the camera and the light emitting component; the adjustment instruction includes a first obstacle target specified by the user; identify the first obstacle target in a sample image, wherein the sample image is an image captured by the camera when the light emitting component projects a corrected image; define a projectable area in the sample image according to the first obstacle target, wherein the projectable area is a rectangular area with a maximum preset aspect ratio accommodated by the area in the sample image excluding the first obstacle target; determine a target projection area according to the projectable area and the transformation matrix, and control the light emitting component to project the projection content onto the target projection area.

[0008] An embodiment of the present application also provides a projection image processing method, which is applied to a projection device, wherein the projection device includes a light emitting component, a camera, and a controller; the projection image processing method includes: in response to an obstacle avoidance instruction, obtaining a transformation matrix and obtaining an adjustment instruction input by a user, wherein the transformation matrix is ​​a homography matrix of coordinates between the camera and the light emitting component; the adjustment instruction includes a first obstacle target specified by the user; identifying the first obstacle target in a sampling image, wherein the sampling image is an image captured by the camera when the light emitting component projects a corrected image; defining a projectable area in the sampling image according to the first obstacle target, wherein the projectable area is a rectangular area with a maximum preset aspect ratio accommodated by the area in the sampling image other than the first obstacle target; determining a target projection area according to the projectable area and the transformation matrix, and controlling the light emitting component to project the projection content onto the target projection area.

[0009] An embodiment of the present application also provides a projection device, including: a light emitting component, configured to project projection content onto a projection surface; a camera, configured to capture a sampling image; a controller, configured to: in response to a projection picture correction instruction, control the light emitting component to project a correction image, wherein the correction image includes a pure color image card and a feature image card; obtain a first sampling image obtained by the camera taking the pure color image card, and a second sampling image obtained by taking the feature image card; determine a characteristic contour area based on the first sampling image, wherein the characteristic contour area is the contour area with the largest area in the first sampling image or a contour area specified by the user; extract feature points in the second sampling image according to the characteristic contour area; calculate the angle between the projection surface and the light emitting component based on the feature points, and control the light emitting component to project the projection content onto the projection surface according to the angle.

[0010] An embodiment of the present application also provides a projection image processing method, which is applied to a projection device, wherein the projection device includes a light emitting component, a camera, and a controller; the projection image processing method includes: in response to a projection image correction instruction, controlling the light emitting component to project a correction image, wherein the correction image includes a pure color image card and a feature image card; obtaining a first sampling image obtained by the camera taking a photo of the pure color image card, and a second sampling image obtained by taking a photo of the feature image card; determining a characteristic contour area based on the first sampling image, wherein the characteristic contour area is the contour area with the largest area in the first sampling image or a contour area specified by the user; extracting feature points in the second sampling image according to the characteristic contour area; calculating the angle between the projection surface and the light emitting component based on the feature points, and controlling the light emitting component to project the projection content onto the projection surface according to the angle.

[0011] An embodiment of the present application also provides a projection device, including: a light-emitting component, configured to project projection content onto a projection surface; a controller, configured to: obtain a screen movement instruction input by a user, the screen movement instruction including a movement direction and a movement distance; in response to the screen movement instruction, obtain the vertex coordinates of the current light-emitting surface and the rotation matrix between the projection surface and the light-emitting surface; calculate the current projection area according to the rotation matrix and the vertex coordinates of the current light-emitting surface; calculate the vertex movement distance according to the vertex coordinates of the current projection area, the movement direction and the movement distance; calculate the target projection coordinates according to the vertex movement distance and the vertex coordinates of the current projection area; based on the rotation matrix, convert the target projection coordinates to the light-emitting surface to obtain light-emitting projection coordinates, and control the light-emitting component to project the projection content onto the projection surface according to the light-emitting projection coordinates.

[0012] An embodiment of the present application also provides a projection image processing method, which is applied to a projection device, wherein the projection device includes a light-emitting component and a controller; the projection image processing method includes: obtaining an image movement instruction input by a user, wherein the image movement instruction includes a movement direction and a movement distance; in response to the image movement instruction, obtaining the vertex coordinates of the current light-emitting surface and the rotation matrix between the projection surface and the light-emitting surface; calculating the current projection area according to the rotation matrix and the vertex coordinates of the current light-emitting surface; calculating the vertex movement distance according to the vertex coordinates of the current projection area, the movement direction and the movement distance; calculating the target projection coordinates according to the vertex movement distance and the vertex coordinates of the current projection area; based on the rotation matrix, converting the target projection coordinates to the light-emitting surface to obtain light-emitting projection coordinates, and controlling the light-emitting component to project the projection content onto the projection surface according to the light-emitting projection coordinates. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] FIG1 is a schematic diagram of a projection state of a projection device according to an embodiment of the present application;

[0014] FIG2 is a schematic structural diagram of a projection device according to an embodiment of the present application;

[0015] FIG3 is a schematic diagram of a circuit structure of a projection device according to an embodiment of the present application;

[0016] FIG4 is a schematic diagram of an optical path of a projection device according to an embodiment of the present application;

[0017] FIG5 is a schematic structural diagram of a lens according to an embodiment of the present application;

[0018] FIG6 is a schematic structural diagram of a distance sensor and a camera according to an embodiment of the present application;

[0019] FIG7 is a schematic diagram of a system framework of a projection device according to an embodiment of the present application;

[0020] FIG8 is a schematic diagram of a projection device performing automatic obstacle avoidance according to an embodiment of the present application;

[0021] FIG9 is a schematic diagram of a projection device performing automatic obstacle avoidance according to a user instruction according to an embodiment of the present application;

[0022] FIG10 is a schematic diagram of performing obstacle avoidance according to a user instruction according to an embodiment of the present application;

[0023] FIG11 is a schematic diagram of a projection device projecting a pure color chart and a feature chart according to an embodiment of the present application;

[0024] FIG12 is a schematic diagram of a projection device according to an embodiment of the present application dividing a projection area according to a first obstacle target;

[0025] FIG13 is a schematic diagram of a projection device inserting an image card into a media data stream according to an embodiment of the present application;

[0026] FIG14 is a schematic diagram of steps performed after a projection device inserts an image card into a media data stream according to an embodiment of the present application;

[0027] FIG15 is a schematic diagram of a projection device recognizing a voice command input by a user according to an embodiment of the present application;

[0028] FIG16 is a schematic diagram of a projection device according to an embodiment of the present application determining whether an adjustment instruction input by a user meets a standard range;

[0029] FIG17 is a schematic diagram of a projection device identifying a second fault target according to an embodiment of the present application;

[0030] FIG18 is a flowchart of a method for processing a projection image according to an embodiment of the present application;

[0031] FIG19 is a schematic diagram of imaging of a projection device when the projection surface is tilted according to an embodiment of the present application;

[0032] FIG20 is a schematic diagram of imaging when there is an obstacle between the projection device and the projection surface according to an embodiment of the present application;

[0033] FIG21 is a schematic diagram of a projection device performing image correction according to an embodiment of the present application;

[0034] FIG22 is a schematic diagram of a first sample image captured by a camera according to an embodiment of the present application;

[0035] FIG23 is a schematic diagram of a second sample image captured by a camera according to an embodiment of the present application;

[0036] FIG24 is a schematic diagram of extracting candidate contour regions from a second sample image according to an embodiment of the present application;

[0037] FIG25 is a schematic diagram of a projection of a light emitting component with a projection surface having concave and convex portions according to an embodiment of the present application;

[0038] FIG26 is a schematic diagram of coordinate system transformation of a camera and a light emitting assembly according to an embodiment of the present application;

[0039] FIG27 is a schematic diagram of demarcating a target projection area based on a projection image according to an embodiment of the present application;

[0040] FIG28 is a schematic diagram of demarcating a target projection area according to an obstacle avoidance instruction according to an embodiment of the present application;

[0041] FIG29 is a second flowchart of the projection image processing method according to an embodiment of the present application;

[0042] FIG30 is a schematic diagram of the projection coordinate movement of the DMD plane according to an embodiment of the present application;

[0043] FIG31 is a schematic diagram of the movement of projection coordinates of a DMD plane when the projection device performs front projection according to an embodiment of the present application;

[0044] FIG32 is a schematic diagram illustrating the movement of projection coordinates of a DMD plane when the projection device is projected sideways according to an embodiment of the present application;

[0045] FIG33 is a schematic diagram of executing screen movement according to an embodiment of the present application;

[0046] FIG34 is a schematic diagram of a process for obtaining a rotation matrix according to an embodiment of the present application;

[0047] FIG35 is a schematic diagram of a current projection area according to an embodiment of the present application;

[0048] FIG36 is a schematic diagram of a process for calculating vertex movement distance according to an embodiment of the present application;

[0049] FIG37 is a schematic diagram of a process for calculating target projection coordinates according to an embodiment of the present application;

[0050] FIG38 is a schematic diagram of a process for determining whether a maximum projection area includes target projection coordinates according to an embodiment of the present application;

[0051] FIG39 is a schematic diagram of side length vectors and connection vectors according to an embodiment of the present application;

[0052] FIG40 is a schematic diagram of a projection image after movement according to an embodiment of the present application;

[0053] FIG41 is a third flowchart of the projection image processing method according to an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose and implementation of this application clearer, the exemplary implementation of this application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of this application. Obviously, the exemplary embodiments described are only part of the embodiments of this application, not all of the embodiments. It should be noted that the brief description of the terms in this application is only for the convenience of understanding the implementation described below, and is not intended to limit the implementation of this application. Unless otherwise specified, these terms should be understood according to their ordinary and usual meanings.

[0055] A projector is a device that projects images or videos onto a screen. It can connect to computers, broadcast networks, the internet, video compact discs (VCDs), digital video discs (DVDs), game consoles, digital video cameras (DVs), and other media through various interfaces to play back video signals. Projectors are widely used in homes, offices, schools, and entertainment venues.

[0056] FIG1 is a schematic diagram of a projection state of a projection device according to an embodiment of the present application, and FIG2 is a schematic diagram of the structure of a projection device according to an embodiment of the present application.

[0057] In some embodiments, referring to Figures 1-2, the projection surface 1 is fixed at a first position, and the projection surface 1 can be a projection screen or a wall. The projection device 2 is placed at a second position so that the image it projects matches the projection surface 1. The projection device 2 includes a laser light source 100, an optical machine 200, and a lens 300, and the projection image is projected onto the projection medium 400. Among them, the laser light source 100 provides illumination for the optical machine 200, the optical machine 200 modulates the light source beam, and outputs it to the lens 300 for imaging, and projects it onto the projection medium 400 to form a projection image. Since the laser light source 100, the optical machine 200, and the lens 300 are used together to emit projection light to project the projection image, the laser light source 100, the optical machine 200, and the lens 300 are collectively referred to as the light output component 21.

[0058] In some embodiments, the laser light source 100 includes a laser assembly and an optical lens assembly. The light beam emitted by the laser assembly can pass through the optical lens assembly to provide illumination for the optical engine. The optical lens assembly requires a high level of environmental cleanliness and airtight sealing. The chamber housing the laser assembly can be sealed with a lower dustproof seal to reduce sealing costs.

[0059] In some embodiments, the light-emitting component of the projector can also be implemented by an LED light source.

[0060] In some embodiments, the optical engine 200 may be implemented to include a blue optical engine, a green optical engine, a red optical engine, and may further include a heat dissipation system, a circuit control system, and the like.

[0061] FIG3 is a schematic diagram of a circuit structure of a projection device according to an embodiment of the present application.

[0062] In some embodiments, referring to FIG3 , a projection device may include a display control circuit 10, a laser light source 20, at least one laser driving component 30, and at least one brightness sensor 40. The laser light source 20 may include at least one laser corresponding to the at least one laser driving component 30. Here, the at least one refers to one or more, and the plurality refers to two or more.

[0063] Based on this circuit architecture, the projection device can achieve adaptive adjustment. For example, by arranging a brightness sensor 40 in the light output path of the laser light source 20, the brightness sensor 40 can detect a first brightness value of the laser light source 20 and send the first brightness value to the display control circuit 10.

[0064] The display control circuit 10 can obtain a second brightness value corresponding to the driving current of each laser, and when it is determined that the difference between the second brightness value of the laser and the first brightness value of the laser is greater than a difference threshold, it is determined that the laser has suffered a catastrophic optical damage (COD) fault; the display control circuit 10 can then adjust the current control signal of the corresponding laser driving component of the laser until the difference is less than or equal to the difference threshold, thereby eliminating the COD fault of the laser. The projection device can promptly eliminate the COD fault of the laser, reduce the damage rate of the laser, and improve the image display effect of the projection device.

[0065] FIG4 is a schematic diagram of an optical path of a projection device according to an embodiment of the present application.

[0066] In some embodiments, referring to Figures 3-4, the laser light source 20 may include independently arranged blue laser 201, red laser 202 and green laser 203. The projection device may also be called a three-color projection device. The blue laser 201, the red laser 202 and the green laser 203 are all small lasers (Multi-chip LD, abbreviated as MCL), which are small in size and conducive to the compact arrangement of the optical path.

[0067] In some embodiments, the laser light source further includes an optical component 210, which is used to combine the laser light emitted by the blue laser 201, the red laser 202 and the green laser 203, and to shape and homogenize the laser light, and finally input a light beam that meets the input requirements into the optical machine.

[0068] In some embodiments, the projection device may be configured with a controller, or the projection device may be connected to the controller. The controller includes at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), a random access memory (RAM), a read-only memory (ROM), first to nth interfaces for input / output, and a communication bus.

[0069] In some embodiments, the projection device may be equipped with a camera, or the projection device may be connected to a camera for coordinated operation with the projection device to achieve adjustment and control of the projection process. The camera may be implemented as a 3D camera or a binocular camera; when the camera is implemented as a binocular camera, it specifically includes a left camera and a right camera. The binocular camera can capture the image and playback content presented by the projection screen corresponding to the projection device, that is, the projection surface. The image or playback content is projected by the built-in optical engine of the projection device.

[0070] The camera can be used to capture the image displayed on the projection surface and can be a webcam. The camera can include a lens assembly, which includes a photosensitive element and a lens. The lens refracts light through multiple lens elements, allowing the image of the scene to be illuminated by the photosensitive element. The photosensitive element can be selected based on the detection principles of a charge-coupled device or a complementary metal oxide semiconductor, depending on the camera's specifications. The light signal is converted into an electrical signal through a photosensitive material and output as image data.

[0071] FIG5 is a schematic structural diagram of a lens according to an embodiment of the present application.

[0072] To support the automatic focusing process of the projection device 2, as shown in Figure 5, the lens 300 may further include a lens assembly 310 and a drive motor 320. The lens assembly 310 is a lens group composed of one or more lenses that can refract the light emitted by the optical engine 200 so that the light emitted by the optical engine 200 can be transmitted to the projection surface to form a transmitted content image.

[0073] The lens assembly 310 may include a lens barrel and a plurality of lenses disposed within the lens barrel. Depending on whether the lens position is movable, the lenses in the lens assembly 310 may be divided into a movable lens 311 and a fixed lens 312. By changing the position of the movable lens 311 and adjusting the distance between the movable lens 311 and the fixed lens 312, the overall focal length of the lens assembly 310 may be changed. Therefore, the drive motor 320, connected to the movable lens 311 in the lens assembly 310, can drive the movable lens 311 to move its position, thereby achieving an automatic focusing function.

[0074] It should be noted that the focusing process refers to changing the position of the movable lens 311 by driving the motor 320, thereby adjusting the distance between the movable lens 311 and the fixed lens 312, that is, adjusting the image plane position. Therefore, the imaging principle of the lens combination in the lens assembly 310, adjusting the focal length is actually adjusting the image distance, but in terms of the overall structure of the lens assembly 310, adjusting the position of the movable lens 311 is equivalent to adjusting the overall focal length of the lens assembly 310.

[0075] When projection device 2 is positioned at different distances from projection surface 1, the lens of projection device 2 needs to be adjusted to different focal lengths to project a clear image on the projection surface. During projection, the distance between projection device 2 and projection surface 1 varies depending on the user's placement, requiring different focal lengths. Therefore, to adapt to different usage scenarios, projection device 2 requires adjusting the focal length of lens assembly 310.

[0076] FIG6 is a schematic structural diagram of a distance sensor and a camera according to an embodiment of the present application.

[0077] As shown in Figure 6, the projection device 2 may also include a built-in or external camera 22. The camera 22 can capture the image projected by the projection device 2 to obtain a projected content image. The projection device 2 then performs a clarity test on the projected content image to determine whether the current lens focal length is appropriate, and adjusts the focal length if it is not appropriate. When automatically focusing based on the projected content image captured by the camera 22, the projection device 2 can continuously adjust the lens position and take photos. By comparing the clarity of the images taken before and after the focus position, the focus position is determined, thereby adjusting the movable lens 311 to the appropriate position. For example, the controller 23 can first control the drive motor 320 to gradually move the movable lens 311 from the focus starting position to the focus ending position, while continuously capturing the projected content image through the camera 22 during this period. The controller 23 can then perform clarity tests on multiple projected content images to determine the position with the highest clarity. Finally, the controller 23 can control the drive motor 320 to adjust the movable lens 311 from the focus end position to the position with the highest clarity, completing the automatic focus.

[0078] When the projection device moves, its projection angle and distance to the projection surface change, causing the projected image to deform. The projected image will appear as a trapezoidal image or other deformed image. The controller can achieve automatic trapezoidal correction based on the image captured by the camera by coupling the angle between the optical machine projection surfaces and the correct display of the projected image.

[0079] FIG7 is a schematic diagram of a system framework of a projection device according to an embodiment of the present application.

[0080] In some embodiments, referring to FIG7 , the projection device has the characteristics of a telephoto micro-projector, and its controller can control the display of the projected image through a preset algorithm to achieve functions such as automatic trapezoidal correction of the display screen, automatic screen entry, automatic obstacle avoidance, automatic focusing, and anti-eye-shooting.

[0081] In some embodiments, the projection device is equipped with a gyroscope sensor; when the device is moving, the gyroscope sensor can sense position movement and actively collect movement data; the collected data is then sent to the application service layer through the system framework layer to support the application data required during user interface interaction and application interaction. The collected data can also be used for data calls by the controller in the algorithm service implementation.

[0082] In some embodiments, the projection device is equipped with a time-of-flight sensor. After the time-of-flight sensor collects the corresponding data, it is sent to the corresponding time-of-flight service of the service layer; after the above-mentioned time-of-flight service obtains the data, it sends the collected data to the application service layer through the process communication framework, and the data will be used for interactive use such as controller data calls, user interfaces, and program applications.

[0083] In some embodiments, the projection device is configured with a camera for capturing images, which may be a binocular camera, a depth camera, or a 3D camera, etc.; the camera capture data will be sent to the camera service, and then the camera service will send the captured image data to the process communication framework, and / or the projection device correction service; the projection device correction service can receive the camera capture data sent by the camera service, and the controller can call the corresponding control algorithm in the algorithm library according to the different functions to be implemented.

[0084] In some embodiments, data is interacted with the application service through the process communication framework, and then the calculation results are fed back to the correction service through the process communication framework; the correction service sends the obtained calculation results to the projection device operating system to generate control signals, and sends the control signals to the optical machine control driver to control the optical machine working conditions and realize automatic correction of the displayed image.

[0085] In some embodiments, when an image correction instruction is detected, the projection device 2 can correct the projected image. For projected image correction, a pre-established relationship between distance, horizontal angle, and offset angle can be established. The controller 23 in the projection device 2 then determines the angle between the light-emitting component 21 and the projection surface at that moment by obtaining the current distance between the light-emitting component and the projection surface and, based on the associated relationship, determines the angle between the light-emitting component 21 and the projection surface at that moment, thereby achieving projected image correction. Specifically, the aforementioned angle refers to the angle between the central axis of the light-emitting component 21 and the projection surface.

[0086] In some embodiments, the projection device 2 automatically completes the calibration and refocuses to detect whether the automatic focus function is turned on; when the automatic focus function is not turned on, the automatic focus service is terminated; when the automatic focus function is turned on, the projection device 2 will obtain the detection distance of the time of flight sensor through the middleware for calculation.

[0087] In some embodiments, the projection device uses an automatic focusing algorithm and its configured laser rangefinder to obtain the current object distance to calculate the initial focal length and search range. The projection device then drives the camera to take a picture and uses a corresponding algorithm to evaluate the clarity. Within the search range, the projection device uses the search algorithm to find the best possible focal length. The projector then repeats the above steps of taking a picture and evaluating the clarity, ultimately finding the optimal focal length through clarity comparison, completing automatic focusing.

[0088] For example, after the projection device is started, the user moves the device; the projection device automatically completes the calibration and refocuses, and the controller will detect whether the autofocus function is turned on; when the autofocus function is not turned on, the controller will end the autofocus service; when the autofocus function is turned on, the projection device will obtain the detection distance of the time of flight sensor through the middleware for calculation.

[0089] Based on the acquired distance, the controller queries a preset mapping table to obtain the focal length of the projection device. The middleware then sets the acquired focal length to the projection device's light-emitting component. After the light-emitting component emits laser light at the aforementioned focal length, the camera executes the photo command. The controller determines whether the projection device's focus is complete based on the acquired captured image and the evaluation function. If the result meets the preset completion conditions, the control automatically controls the focus adjustment process to end. If the result does not meet the preset completion conditions, the middleware fine-tunes the focal length parameters of the projection device's light-emitting component, for example, by gradually fine-tuning the focal length with a preset step size, and then sets the adjusted focal length parameters back to the light-emitting component. This allows for repeated photo-taking and clarity evaluation steps, ultimately completing automatic focus by finding the optimal focal length through clarity comparison.

[0090] In some embodiments, as shown in Figure 8 , while projecting media data onto a wall (projection surface 1), projection device 2 detects a control switch (obstacle 3) on the wall. After identifying the control switch, projection device 2 determines that it is obstacle 3, which will affect the projection effect, and therefore activates the automatic obstacle avoidance function. After the automatic obstacle avoidance function is executed, projection device 2 re-divides the projection area on the wall (projection surface 1) to avoid the control switch (obstacle 3) and projects the media data into the projection area. However, it can be seen that the area of ​​the re-divided projection area is smaller than the original projection area.

[0091] When projecting media data, projection device 2 can select from a variety of projection ratios based on the defined projection area to achieve the best projection effect. These multiple projection ratios can also meet various user needs. However, automatic obstacle avoidance reduces the size of the projection area, directly reducing the variety of projection ratios available when projecting media data, and thus failing to achieve the optimal projection effect.

[0092] In view of this, as shown in Figures 9 and 10, this application provides an embodiment of a projection device, including a light emitting component 21, a camera 22, and a controller 23. The light emitting component 21 is used to project the projection content onto the projection surface, and the camera 22 is used to capture the sample image. When an obstacle is identified in the projection area, the camera 22 also obtains the user's input instructions regarding the obstacle information, and determines whether it is necessary to avoid the obstacle in the projection area in combination with the user's input instructions. The controller 23 is configured to:

[0093] S100: In response to an obstacle avoidance instruction, obtaining a transformation matrix and obtaining an adjustment instruction input by a user.

[0094] The transformation matrix is ​​the homography matrix for the coordinates between camera 22 and light output assembly 21. It is used to convert the coordinates of objects at the same location in the projection area between the camera coordinate system and the light output assembly coordinate system. When an obstacle 3 appears, camera 22 first captures an image of the projection area to confirm the location of obstacle 2. Then, based on the obstacle's location and the projection area required by the projected media data, controller 23 controls light output assembly 21 to automatically avoid the obstacle.

[0095] In the image captured by camera 22, obstacle 3 is located in the camera's coordinate system, and its coordinates are determined based on the camera's coordinate system. When light output component 21 projects media data into the projection area, the coordinates of the media data's distribution within the projection area are determined based on the light output component's coordinate system. Therefore, the position of obstacle 3 in the projection area in the camera's coordinate system must be converted to the light output component's coordinate system. Controller 23 automatically avoids obstacles by identifying the presence of obstacle 3 within the projection area in the light output component's coordinate system. This coordinate conversion is calculated based on the homography matrix between the camera and the optical engine.

[0096] The adjustment instruction includes a user-specified first obstacle target within the projection area. Since multiple obstacles can exist within the projection area, obstacle avoidance must be confirmed and executed for each of them. The first obstacle target can refer to multiple obstacles of the same type, such as multiple control switches on a wall. Alternatively, it can refer to a single obstacle target, with the controller 23 performing obstacle avoidance based on the actual conditions within the projection area. The first obstacle target is the obstacle that the user wants the projection device to avoid during operation; that is, the first obstacle target can affect the projection effect.

[0097] There is no order restriction on the time when the user inputs the adjustment instruction and the time when the projection device responds to the obstacle avoidance instruction. The user can input the adjustment instruction to determine the obstacles to be avoided while the projection device is detecting the projection area; the user can also input the adjustment instruction immediately after the projection device is turned on to determine the obstacles to be avoided; or the user can input the adjustment instruction after starting to project media data and find that an obstacle that was not selected to be avoided before projection actually affects the projection effect, and thus input the adjustment instruction to avoid the obstacle.

[0098] In some embodiments, after the projection device is turned on, it begins obstacle detection in the projection area according to a preset program. Simultaneously, the user issues a voice command to the projection device, "Avoid the hook on the wall," and the controller 23 receives the user's command via the audio receiving device. Simultaneously, the transformation matrix used to convert between the camera coordinate system and the light output component coordinate system is obtained, providing a basis for calculating the coordinate transformation of subsequent obstacles.

[0099] Adjustment commands can be issued via voice input, a remote control device (e.g., a remote controller), or a control interface on the projector. However, voice input is much easier for users to implement and enhances the user experience by enhancing the user experience. Therefore, voice input is the primary method for users to issue adjustment commands, while other methods can assist users in issuing adjustment commands in specific scenarios.

[0100] S200: Identify a first obstacle target in a sample image, and define a projectable area in the sample image according to the first obstacle target.

[0101] As shown in Figure 12, the sampling image is the image captured by the camera 22 when the light emitting component 21 projects the correction image. The correction image projected by the light emitting component 21 is to obtain image information of the entire projection area and the obstacles that the user points out need to be avoided, and to establish a coordinate system for the sampling image based on the light emitting component coordinate system. With the assistance of the correction image, the existence of obstacles can be more accurately identified, and then the position of the obstacle in the sampling image can be obtained by coordinating the coordinate system. The controller 23 can re-divide the projection area according to the position of the obstacle in the sampling image, and transform the divided projection area according to the homography matrix, and then control the light emitting component 21 to project media data to the divided projection area.

[0102] Continuing with the above embodiment, after receiving the adjustment command input by the user, the projection device confirms that the hook is an obstacle, i.e., determines that an obstacle exists within the projection area. After determining that an obstacle exists within the projection area, controller 23 controls light output assembly 21 to project a corrected image onto the wall of the projection area. When the corrected image is projected onto the wall, controller 23 also controls camera 22 to capture a sample image (i.e., capture the projection area). Controller 23 confirms the position of the hook on the sample image and, based on the sample image after the hook is removed, re-divides the projection area into a new area.

[0103] When dividing the projectable area, the projectable area is a rectangular area with the maximum preset aspect ratio that can be accommodated by the area in the sampled image excluding the first obstacle target. Since the main reason for using a projection device is to obtain a media data playback effect with a larger aspect ratio, it is necessary to ensure that the projectable area has a larger aspect ratio as much as possible. In addition, currently only the first obstacle target is detected, and there may be other obstacle targets in the projection area, but the current user has not issued relevant instructions. In order to ensure that the media data can be projected smoothly, it is also necessary to reserve a larger projection area to cope with the continued division of the projection area.

[0104] In the process of identifying obstacles through the correction image, only projecting a single correction image or projecting the same correction image cannot accurately identify the obstacle. Therefore, the correction image includes two different images. By comparing the information on the two images, the obstacle and its location can be confirmed. As shown in Figure 11, the correction image in the embodiment of the present application includes a solid color image card and a feature image card. After responding to the obstacle avoidance instruction, the controller 23 also performs the following operations:

[0105] In response to the obstacle avoidance instruction, controlling the light emitting component to project the correction image;

[0106] The camera is controlled to take pictures of the corrected image displayed on the projection surface when the light emitting component projects the pure color image card and the feature image card respectively, so as to obtain a first sampling image and a second sampling image.

[0107] As shown in Figure 11, the solid-color chart included in the correction image is typically white, while the feature chart includes a feature image on its screen. The specific form of the feature image is not limited. Comparing the feature image with the white chart image facilitates the detection of obstacles in the projection area.

[0108] While the light output assembly projects the white card and the characteristic card, the controller cannot directly determine obstacles based on the projection results. Therefore, a camera is required to capture and record the projection area during the projection of the white card and the characteristic card, generating a first sample image and a second sample image, respectively. The controller then compares the first and second sample images to obtain information about obstacles.

[0109] The first sampled image, generated by the light-emitting component projecting a white chart, will not show any content on the wall when captured by the camera if there are no obstacles on the wall. Instead, it may appear as a white image that is somewhat different from the wall's natural color. However, if there are obstacles, the camera will still capture them. However, the content of the first sampled image alone is insufficient to determine the location of obstacles on the wall. When re-defining the projectable range, the obstacles and the area surrounding them must be removed, making coordinate conversion difficult.

[0110] Furthermore, with sufficiently high recognition accuracy, some insects on the wall may be identified as obstacles. However, after projecting the white image card, the insects are likely to move away due to the influence of the projected light. If the controller then determines that an obstacle is present, it will unnecessarily perform obstacle avoidance, resulting in a poor projection effect. Therefore, it is necessary to project the feature card again to further confirm the obstacle. The feature image on the feature card also facilitates the establishment of a coordinate system. This establishment of a coordinate system facilitates the description of the obstacle's location, the reconstruction of the coordinate system after the obstacle is removed, and the conversion to the light output component's coordinate system.

[0111] If the projection device has already started projecting media data and received an obstacle avoidance instruction, the light emitting component needs a certain amount of conversion time when switching between projecting media data and projecting pure color image cards or feature image cards. This will increase the user's waiting time and affect the user's experience. To solve this problem, the controller 23 performs the following steps:

[0112] Detect the moment when the obstacle avoidance command is input and the projection status of the light-emitting component.

[0113] The projection device must immediately execute the obstacle avoidance function upon receiving the obstacle avoidance command. The obstacle avoidance function can be enabled while media data projection is already in progress, or before it begins. Specifically, the current projection state can be determined by detecting the projection status of the optical component. If media data projection has not yet begun, the solid color image card and the feature image card can be projected sequentially, eliminating the need to waste time switching from media data to solid color image card and feature image card.

[0114] As shown in FIG13 , if the light emitting component is projecting a media image, a pure color image card and a feature image card are inserted into the media data stream to be projected.

[0115] In the case where media data has already been projected, you can insert a feature image card of a solid color image card into the media data stream so that the process of projecting the solid color image card and the feature image card conforms to the playback time of the media data and is projected naturally without changing the projection source, so the user does not need to wait too long.

[0116] The shooting time is calculated according to the insertion positions of the pure color image card and the feature image card in the media data stream, and the camera is controlled to shoot according to the shooting time to obtain the first sampling image and the second sampling image.

[0117] To ensure the camera accurately captures the first and second sample images corresponding to the solid-color and feature cards, the time the obstacle avoidance command is received must be recorded to determine the initial time. This time, combined with the time the subsequent card is inserted into the media data, can be used to calculate the camera's capture time. When the time reaches the two capture times, the camera executes the capture function, obtaining the first and second sample images.

[0118] As shown in FIG14 , in some embodiments, the projection device is playing a ten-minute video. At the fifth minute of the video, the controller receives an obstacle avoidance instruction input by the user, records the obstacle avoidance instruction time as five minutes, and adds a solid color image card to the fifth minute and tenth second of the video data stream, and adds a feature image card to the fifth minute and fifteenth second. The controller also calculates that the camera starts the first shot after ten seconds and the second shot after fifteen seconds based on the time when the image cards are inserted into the data stream. When the video plays to five minutes and ten seconds, the light-emitting component projects the solid color image card and the camera takes a photo to obtain the first sampled image. When the video plays to five minutes and fifteen seconds, the light-emitting component projects the feature image card and the camera takes a photo to obtain the second sampled image.

[0119] As can be seen, by adding image cards to the media data stream, switching projection sources is avoided, making the process of obtaining sample images more natural, reducing user waiting time, and improving the efficiency of obstacle avoidance. After obtaining the first and second sample images, the first obstacle target can be identified and obstacle avoidance can be performed.

[0120] A first obstacle target is identified in the first sampling image, and a transformation matrix is ​​established according to the shape of the feature map in the second sampling image.

[0121] By comparing the first sampling image and the second sampling image, the position and coordinates of the first obstacle target can be identified. In the first sampling image, there is no interference from the feature points on the feature image, so the outline of the first obstacle target can be fully identified and its coordinates can be obtained. The feature points on the second sampling image have characteristics such as symmetry and uniform distribution, which is conducive to establishing a coordinate system to obtain coordinates. After obtaining the coordinates, the transformation matrix between the camera coordinate system and the light output component coordinate system can be further established. In the process of establishing the transformation matrix using the feature points on the second image, the controller 23 performs the following steps:

[0122] The color values ​​of the pixels in the second sampled image are traversed, and feature points are identified in the second sampled image according to the color values.

[0123] An image is composed of multiple pixels, and its specific color is also affected by the color values ​​of multiple pixels. The color of the feature point is different from the solid color chart in the first sampled image. Solid color charts are mostly white, so the feature points in the feature chart can use colors that contrast sharply with white. Feature points can be identified by identifying the color values ​​of the pixels in the second sampled image, that is, by identifying color values ​​that differ from white.

[0124] Extract feature distribution information from the feature map corresponding to the second sampled image, and establish a homography matrix based on the feature points and the feature distribution information.

[0125] Multiple feature points can be used to generate a feature pattern based on their different distributions. Multiple feature points are also helpful in establishing a coordinate system suitable for the current feature pattern, further facilitating the establishment of a homography matrix for transforming between the camera coordinate system and the light-emitting component coordinate system. After extracting the feature points and feature distribution information, the following formula can be used to determine the homography matrix. The feature point coordinates in the second sample image captured by the camera can be expressed in a matrix as follows:

[0126] (x, y, 1) is the coordinate of the feature point in the camera coordinate system.

[0127] (x1, y1, 1) is the coordinate of the feature point in the light emitting component coordinate system.

[0128] The coordinates in the camera coordinate system are converted to the coordinates in the light emitting component coordinate system, which can be regarded as:

[0129] H is the homography matrix, which can be calculated using the known camera coordinates and light-emitting component coordinates. The calculation process is a general method, so it will not be further explained.

[0130] Store the homography matrix to get the transformation matrix.

[0131] After calculating the homography matrix H, store it in memory, forming the transformation matrix for the camera-to-light-emitting component coordinate transformation, ready for future use. The specific values ​​contained in the homography matrix may vary depending on how the feature map coordinate system is established, but this has no specific impact on the camera-to-light-emitting component coordinate transformation. The actual content of the homography matrix is ​​the conversion rules for converting camera coordinates to light-emitting component coordinates, and the corresponding conversion can be implemented.

[0132] S300: determining a projection area according to the projectable area and the transformation matrix, and controlling a light emitting component to project the projection content onto a target projection area.

[0133] The projected content is the media data. When dividing the projectable area, the projection device first divides it on the sampled image captured by the camera. Projecting the media data requires the light-emitting component to perform relevant actions. Therefore, the coordinates of the projectable area, which have been re-divided on the sampled image, must be converted to coordinates of the projectable area suitable for projection by the light-emitting component. During this conversion process, the coordinates on the sampled image are converted one by one into coordinates in the light-emitting component's coordinate system by calling a transformation matrix to form the projection area.

[0134] Continuing with the above embodiment, after redividing the projectable area, the projection device immediately calls the acquired transformation matrix, i.e., the homography matrix for coordinate conversion. The controller converts the coordinates of the redivided projectable area on the sampled image from the camera coordinate system to the light output component coordinate system. After obtaining the coordinates of the projectable area in the light output component coordinate system, the controller projects the media data onto the projectable area (e.g., a wall) through the light output component.

[0135] When dividing the projection area according to the first obstacle target, since the command of the user specifying the first obstacle target is received, it is necessary to identify the first obstacle target specified by the user. As shown in Figure 15, the controller performs corresponding actions according to the recognition result of the first obstacle target:

[0136] The recognition model is called according to the type of the first obstacle target specified by the user, and the sampled image is input into the recognition model.

[0137] The recognition model is a neural network model trained using sample image data. The controller parses the user's input of the first obstacle target and determines its type based on keywords and other information. Based on the type, the recognition model is then called to determine whether the first obstacle target exists in the sampled image. The determination of the first obstacle target can be expressed not as a precise value but as a probability, such as similarity. For example, after receiving the sampled image, the recognition model determines the classification probability that the sampled image contains the first obstacle target.

[0138] If the recognition result includes the first obstacle target, the step of dividing the projectable area according to the position of the first obstacle target in the sample image is performed.

[0139] For example, if the user inputs the first obstacle as "hook on the wall," the controller will identify the first obstacle by parsing the keywords "on the wall" and "hook" and calling the "household accessories" recognition model. The controller will determine that the sampled image contains a hook with a 90% probability, thus determining that the first obstacle—the hook—is present in the sampled image. The projectable area is then re-divided based on the hook's coordinates.

[0140] If the recognition result does not include the first obstacle target, first prompt information is generated, and the light emitting component is controlled to project the first prompt information.

[0141] In some embodiments, the recognition model determines that the sample image contains a hook with a probability of 10%, determines that the first obstacle target does not exist in the sample image, and generates a first prompt message to remind the user that the first obstacle target has not been identified in the current projection area. The first prompt message can be presented via projection or audio. After receiving the prompt message, the user can re-enter a command to specify the first obstacle target, and the controller will re-perform obstacle avoidance processing.

[0142] In addition to the first obstacle target, the adjustment instruction input by the user may also include projection parameters. The projection parameters are used to define the target projection area. As shown in Figure 16, when the controller receives the adjustment instruction including the projection parameters, it performs the following steps according to the projection parameters:

[0143] According to the transformation matrix, the projectable area is fitted into the valid projection range.

[0144] The projectable area includes the target projection area required to project the media data, as well as some reserved blank areas. These areas are the optimal area calculated by the controller for the current situation. However, due to different usage scenarios and user needs, users may set projection parameters based on their actual needs to adjust the actual projection area.

[0145] The projection parameters are extracted from the adjustment instruction, and the target projection area is delineated within the effective projection range according to the projection parameters.

[0146] Projection parameters can directly control the aspect ratio, for example, "play media data in a format of 2 meters in length and 1 meter in width." Based on the projection parameters, the controller further adjusts the actual projection range within the valid projection range to meet the user's requirements, resulting in the target projection area. In some cases, the projection parameters entered by the user may be unclear or exceed the valid projection range. The controller needs to determine whether the projection parameters meet the valid projection range and perform the following steps:

[0147] Gets the bounding dimensions of the effective projection range.

[0148] If the boundary size is greater than or equal to the specified screen size, the target projection area is delineated within the effective projection range according to the specified screen size.

[0149] If the border size is smaller than the specified screen size, second prompt information is generated, and the light emitting component is controlled to project the second prompt information.

[0150] The effective projection range's bounding dimensions define the maximum size of the projection area. Exceeding these dimensions will affect projection resolution and scaling, resulting in distortion of the projected media data. When users enter adjustment commands that include projection parameters, simply visually observing and inputting projection parameters based on their own needs may result in the projection exceeding these bounding dimensions. Therefore, when generating a target projection area based on user-entered projection parameters, the projection parameters are compared with the bounding dimensions to determine whether an appropriate target projection area can be created based on the user's needs.

[0151] In some embodiments, the adjustment instruction input by the user includes "the width of the projected media data is 2.56 meters and the height is 1.44 meters". After receiving the adjustment instruction, the controller judges the projection parameters according to the boundary size of the effective projection range. If it is found that the distance projection parameters are consistent with the effective projection range, the target projection area is determined according to the projection parameters.

[0152] In some embodiments, the user enters an adjustment instruction including "The width of the projected media data is 3 meters and the height is 2 meters." Upon receiving the adjustment instruction, the controller determines the boundary dimensions based on the effective projection range. If it finds that the projection parameters exceed the effective projection range, the controller projects a second prompt message, "Failed to divide the target projection area. Projection parameters exceed the range," onto the wall through the light output assembly. After seeing the out-of-range prompt, the user can re-enter the projection parameters to adjust them again.

[0153] User-entered adjustment commands can also include a specified spacing distance, for example, "Project media data five inches from the right side of the wardrobe" or "Project media data three inches from the ground." This specified spacing distance leaves a blank area within the effective projection range, further reducing the target projection area. To determine whether the boundary dimensions are exceeded, the controller adds the specified spacing distance to the width and height of the projected media data and then compares the result with the boundary dimensions.

[0154] Calculates the extreme size based on the specified screen size and the specified interval distance.

[0155] The extreme dimensions include the minimum width and minimum height. The minimum width is the sum of the specified image width in the specified image size and the horizontal distance in the specified interval distance. The minimum height is the sum of the specified image height in the specified image size and the vertical distance in the specified interval distance. The extreme dimensions correspond to the minimum width and minimum height of the effective projection area.

[0156] Get the effective width and effective height of the effective projection range and make a judgment.

[0157] If the effective width is smaller than the minimum width, and / or the effective height is smaller than the minimum height, a third prompt message is generated.

[0158] If the effective width is greater than or equal to the minimum width, and the effective height is greater than or equal to the minimum height, the target projection area is delineated within the effective projection range according to the specified interval distance and the specified screen size.

[0159] In some embodiments, the controller calculates the minimum width to be 4.56 meters based on the projection parameters of "the width of the projected media data is 2.56 meters" and "the horizontal distance from the wardrobe is 2 meters" and the specified interval distance; and calculates the minimum height to be 1.94 meters based on the projection parameters of "the height of the projected media data is 1.44 meters" and "0.5 meters from the ground" and the specified interval distance. The current effective projection range has a width of 4 meters and a height of 2 meters. After the controller calls the parameters of the effective projection range, it determines that the width exceeds the range and the height is within the range. The boundary size of the effective projection range does not meet the adjustment instructions entered by the user, and the light-emitting component projects "Failed to adjust the projection range according to the adjustment instructions" to prompt the user to re-enter the adjustment instructions. At the same time, the user will also be prompted by the message "The current maximum size is 4 meters in width and 2 meters in height."

[0160] In some embodiments, the effective projection range is 5 meters wide and 2 meters high. The controller determines that the width and height are within the range, and the boundary size of the effective projection range meets the adjustment instructions input by the user, and divides the target projection area according to the adjustment instructions.

[0161] By enriching the content of adjustment instructions, projected media data better meets user needs and increases the human-computer interaction process between the user and the projection device, which is conducive to improving user satisfaction. When adjusting the target projection area according to the adjustment instructions, the width and height are judged to ensure the image quality of the media data after the projection device projects it. If the media data does not meet the standard range, it will not be projected. If it does meet the standard range, the media data will be projected.

[0162] By identifying the first obstacle target, if there are no other obstacles, the projection device can project media data to the target projection area. If there are other obstacles, as shown in Figure 17, the controller will also interact with the user to confirm whether other obstacles need to be avoided and perform subsequent actions based on the user's input instructions:

[0163] A second obstacle target is identified in the projectable area.

[0164] The projection area at this point is the projection area re-divided by the controller based on the first obstacle target. The second obstacle target is an object in the sampled image other than the first obstacle target. The second obstacle target's category can be different from the first obstacle target, so the controller needs to confirm with the user whether to execute the obstacle avoidance function. The second obstacle target can be identified by the controller itself or input by the user. The process for identifying the second obstacle target is the same as in the above embodiment and will not be repeated here.

[0165] If there is a second obstacle target in the projection area, fourth prompt information is generated, and / or a query instruction is generated.

[0166] After confirming the presence of the second obstacle target, the controller generates audio / projection information to prompt the user of the presence of the second obstacle target. At the same time, it also issues a query instruction to the user to confirm whether obstacle avoidance processing is required.

[0167] A confirmation instruction input by the user based on the fourth prompt information and / or the inquiry instruction is obtained, and the projection area is redefined according to the confirmation instruction.

[0168] The confirmation instruction is an instruction input by the user to the controller to determine whether to perform obstacle avoidance processing. After receiving the user's instruction to confirm obstacle avoidance, the controller re-divides the current projection area into new projection areas according to the second obstacle target according to the process of the above embodiment.

[0169] In some embodiments, the controller generates a projection area by avoiding hooks on the wall, in response to user input. Simultaneously, the controller identifies a control switch on the wall and immediately issues an audio message to the user, asking, "Do you need to avoid the switch?" The user then voice-informs the controller, "Do you need to avoid the switch?" Upon receiving the "Need to avoid the switch" instruction, the controller divides the projection area into new areas based on the projection area division method. Furthermore, the controller combines user needs with projection standards to form a target projection area, and then projects the media data into the target projection area.

[0170] Identifying the second obstacle ensures projection quality within the target projection area. This also enhances the user experience through user interaction. It's understood that users can also use voice input to indicate "no need to avoid the switch," at which point the controller will project media data directly into the target projection area.

[0171] Based on the same inventive concept, an embodiment of the present application further provides a projection image processing method, which is applied to the above-mentioned projection device. The projection device includes a light output component, a camera, and a controller. As shown in FIG18 , the projection image processing method includes:

[0172] S100: Responding to an obstacle avoidance instruction, obtaining a transformation matrix and obtaining an adjustment instruction input by a user;

[0173] The transformation matrix is ​​the homography matrix of the coordinates between the camera and the light-emitting component; and the adjustment instruction includes the first obstacle target specified by the user.

[0174] S200: identifying a first obstacle target in the sampled image;

[0175] The sampled image is an image captured by the camera when the light-emitting component projects the corrected image.

[0176] S300: Delimiting a projectable area in the sampled image according to the first obstacle target;

[0177] The projectable area is a rectangular area with a maximum preset aspect ratio that is contained in the area other than the first obstacle target in the sampled image.

[0178] S400: determining a target projection area according to the projectable area and the transformation matrix, and controlling the light emitting component to project the projection content onto the target projection area.

[0179] When the projection device recognizes that there is an obstacle in the projection area, it also obtains the user's input instruction for specifying the first obstacle target to confirm whether obstacle avoidance is required during the projection process. By shooting a sampled image, it is confirmed whether the first obstacle target exists and its position when it exists. Among them, the coordinates of the sampled image are converted to the light-emitting component coordinate system through the homography matrix of the coordinates between the camera and the light-emitting component. When projecting media data, the projection device re-divides the projection area according to the position of the first obstacle target and projects the media data to the re-divided projection area. When the projection device recognizes an obstacle, it combines the user's obstacle avoidance instruction and does not execute the automatic obstacle avoidance function when the obstacle does not affect the projection effect, thereby ensuring the projection effect and improving the user experience.

[0180] In some embodiments, as shown in Figures 19 and 20, when a user uses the projection device 2 to project content, due to the complexity of the environment, it is inevitable that the light-emitting component of the projection device is not perpendicular to the projection wall or screen (projection surface 1), causing the projection image to be deformed, and there are obstacles in the projection area, causing the projection image to be blocked by the obstacles, which greatly affects the user experience.

[0181] To address this, projection equipment is typically designed with automatic obstacle avoidance and automatic correction features. During projection, the projection equipment needs to automatically detect obstacles to avoid them and use automatic correction to re-correct the projected image. The automatic correction feature utilizes the feature card projected without determining the spatial position of the feature points. Instead, the camera 22 captures the feature points, identifies them, and then 3D reconstructs the projection surface to achieve automatic correction.

[0182] However, if there is more than one projection wall, or some feature points are projected onto other objects rather than a plane, the projection will be projected onto multiple areas. If feature points are recognized in multiple areas, a fitting plane error will occur, causing the automatically corrected shape to no longer be a rectangle, resulting in an incorrect correction result.

[0183] In order to be able to project a corrected projection image even when the projection surface 1 is blocked by obstacles during the projection process, the embodiment of the present application further provides a projection device, as shown in FIG21 , which may include a light emitting component 21, a camera 22, and a controller 23. The light emitting component 21 is used to project the projection content onto the projection surface. The camera 22 is used to capture the sample image. The controller 23 is configured to:

[0184] S100: In response to a projection image correction instruction, controlling the light emitting component to project a correction image.

[0185] The projection device can receive projection image correction instructions input by the user and control the light-emitting component to project a corrected image based on the projection image correction instructions. The projection image correction instructions can be issued by the user through a button on the projection device's control device (such as a remote control) or through a mobile terminal (such as a smartphone or laptop) that has established a communication connection with the projection device.

[0186] The corrected image includes a pure color chart and a feature chart. When the corrected image is a pure color chart, the projection device projects a pure color projection image onto the projection surface through the light emitting component 21. When the light emitting component 21 projects the pure color chart onto the projection surface, some of the light from the pure color chart will illuminate the obstacle between the light emitting component 21 and the projection surface. A shadow area will appear at the corresponding position on the projection surface due to the obstruction caused by the obstacle. The controller 23 can identify the shadow area in the pure color chart to identify the location of the obstacle and perform subsequent obstacle avoidance functions based on the location of the obstacle.

[0187] It should be noted that in some embodiments, to more clearly identify shadow areas, the solid color chart should be a light color, such as light yellow, light blue, white, gray, etc. A light color is defined as a color less than or equal to 1 / 12 of the standard color depth of a dye. The present embodiment does not impose any other restrictions on the color of the solid color chart.

[0188] When the correction image is a characteristic image card, the light emitting component 21 projects a projection image with several characteristic points onto the projection surface. The characteristic points represent the characteristics of a preset area range of the projection image and are used to perform the automatic correction function on the projection image later.

[0189] S200: Acquire a first sampled image obtained by photographing a pure color chart with a camera, and a second sampled image obtained by photographing a feature chart with a camera.

[0190] The controller 23 can also control the camera 22 to shoot the pure color image card and the feature image card to obtain a first sampling image obtained by shooting the pure color image card and a second sampling image obtained by shooting the feature image card. In some embodiments, the camera 22 of the projection device can be built-in or externally installed, and the camera 22 can shoot the projection screen projected by the projection device through the light-emitting component 21 to obtain a sampling image. After shooting by the camera 22, the clarity of the sampling image shot by the camera 22 can also be detected to determine whether the focal length of the projection device is appropriate. If the clarity of the detected sampling image is low, adjust the focal length of the projection device, and reshoot the sampling image through the camera 22. Compare the clarity of the sampling image in the order of shooting time to determine the focal length parameters of the projection device when the clarity of the sampling image is the highest.

[0191] In some embodiments, during the process of determining the focal length parameter of the projection device, the camera 22 may be switched to continuous shooting mode while adjusting the focal length parameter. A removable watermark representing the focal length parameter is added to the captured sample images. The clarity of all sample images is compared to determine the focal length parameter of the projection device when the sample image has the highest clarity.

[0192] For example, the first sampling image acquired by the controller 23 is shown in FIG22 . In the first sampling image, there are a rectangular obstacle 31 and a circular obstacle 32 .

[0193] S300: Determine a characteristic contour area according to the first sampling image.

[0194] The characteristic contour region is the contour region with the largest area in the first sampled image or a contour region specified by the user. Controller 23 can determine the characteristic contour region based on the positions of obstacles 31 and 32 in the first sampled image to ensure that the projection image projected by the projection device avoids obstacles 31 and 32, allowing the user to view the projection image without being obstructed by the obstacles.

[0195] In some embodiments, the characteristic contour area can be the contour area with the largest area in the first sampling image or the contour area specified by the user. The controller 23 can determine and select one of the above contour areas as the characteristic contour area based on the setting status of the obstacle avoidance switch of the projection device. Because the obstacle between the light-emitting component 21 and the projection surface 1 will form a shadow area on the projected image, and the shadow area will split the original projection image into multiple contour areas without shadow areas. At this time, the controller 23 can extract all the contour areas in the first sampling area and detect the setting status of the obstacle avoidance switch of the projection device. The setting status of the obstacle avoidance switch includes on and off. The setting status can be manually set by the user using the projection device to be on and off, or the projection device can automatically switch to the on state when it recognizes that there is an obstacle in the projection image.

[0196] When the projection device is in the obstacle avoidance state, the projection image needs to be projected away from obstacles. If the setting state is on, the controller 23 will traverse the area of ​​each contour area in the first sampling image and filter out the candidate contour area. The candidate contour area is used by the user to specify as a feature contour area. During the screening process, in addition to the contour area, other parameters for screening can also be set. For example, in order to enable the user to view the projection image more clearly, the controller 23 sets the distance between the contour area and the user as a screening parameter. For another example, the shape of the contour area is an irregular figure, and the projection needs to be projected into a rectangular area. At this time, the controller 23 also needs to set the maximum rectangular area in each contour area, and use the maximum rectangular area as the screening parameter.

[0197] Images are all composed of pixels. In some embodiments, the controller 23 can also delineate the contour area based on the color values ​​of the pixels in the first sampled image. Since the color of the solid color image card is light, and the shadow area after being blocked by an obstacle is usually black, there is a big difference in color between the two. Therefore, the controller 23 can also set a color difference threshold, and traverse the color values ​​of the pixels in the first sampled image, obtain the pixels whose color difference values ​​of adjacent pixels are greater than or equal to the color difference threshold, and identify the boundary graphics based on the pixels whose color difference values ​​of adjacent pixels are greater than or equal to the color difference threshold. The boundary graphics are the shape graphics of the obstacle between the light emitting component 21 and the projection surface 1.

[0198] After identifying the boundary pattern, controller 23 delineates the contour area based on the edges of the first sampled image. This allows the pixels within the shadow area created by the obstacle to be black, with a small color difference between adjacent pixels that does not reach the color difference threshold. Pixels within the shadow area's outline are black, while pixels outside the outline are the solid color of the image card, significantly different from black. This allows accurate identification of the shadow area created by the obstacle and precise delineation of the contour area.

[0199] In some embodiments, the controller 23 may also input the first sample image captured by the camera 22 into a recognition model. The recognition model is a neural network model trained based on sample images. The recognition model can be trained until convergence using a large number of sample images with and without obstacles. After inputting the first sample image into the recognition model, the controller 23 obtains a recognition result output by the recognition model. The recognition result is the classification probability that the first sample image contains an obstacle target. Based on the classification probability, whether the first sample image contains an obstacle target can be determined. If the recognition result indicates that an obstacle target is contained, it indicates that there is an obstacle between the light emitting component 21 and the projection surface 1. The controller 23 removes the contour area corresponding to the obstacle target from the first sample image and determines the characteristic contour area in the first sample image after the obstacle target is removed. If the recognition result indicates that an obstacle target is not contained, it indicates that there is no obstacle between the light emitting component 21 and the projection surface 1. The controller 23 then determines the characteristic contour area based on the first sample image.

[0200] In some embodiments, if the setting state of the obstacle avoidance switch is off, in order to avoid obstacles while allowing the user to clearly see the projected content in the projection screen, the controller 23 will traverse the area of ​​each contour area in the first sampling image, and filter out the contour area with the largest area as the characteristic contour area.

[0201] In some embodiments, the parameters for filtering the outline areas can also be prioritized. For example, the area of ​​each outline area can be prioritized as the first filtering parameter. If the outline areas are the same, the distance between the outline area and the user can be selected as the second filtering parameter. The above is merely an example of this embodiment. When setting priorities, the filtering parameters can be interchanged, and some embodiments of this application do not impose specific limitations on this.

[0202] In some embodiments, the controller 23 may further set a parameter threshold based on the screening parameters before screening the contour areas. For example, when the area of ​​the contour area is used as the screening parameter, an area parameter threshold may be set. If the area of ​​the contour area is greater than the parameter threshold, then the contour area meets the screening criteria and can be used as a candidate contour area. If the area of ​​the contour area is less than the parameter threshold, then the contour area is too small to be clearly visible to the user at the specified distance. Therefore, the contour area does not meet the screening criteria and cannot be used as a candidate contour area.

[0203] Before filtering candidate contour regions, the controller 23 may also generate a contour region list based on the areas of each contour region in the first sampled image. If a contour region in the contour region list does not meet the filtering criteria, the controller 23 removes the corresponding contour region from the contour region list. After all contour regions have been filtered, the contour region list contains candidate contour regions that meet the filtering criteria. This results in a candidate contour region list for the user to designate as a feature contour region.

[0204] Before screening the contour regions, the controller 23 may further set the number of candidate contour region lists, for example, to three. In this case, after traversing the areas of the contour regions in the first sample image, the controller 23 screens out three candidate contour regions based on their area sizes and arranges them in the candidate contour region list in a default or user-specified order.

[0205] S400: Extracting feature points in the second sampling image according to the feature contour area.

[0206] In some embodiments, after traversing the area of ​​each contour in the first sampling image and screening out the alternative wheel library area, the controller 23 will also control the light-emitting component 21 to project the feature map card onto the projection surface 1, and after the light-emitting component projects the feature map card, control the camera 22 to shoot the feature map card to obtain a second sampling image. Figure 21 shows a second sampling image of an embodiment of the present application. Then, the controller 23 can also identify the feature points in the same contour area in the second sampling image based on the alternative contour area screened from the first sampling image, and calculate the average depth of the feature points in the same alternative contour area in the second sampling image. A feature point refers to a point where the grayscale value of the image changes dramatically or a point with a larger curvature on the edge of the image, that is, the intersection of two edges. The annular rectangle in Figure 23 is the feature point on the feature map card.

[0207] The method for calculating the feature points can be geometric triangulation, inverse depth, particle filtering, etc. In the process of calculating the average depth of the feature points located in the same alternative contour area in the second sampled image, the controller 23 can also traverse the color values ​​of the feature points in the alternative contour area. The color of the feature point outline should have a large color difference from the solid color chart so that the feature point outline and the feature point position can be clearly reflected. In some embodiments, in order to distinguish it from the shadow formed by the obstacle on the projection surface, the color of the feature point can be any dark color other than black, and the color of the feature point can be non-uniform. Dark color is defined as a color that is greater than 1 / 12 of the standard depth of the dye color. The embodiment of the present application does not impose any other restrictions on the color of the feature point.

[0208] Figure 24 shows the outlines of two alternative regions in the second sampling image. In Figure 24, the two dotted lines in the second sampling image are the outlines of the alternative regions, where the alternative contour region to the left of the obstacle 32 is defined as the first alternative contour region, and the alternative contour region to the right of the obstacle 32 is defined as the second alternative contour region. When calculating the average depth of feature points, the controller 23 will calculate based on the feature points within the same alternative contour region. For the first alternative contour region, the controller 23 will calculate the average depth based on the 9 feature points within the first alternative contour region. For the second alternative contour region, the controller 23 will calculate the average depth based on the 3 feature points within the second alternative contour region.

[0209] In some embodiments, when calculating the average depth of feature points within the same candidate contour region in the second sampled image, the controller 23 may also traverse the color values ​​of the feature points within the candidate contour region. When the controller 23 controls the light emitting assembly 21 to project the feature card, the location of the feature point can be visualized because the color value of the feature point is significantly different from the color value in the solid color card.

[0210] Due to the complexity of the environment, there may be various situations when the light-emitting component 21 projects the projection image onto the projection surface 1. If the projection surface 1 is a wall, the same projection image may be projected onto two walls at different distances. At this time, the image displayed on the wall closer to the projection device is larger than the image displayed on the wall farther from the projection device, which will cause the image sizes to be different and the projection shape to be deformed. If the projection surface 1 is a projection screen, problems such as deformation of the projection image may occur due to the placement of the projection equipment. For example, the projection equipment is placed at different heights in the upper and lower directions, or the projector is placed to the left or right, resulting in the projection image being trapezoidal.

[0211] In this regard, the controller 23 can also extract the projected shape based on the color values ​​of multiple feature points. In this process, in order to extract the projected shape more quickly, the controller 23 can obtain boundary feature points in the candidate area outline. Boundary feature points are feature points that can reflect the shape of the candidate outline area. For example, when the candidate outline area is a rectangle, four feature points corresponding to the four corners of the rectangle can be extracted, and the projected shape can be extracted based on these four feature points. Alternatively, two feature points corresponding to two diagonal corners of the rectangle can be extracted, and the projected shape can be extracted based on these two feature points.

[0212] Because the projection area of ​​a projection device is mostly rectangular, if there is an obstacle between the light output assembly 21 and the projection surface 1, the shadow area formed by the obstacle in the projection image will divide the projection image into irregular shapes. In this regard, the controller 23 can also, after screening the candidate contour areas, delineate the largest rectangular area in the candidate contour areas, and use the largest rectangular area in the candidate contour areas as the effective projection area in the candidate contour areas.

[0213] After the controller 23 extracts the projection shape, it can also obtain the hardware parameters of the camera 22 and the light output component 21. The hardware parameters of the camera 22 include sensitivity, white balance, metering, focus, exposure compensation, and focal length. Before the camera 22 sets the sensitivity, the controller 23 can also use the camera to obtain the current ambient light and adjust the sensitivity based on the intensity of the light. For better viewing effect, the light environment in which the projection device is located is usually weak during projection. Therefore, the sensitivity of the camera 22 needs to be adjusted to a higher value.

[0214] The focus modes are single-point autofocus AF-S, servo autofocus AF-C, intelligent autofocus AF-A and manual focus.

[0215] When setting exposure compensation, the camera 22 can also detect the brightness of the photo. When the brightness of the photo is moderate, just keep it at "0"; when the photo is dark, increase the exposure compensation; when the photo is bright, reduce the exposure compensation.

[0216] When setting the focal length of the camera 22, if you want to capture a wider field of view, you can simply adjust the lens zoom ring to the minimum. If you want to capture a distant scene, you can simply extend the lens. At the same time, a wider focal length will produce a greater depth of field, while a longer focal length will produce a shallower depth of field.

[0217] The hardware parameters of the light emitting component 21 include resolution, projection brightness, contrast, focus mode and display ratio.

[0218] In some embodiments, while obtaining the hardware parameters of camera 22 and light output assembly 21, controller 23 can also obtain the standard shape on the feature map in the second sample image. When light output assembly 21 projects the feature map perpendicularly onto a regular projection surface 1, the projected shape is identical to the standard shape. At this point, controller 23 can calculate the distance from the light output assembly to each feature point in the candidate contour area, i.e., the depth of the feature point.

[0219] However, for the uneven projection surface 1, the feature map card will be projected on planes at different distances, and correspondingly, the planes on which the feature points on the feature map card are located are also different. For feature points in the same alternative contour area, since the distances between each plane and the optical machine 200 in the projection surface 1 are different, the depths of the feature points are also different. The controller 23 will calculate the distances between the multiple feature points and the light output component 21 based on the projection shape extracted from the color values ​​of the multiple feature points, the standard shape on the feature map card in the second sampling image, and the hardware parameters of the camera 22 and the light output component 21. And calculate the average value of these distances to obtain the average depth.

[0220] During the calculation process, controller 23 can calculate the average depth of feature points on the same plane in projection surface 1. Before the calculation, controller 23 can also obtain the number of planes contained in projection surface 1 by identifying the characteristic contours of projection surface 1. As shown in Figure 25, light output component 21 projects a feature map onto projection surface 1. There are two raised walls in projection surface 1, dividing projection surface 1 into five planes. Three of these planes are located at the same distance from light output component 21, and the other two planes are closer to light output component 21.

[0221] For the five planes described above, the controller 23 can determine which planes are included in each candidate contour region, and then calculate the depths of the feature points within the included planes. Based on the depths of the feature points, the controller 23 can calculate the average depth of the feature points within the plane. After calculating the average depths of all planes included in the candidate contour region, the controller 23 can calculate the average depth of the feature points in the candidate contour region based on the average depths of the planes.

[0222] In some embodiments, if one of the planes in the projection surface 1 is smaller in area and does not contain feature points, the controller 23 may divide the area of ​​the plane into adjacent planes to ensure that every area in the projection surface 1 is calculated.

[0223] After calculating the average depth of the feature points, for each alternative area outline, the controller 23 can also calculate the area of ​​each alternative area outline and the area of ​​the second sampling image, and calculate the area ratio of the alternative contour area relative to the second sampling image based on the area of ​​the alternative area outline and the area of ​​the second sampling image. The area ratio can be sorted from large to small according to the numerical value. After calculating the average depth of the feature points in the alternative contour area and the area ratio of the alternative contour area relative to the second sampling image, the controller 23 can also generate a first prompt message based on the above two data, and control the light-emitting component 21 to project the first prompt message when the setting state of the obstacle avoidance switch is turned on. The first prompt message includes each alternative contour area, and the average depth and area ratio of the feature points corresponding to each alternative contour area.

[0224] The user can view the selectable candidate contour areas through the first prompt information projected onto the projection surface 1. The user can then select one of the selectable candidate contour areas as the characteristic contour area. Based on the first prompt information, the user can generate a selection command using a button on the control device of the projection device. Upon receiving the selection command, the controller 23 responds to the selection command by marking the candidate contour area specified in the selection command as the characteristic contour area.

[0225] In some embodiments, the first prompt information can be displayed in the form of a list, which includes selectable alternative contour areas and the average depth and area ratio of the feature points corresponding to the alternative contour areas. When the user selects an alternative contour area through the control device, the controller 23 can also control the light-emitting component 21 to mark the contour portion of the selected alternative contour area so that the user can more intuitively see the area of ​​the alternative contour area. The controller 23 can choose other colors that are different from the color of the feature card or the feature point as the marking color. The embodiment of the present application does not impose any specific restrictions on the marking color.

[0226] S500: Calculating the angle between the projection surface and the light emitting component based on the feature points, and controlling the light emitting component to project the projection content onto the projection surface according to the angle.

[0227] After determining the characteristic contour area, the controller 23 extracts characteristic points in the second sampling image based on the characteristic contour area, and calculates the angle between the projection surface 1 and the light-emitting component 21 based on the characteristic points, and controls the light-emitting component 21 to project the corrected projection content onto the projection surface 1 based on the angle.

[0228] In some embodiments, the controller 23 may also control the camera 22 to call out the camera coordinate system and obtain the feature point coordinates of the feature points within the feature contour area in the camera coordinate system during the process of calculating the angle between the projection surface 1 and the light output component 21 based on the feature points. The feature point coordinates are generally the center of the figure where the feature point is located. For example, when the feature point is a square or rectangle, the feature point coordinates are the coordinates of the center of the square or rectangle. When the feature point is a circle, the feature point coordinates are the coordinates of the center of the circle.

[0229] To facilitate camera 22 capturing sample images, the camera 22 is typically positioned directly in front of the projection device along with the light output assembly 21. After acquiring the feature point coordinates in the camera coordinate system, the controller 23 controls the light output assembly 21 to retrieve the light output assembly coordinate system. Based on the hardware parameters of the camera 22 and light output assembly 21, the feature point coordinates are converted to the light output point coordinates in the light output assembly coordinate system. These hardware parameters can be the vector displacement between the lens center of the camera 22 and the center of the light output assembly 21.

[0230] In some embodiments, the controller 23 can first obtain the coordinates of the center of the camera lens of the camera 22. As shown in Figure 26, the X1Y1 coordinate system is the light-emitting component coordinate system, and the X2Y2 is the camera coordinate system. When the coordinates of the center of the camera lens of the camera 22 in the camera coordinate system are (0,0), the above coordinates are then converted to the light-emitting component coordinate system, and the coordinates of the center of the camera lens are repositioned. For example, the coordinates after repositioning are (30,-40). Then the controller 23 can calculate the vector displacement value of 50 between the center of the lens of the camera 22 and the center of the light-emitting component 21 based on the coordinates before and after repositioning. After calculating the vector displacement value, the controller 23 can convert all the feature point coordinates in the camera coordinate system into the light-emitting point coordinates in the light-emitting component coordinate system according to the vector displacement value.

[0231] After converting the feature point coordinates into the light-emitting point coordinates in the light-emitting component coordinate system, the controller 23 can fit a new projection surface 1 in the light-emitting component coordinate system based on the multiple light-emitting point coordinates. After fitting the new projection surface 1, the controller 23 calculates the angle between the projection surface 1 and the light-emitting surface corresponding to the light-emitting component 21, and controls the light-emitting component 21 to project the corrected projection content onto the projection surface based on the angle, so that the user can see the corrected projection image.

[0232] In some embodiments, in the process of controlling the light emitting component 21 to project the projection content onto the projection surface 1 according to the angle, the controller 23 can also obtain the operating parameters of the light emitting component 21, and calculate the projectable area based on the operating parameters and the angle between the projection surface 1 and the light emitting component 21. Among them, the operating parameters may include the projection distance, the focal length or resolution of the light emitting component 21, etc. For example, the controller 23 can calculate the area of ​​the projection screen when projecting vertically (90°) based on the projection distance. Then, based on the angle between the projection surface 1 and the light emitting component 21, the trigonometric function value corresponding to the angle is determined. Finally, the projectable area of ​​the projection screen is calculated based on the area and trigonometric function value of the projection screen.

[0233] In order to be suitable for playback devices such as mobile terminals or smart TVs, the projection screen projected by the projection device is usually rectangular. When the projectable area is an irregular shape or a non-rectangular shape, the controller 23 can also define a target projection area in the projectable area. The target projection area is the largest inscribed rectangular area in the projectable area, and the rectangular area has a preset aspect ratio. The target projection area can be adapted to projection screens played horizontally or vertically. As shown in Figure 27, before defining the target projection area, the controller 23 can also identify the video or picture playback source to be projected and extract the screen playback ratio of the above-mentioned projection content. Common playback ratios include 4:3, 16:9, 2.39:1 or 1.85:1, etc. Based on the comparison between the extracted screen playback ratio and the aspect ratio of the rectangular area, it is determined whether the target projection area is the largest inscribed rectangular area in the horizontal direction or the largest inscribed rectangular area in the vertical direction.

[0234] FIG28 shows a flow chart of dividing a target projection area according to an obstacle avoidance instruction, which specifically includes the following steps:

[0235] S2801: Calculate the projectable area;

[0236] S2802: Determine whether an obstacle avoidance instruction is recognized; if an obstacle avoidance instruction is recognized, execute S2803 to S2804; if no obstacle avoidance instruction is recognized, execute S2805 to S2806;

[0237] S2803: Set the obstacle avoidance state to on;

[0238] S2804: extracting feature points from the second sample image to define a target projection area;

[0239] S2805: Set the obstacle avoidance state to off;

[0240] S2806: Delineate a target projection area according to the vertex coordinates of the light emitting surface of the light emitting component.

[0241] When defining the target projection area, the controller 23 can also detect user-inputted obstacle avoidance commands to activate the obstacle avoidance function. If an obstacle avoidance command is detected, the controller 23 switches the projection device's obstacle avoidance mode to on and defines the target projection area based on the feature points extracted from the second sampled image. If no obstacle avoidance command is detected, the projection device's obstacle avoidance mode remains off, and the controller 23 defines the target projection area based on the vertex coordinates of the light-emitting surface of the light-emitting assembly 21.

[0242] After defining the target projection area, the controller 23 may also inversely transform the coordinates of the target projection area to the light emitting surface of the light emitting component, and control the light emitting component to project the projection content according to the inversely transformed coordinates of the target projection area.

[0243] Based on the same inventive concept, an embodiment of the present application further provides a projection image processing method, which is applied to a projection device. The projection device includes a light output component, a camera, and a controller. As shown in FIG29 , the projection image processing method includes:

[0244] S100: In response to the projection image correction instruction, controlling the light emitting component to project the correction image;

[0245] The rectified images include pure color charts and feature charts.

[0246] S200: Acquire a first sampled image obtained by photographing a pure color chart with a camera, and a second sampled image obtained by photographing a feature chart with a camera.

[0247] S300: determining a characteristic contour area according to the first sampling image;

[0248] The characteristic contour region is the contour region with the largest area in the first sampling image or the contour region specified by the user.

[0249] S400: Extracting feature points in the second sampling image according to the feature contour area.

[0250] S500: Calculating the angle between the projection surface and the light emitting component based on the feature points, and controlling the light emitting component to project the projection content onto the projection surface according to the angle.

[0251] It can be seen from the above scheme that after receiving the projection picture correction instruction, the projection device controls the light-emitting component to project the corrected images of the primary color card and the feature card. And obtain the first sampling image of the pure color card taken by the camera and the second sampling image of the feature card taken by the camera. The characteristic contour area is determined according to the first sampling image, and the characteristic points are extracted in the second sampling image according to the characteristic contour area. The angle between the projection surface and the light-emitting component is calculated based on the characteristic points extracted from the second sampling image. The light-emitting component projects the projection content to the projection surface according to the angle to ensure that when the projection surface is blocked by obstacles, the projection device can extract feature points according to the characteristic contour area after obstacle avoidance, and project the corrected projection image on the projection surface, so as to improve the user experience when using the projection device.

[0252] In some embodiments, when the user turns on the projection device 2, the projection device 2 can project the content preset by the user onto the projection surface, which can be a wall or a screen. The projection image can be displayed on the projection surface for the user to watch.

[0253] In some embodiments, a projection device includes a light-emitting assembly for projecting content onto a projection surface. The light-emitting assembly includes a laser light source, an optical engine, and a lens. The optical engine houses a digital micromirror device (DMD), the core imaging component of the projection device, responsible for image formation. Specifically, the laser light source provides illumination for the optical engine. The DMD in the optical engine adjusts the light source beam according to the projection coordinates of the DMD plane and outputs it to the lens for imaging, which is then projected onto the projection medium to form a projection image.

[0254] When the projected image is projected onto the projection surface, if there is an obstacle on the projection surface or the automatic obstacle avoidance function fails to avoid the obstacle after it is turned on (obstacle recognition error or incomplete obstacle avoidance), the projector can avoid the obstacle by using the screen movement function. In addition, since the obstacle avoidance area is not controlled by the user, when there is an obstacle in the center of the projection, the obstacle avoidance strategy will project the obstacle to the left, while the user expects it to be on the right. In this case, the screen movement function of the projector can also be used to solve the problem. The screen can be adjusted to the position and size that meets the user's needs through screen zooming and moving.

[0255] In some embodiments, image scaling and panning are achieved by adjusting the projection coordinates of the DMD plane. However, when the projection device is tilted, in order to maintain a rectangular shape on the projection surface, the projection device must automatically adjust the projection coordinates of the DMD plane to a trapezoidal shape. Directly moving the projection coordinates of the optical engine DMD plane in this situation will cause the projection shape on the projection surface to change, reducing the user experience.

[0256] Figure 30 is a schematic diagram of the projection coordinate movement of the DMD plane. The trapezoid at Position 1 represents the DMD plane projection coordinate shape when the projection onto the projection surface is a rectangle after automatic keystone correction when the projection device is projected sideways. When the image movement command is executed, the trapezoid at Position 1 on the DMD plane moves to Position 2. Since the projection onto the projection surface is a rectangle in Position 1, when it moves to Position 2, the right side of the trapezoid at Position 1 overlaps with the left side of the trapezoid at Position 2, and the right side of the trapezoid at Position 1 is longer than the left side of the trapezoid at Position 2. Clearly, when it moves to Position 2, the left side of the trapezoid at Position 2 must be shorter than the right side of the trapezoid at Position 1 when projected onto the projection surface. Therefore, when the projection device is projected sideways, direct translation of the trapezoid's projection coordinates on the DMD plane will cause the projection onto the projection surface to distort.

[0257] The principle of image movement is to control the image display at different projection coordinates on the DMD plane, thereby achieving the effect of projected image movement. Figure 31 shows a schematic diagram of the projection coordinate movement of the DMD plane during front projection. DMDs are available in 2K, 4K, and 8K sizes. Taking 2K as an example, when the projector is projecting normally without correction, obstacle avoidance, scaling, or other cropping, the maximum projection coordinates of the DMD plane are (0, 0), (1920, 0), (1920, 1080), and (0, 1080). Projecting according to these projection coordinates results in the maximum projection. Since there is no image cropping, the projected image cannot be shifted. In other words, the projection image must be shifted so that the projection coordinates of the DMD plane are less than the maximum projection coordinates. When the light output component projects the content according to the maximum projection coordinates of the DMD plane, the projected area is maximized, i.e., the maximum projection area. The projected image movement must be within this maximum projection area. For example, when the projection device is projecting forward, there is no angle between the light-emitting component and the projection surface. At this time, the projection coordinates of the DMD plane are rectangular, and the shape of the projection projected onto the projection surface is also rectangular. Therefore, the projected image is rectangular before and after the movement. However, when the projection device is projecting sideways, as shown in Figure 32, a schematic diagram of the movement of the projection coordinates of the DMD plane when the projection device is projecting sideways. To achieve a rectangular shape for the projection projected onto the projection surface, automatic trapezoidal correction is required to adjust the projection coordinates of the DMD plane to a trapezoidal shape. Consequently, to ensure that the projected image is rectangular before and after the movement, the trapezoidal shape of the projection coordinates after the movement is inconsistent with the trapezoidal shape of the projection coordinates before the movement.

[0258] In order to solve the problem of deformation of the projected image when the projection device moves the image under side projection, an embodiment of the present application also provides a projection device, as shown in Figure 33, including a light output component 21 and a controller 23. The controller can adjust based on the projection coordinates of the DMD plane and the image movement instruction to obtain the projection coordinates after the DMD plane moves, and project the projection content onto the projection surface according to the moved projection coordinates to solve the problem of deformation of the projected image when moving. As shown in Figure 33, the controller 23 can be used to execute the program steps corresponding to the projection image movement, including the following:

[0259] S100: Acquire a screen movement instruction input by a user.

[0260] Among them, the screen movement instruction includes the moving direction and the moving distance. There are multiple ways for the user to input the screen movement instruction, and the ways of inputting the screen movement instruction may include manual input and voice input. Among them, manual input can be input through the physical buttons on the projection device, or the physical buttons on the remote control of the projection device. For example, when the user inputs the moving distance, the moving direction and moving distance can be input through the up, down, left, and right keys on the remote control of the projection device, thereby controlling the movement of the projected screen. Voice input can generate a screen movement instruction by recognizing the user's voice. For example, after turning on the power of the projection device, the user can press the screen movement button on the projection device or the remote control of the projection device, and then input voice, for example, move left three steps. The projection device recognizes the moving direction and moving distance in the voice, and obtains the screen movement instruction.

[0261] In some embodiments, because the user inputs multiple movement directions or the projection device recognizes multiple movement directions when recognizing the user's voice, the acquired screen movement instructions contain multiple directions, such as left and right movements at the same time, or up and down movements at the same time, and thus the direction of movement of the projected screen cannot be clearly determined.

[0262] Therefore, after obtaining the screen movement instruction input by the user, the controller 23 can respond to the screen movement instruction and detect the movement direction in the screen movement instruction; if the screen movement instruction also contains opposite movement directions, it generates a second prompt information and controls the light-emitting component 21 to project the second prompt information.

[0263] For example, after the user inputs the screen movement instruction, the controller 23 detects the screen movement instruction. When it detects that the opposite movement direction exists at the same time, it generates a second prompt information for prompting the user that the movement direction input is abnormal, and controls the light-emitting component 21 to project the second prompt information onto the projection surface 1. The second prompt information can be a prompt such as "The movement direction is abnormal, please re-enter" to prompt the user to re-enter the screen movement instruction, so that the user re-enters the screen movement instruction, and the controller 23 re-detects the screen movement instruction. If no abnormal movement direction is detected, the subsequent projection screen movement operation can be performed according to the screen movement instruction.

[0264] S200: In response to the image movement instruction, the vertex coordinates of the current light-emitting surface and the rotation matrix between the projection surface and the light-emitting surface are acquired.

[0265] The light emitting surface is the plane of the core imaging device DMD used for image formation in the light emitting assembly 21. The vertex coordinates of the current light emitting surface are the vertex coordinates of the light emitting surface after the projection device automatically avoids obstacles or corrects the image scaling.

[0266] In some embodiments, the projection device can automatically detect obstacles in the projection area and project the image only after determining that there are no obstacles in the projection area based on the obstacle detection results, thereby implementing automatic obstacle avoidance. In other words, if an obstacle exists in the projection area, the projection area before the projection device performs the automatic obstacle avoidance process will be different from the projection area after the obstacle avoidance process is completed.

[0267] In some embodiments, when the projection angle of the projection device and the distance to the projection surface change, the projected image may be deformed. To ensure that the projected image is rectangular, the projection device may perform automatic keystone correction.

[0268] Therefore, after the projector automatically avoids obstacles or automatically corrects the shape, the vertex coordinates of the DMD plane will change. The projector can be pre-configured with a storage module that can record and store the automatic obstacle avoidance and automatic keystone correction of the projector in real time.

[0269] After receiving the image movement instruction, the projection device can read the angle between the projection surface 1 and the light emitting component 21 from the storage module and calculate the rotation matrix between the projection surface 1 and the light emitting surface. In some embodiments, as shown in FIG34 , the projection device further includes a camera 22 configured to capture the projected content image; the controller 23 is further configured to execute the following:

[0270] S3401: Acquire the projection image captured by the camera;

[0271] S3402: Identify the coordinates of feature points in the projected image in the camera coordinate system;

[0272] S3403: Convert the feature point coordinates to the light output component coordinate system according to the hardware parameters of the light output component and the camera;

[0273] S3404: Fitting the projection plane according to the coordinates of the feature points in the light emitting component coordinate system;

[0274] S3405: ​​Calculate the angle between the projection plane and the light-emitting surface;

[0275] S3406: Construct a rotation matrix based on the angle between the projection plane and the light-emitting surface.

[0276] In the step of determining the rotation matrix, the controller 23 can control the light-emitting component 21 to project the calibration chart onto the projection surface, and then control the camera 22 to capture the projected image of the calibration chart. The camera 22 can be a binocular camera or an RGBD camera. Based on the binocular camera or the RGBD camera, the coordinates of the feature points of the calibration chart in the projected image in the camera coordinate system can be obtained. Then, based on the external parameters between the light-emitting component 21 and the camera 22, the controller 23 can convert the feature point coordinates from the camera coordinate system to the light-emitting component coordinate system, and then fit all the feature points in the light-emitting component coordinate system, fit the projection plane, and then fit the angle between the projection plane and the light-emitting surface. According to the angle between the projection plane and the light-emitting surface, the rotation matrix between the projection plane and the light-emitting surface can be calculated.

[0277] S300: Calculating the current projection area according to the rotation matrix and the vertex coordinates of the current light-emitting surface.

[0278] The controller 23 can convert the vertex coordinates of the current light-emitting surface in the light-emitting surface coordinate system into coordinates in the projection surface coordinate system based on the acquired conversion matrix, and thus obtain the actual position coordinate values ​​of the current actual projection area.

[0279] For example, based on the rotation matrix between the projection surface and the light-emitting surface, when the vertex coordinates of the current light-emitting surface are converted to the projection surface coordinate system, as shown in Figure 35, the corresponding coordinates are A(x1,y1), B(x2,y2), C(x3,y3), and D(x4,y4), which are the vertex coordinates of the current projection area. Specifically, the vertex coordinates of the current light-emitting surface are substituted into the following formula to calculate the current projection area:

[0280] Among them, M is the hardware parameter of the light emitting component, that is, the internal parameter, and R is the rotation matrix.

[0281] S400: Calculating vertex moving distances according to vertex coordinates, moving directions, and moving distances in the current projection area.

[0282] After obtaining the current projection area, the projection area after moving according to the screen movement instruction input by the user can be calculated.

[0283] In some embodiments, the vertex coordinates of the current projection area can be moved according to the moving direction and moving distance to obtain the vertex coordinates of the moved projection area to determine the moved projection area. Since the size of the projection surface changes with the change of the projection distance, the vertex moving distance of the vertex coordinates cannot be represented by a fixed length. Therefore, as shown in Figure 36, the controller 23 is further configured to execute the following:

[0284] S3601: Obtain the movement mode to which the movement direction belongs; if the movement mode is left-right movement, execute S3602-S3603; if the movement mode is up-down movement, execute S3604-S3605;

[0285] S3602: Calculate the projection width ratio according to the vertex coordinates of the current projection area;

[0286] S3603: Calculate the vertex moving distance according to the projection width ratio and the moving distance;

[0287] S3604: Calculate the projection height ratio according to the vertex coordinates of the current projection area;

[0288] S3605: Calculate the vertex moving distance according to the projection height ratio and the moving distance.

[0289] In some embodiments, the projection width ratio is calculated based on the actual width of the current projection area, and the projection height ratio is calculated based on the actual height of the current projection area. Therefore, the controller 23 is further configured to perform the following:

[0290] If the movement mode is left and right movement, calculate the first horizontal coordinate difference and the second horizontal coordinate difference of the current projection area. The first horizontal coordinate difference is the difference between the horizontal coordinates of the two vertices of the upper side of the current projection area, and the second horizontal coordinate difference is the difference between the horizontal coordinates of the two vertices of the lower side of the current projection area.

[0291] For example, as shown in FIG35 , the first horizontal coordinate difference width1 is the difference between the horizontal coordinates of the two vertices A and B on the upper side of the current projection area; the second horizontal coordinate difference width2 is the difference between the horizontal coordinates of the two vertices C and D on the lower side of the current projection area.

[0292] The projection width ratio is calculated according to the first horizontal coordinate difference and the second horizontal coordinate difference. The projection width ratio is the average of the first horizontal coordinate difference and the second horizontal coordinate difference, that is, (width1+width2) / 2.

[0293] Similarly, if the movement mode is up and down, calculate the first ordinate difference and the second ordinate difference of the current projection area. The first ordinate difference is the difference between the ordinates of the two vertices on the left side of the current projection area, and the second ordinate difference is the difference between the ordinates of the two vertices on the right side of the current projection area.

[0294] For example, as shown in FIG35 , the first horizontal coordinate difference height1 is the difference between the vertical coordinates of the two vertices A and D on the upper side of the current projection area; the second horizontal coordinate difference height2 is the difference between the vertical coordinates of the two vertices B and C on the lower side of the current projection area.

[0295] The projection height ratio is calculated according to the first ordinate difference and the second ordinate difference. The projection height ratio is the average of the first ordinate difference and the second ordinate difference, that is, (height1+height2) / 2.

[0296] In some embodiments, the movement mode to which the movement direction belongs is obtained. If the movement mode is left-right movement, the vertex movement distance is calculated according to the following formula:

[0297] Wherein, width1 is the first horizontal coordinate difference, width2 is the second horizontal coordinate difference, and step is the moving distance (number of steps) input by the user.

[0298] In some embodiments, the movement mode to which the movement direction belongs is obtained. If the movement mode is up and down, the vertex movement distance is calculated according to the following formula:

[0299] Among them, height1 is the first vertical coordinate difference, height2 is the second vertical coordinate difference, and step is the moving distance (number of steps) entered by the user.

[0300] S500: Calculating target projection coordinates according to vertex movement distances and vertex coordinates of the current projection area.

[0301] For different movement modes, the coordinates of the vertices of the current projection area are also different. When the current projection area moves left and right, the vertical coordinates of the vertices of the current projection area will not change. When the current projection area moves up and down, the horizontal coordinates of the vertices of the current projection area will not change. Therefore, in order to improve the response speed, as shown in Figure 37, the controller 23 is further configured to execute the following:

[0302] S3701: Get the movement mode of the movement direction; if the movement mode is left-right movement, execute S3702; if the movement mode is up-down movement, execute S3703;

[0303] S3702: Move the horizontal coordinate of the vertex in the current projection area by the vertex moving distance in the moving direction;

[0304] S3703: moving the vertical coordinate of the vertex in the current projection area by the vertex moving distance in the moving direction;

[0305] S3704: Obtain target projection coordinates.

[0306] For example, if the movement mode is left or right, you only need to move the horizontal coordinate of the vertex in the current projection area. If the movement direction is left, subtract the vertex movement distance from the horizontal coordinate of the vertex in the current projection area, that is, the target projection coordinates are: A'(x1-Disstep1,y1), B'(x2-Diststep1,y2), C'(x3-Diststep1,y3), D'(x4-Diststep1,y4).

[0307] If the movement direction is right, add the horizontal coordinate of the vertex in the current projection area to the vertex movement distance, that is, the target projection coordinates are: A'(x1+Disstep1,y1), B'(x2+Disstep1,y2), C'(x3+Disstep1,y3), D'(x4+Disstep1,y4).

[0308] If the movement mode is up and down, just move the vertical coordinate of the vertex in the current projection area. If the movement direction is upward, subtract the vertex movement distance from the vertical coordinate of the vertex in the current projection area. The target projection coordinates are: A'(x1,y1-Disstep2), B'(x2,y2-Disstep2), C'(x3,y3-Disstep2), D'(x4,y4-Disstep2).

[0309] If the moving direction is downward, add the vertical coordinate of the vertex in the current projection area to the vertex moving distance, and the target projection coordinates are: A'(x1,y1+Disstep2), B'(x2,y2+Disstep2), C'(x3,y3+Disstep2), D'(x4,y4+Disstep2).

[0310] In some embodiments, since the vertex coordinates of the current projection area are the ideal actual projection area calculated based on the vertex coordinates of the current light-emitting surface, there are actual errors. If the calculated projection area is corrected, the error will accumulate as the number of times the projection image moves increases, and the final effect is that the projection image is severely deformed. To avoid this problem, the projection shape needs to be corrected each time the projection area is calculated to prevent the error from being transferred to the next movement after the projection image moves. The correction method is to select the largest inscribed 16:9 rectangle within the calculated projection areas A, B, C, and D, and then move the projection image.

[0311] In some embodiments, the projection image needs to move within the maximum projection area that the light output component 21 can project. Therefore, according to the image movement instruction, after determining the projection area after the movement, the controller 23 needs to determine whether the projection area after the movement is within the maximum projection area. The controller 23 is further configured to perform the following:

[0312] The maximum projection area is calculated based on the rotation matrix and the vertex coordinates of the maximum light-emitting surface.

[0313] For example, the vertex coordinates of the maximum light-emitting surface of the light-emitting component 21 are amax(0,0), bmax(1920,0), cmax(1920,1080), and dmax(0,1080). Based on the vertex coordinates of the maximum light-emitting surface, the vertex coordinates of the maximum projection area are calculated as Amax(x1max,y1max), Bmax(x2max,y2max), Cmax(x3max,y3max), and Dmax(x4max,y4max). For the specific calculation formula, please refer to the calculation steps of the current projection area and will not be repeated here.

[0314] If the maximum projection area includes the target projection coordinates, a step of converting the target projection coordinates to the light emitting surface is performed.

[0315] If the maximum projection area does not include the target projection coordinates, first prompt information is generated, and the light emitting component 21 is controlled to project the first prompt information.

[0316] To determine whether the projection area after moving is within the maximum projection area, it is necessary to determine the vertex coordinates of the projection area after moving, that is, whether the target projection coordinates are all within the maximum projection area. For example, if the target projection coordinates are all within the maximum projection area, then the projection area after moving is within the maximum projection area, and the controller 23 can continue to execute the step of converting the target projection coordinates to the light-emitting surface. If there are target projection coordinates that are not within the maximum projection area, then the projection area after moving is not completely within the maximum projection area, that is, it is moved outside the maximum projection area. The controller 23 can generate a first prompt message for prompting the user that the projection area after moving has reached the boundary, and control the light-emitting component to project the first prompt message onto the projection surface, prompting the user that the projection image after moving according to the image movement instruction exceeds the maximum projection area that can be projected.

[0317] In some embodiments, determining whether the target projection coordinates are within the maximum projection area is based on the principle of two-dimensional vector cross product: Specifically, vector a and vector b are cross-producted. If the result is less than 0, vector b is clockwise relative to vector a; if the result is greater than 0, vector b is counterclockwise relative to vector a; and if the result is equal to 0, vector a and vector b are collinear. Clockwise and counterclockwise mean that the two vectors are connected by translating to the starting point, and that the rotation from one direction to the other is less than 180 degrees.

[0318] It can be understood that if the vertex of the moved projection area is within the maximum projection area, the vertex will always maintain the same direction relative to the walking path when walking along the maximum projection area. If the vertex of the moved projection area is outside the maximum projection area, the vertex will have a different direction relative to the walking path when walking along the maximum projection area.

[0319] Therefore, based on the above principles, as shown in FIG38 , the controller 23 is further configured to execute the following:

[0320] S3801: Determine a side length vector and a connection vector based on the vertex coordinates and the target projection coordinates of the maximum projection area. The side length vector is a vector between the coordinates of two adjacent vertices in the maximum projection area, and the connection vector is a vector between the starting coordinates of each side length vector and the target projection coordinates.

[0321] S3802: Perform vector cross product of the side length vector and the connection vector to obtain a cross product vector;

[0322] S3803: Determine whether the values ​​of the cross-product vectors have the same sign; if the values ​​of the cross-product vectors have the same sign or are zero, execute S3804; if the values ​​of the cross-product vectors have different signs, execute S3805;

[0323] S3804: Determine that the maximum projection area includes the target projection coordinates;

[0324] S3805: Determine that the maximum projection area does not include the target projection coordinates.

[0325] For example, as shown in Figure 39, the four vertices A', B', C', and D' of the moved projection area are sequentially determined to determine whether they are within the maximum projection area. Taking the vertex coordinate A' of the moved projection area as an example, the four side length vectors of the four sides of the maximum projection area are selected in clockwise order, namely: AmaxBmax = (x2max-x1max, y2max-y1max); BmaxCmax = (x3max-x2max, y3max-y2max); CmaxDmax = (x4max-x3max, y4max-y3max); and DmaxAmax = (x1max-x4max, y1max-y4max).

[0326] Four connection vectors corresponding to the connection lines between the four vertices of the maximum projection area and the vertex coordinates A' of the projection area after the movement are calculated according to the moving direction.

[0327] If the movement direction is left, the four connection vectors are: AmaxA' = (x1-Disstep1-x1max, y1-y1max); BmaxA' = (x1-Disstep1-x2max, y1-y2max); CmaxA' = (x1-Disstep1-x3max, y1-y3max); DmaxA' = (x1-Disstep1-x4max, y1-y4max). If the movement direction is right, the four connection vectors are: AmaxA' = (x1+Disstep1-x1max, y1-y1max); BmaxA' = (x1+Disstep1-x2max, y1-y2max); CmaxA' = (x1+Disstep1-x3max, y1-y3max); DmaxA' = (x1+Disstep1-x4max, y1-y4max). If the movement direction is upward, the four connection vectors are: AmaxA' = (x1-x1max, y1-Disstep2-y1max); BmaxA' = (x1-x2max, y1-Disstep2-y2max); CmaxA' = (x1-x3max, y1-Disstep2-y3max); and DmaxA' = (x1-x4max, y1-Disstep2-y4max). If the movement direction is downward, the four connection vectors are: AmaxA' = (x1-x1max, y1+Disstep2-y1max); BmaxA' = (x1-x2max, y1+Disstep2-y2max); CmaxA' = (x1-x3max, y1+Disstep2-y3max); and DmaxA' = (x1-x4max, y1+Disstep2-y4max).

[0328] Perform vector cross product on the four side length vectors and the four connection vectors respectively to obtain four cross product vectors. If the cross product vector is less than 0, the connection vector is located in the clockwise direction of the side length vector. If the cross product vector is greater than 0, the connection vector is located in the counterclockwise direction of the side length vector. If the cross product vector is equal to 0, the connection vector and the side length vector are collinear.

[0329] Determine the values ​​of the four cross product vectors in sequence. If the values ​​of the cross product vectors are different in sign and one is greater than 0, that is, the connecting vector is located counterclockwise of the side length vector, then it is determined that the maximum projected area does not include the vertex coordinate A' of the moved projected area. If the values ​​of the four cross product vectors are the same in sign or 0, and all are less than or equal to 0, that is, the connecting vector is located clockwise of the side length vector or the connecting vector and the side length vector are collinear, then it is determined that the maximum projected area includes the vertex coordinate A' of the moved projected area.

[0330] For example, as shown in Figure 29, AmaxBmax×AmaxA' < 0; BmaxCmax×BmaxA' < 0; CmaxDmax×CmaxA' < 0; DmaxAmax×DmaxA' < 0. That is, the four connection vectors are all in the clockwise direction of the four side length vectors, and the vertex A' of the moved projection area is within the maximum projection area.

[0331] Similarly, according to the above steps, it can be determined whether the other three vertices B', C', and D' in the moved projection area are within the maximum projection area. If at least one of the vertices A', B', C', and D' in the moved projection area is outside the maximum projection area, a first prompt message is generated to prompt the user to move to the edge.

[0332] In some embodiments, the controller 23 can select four side length vectors of the four sides of the maximum projection area in a counterclockwise direction, namely Amax Dmax, DmaxCmax, CmaxBmax, and BmaxAmax. Perform vector cross multiplication on the four side length vectors and the four connection vectors respectively to obtain four cross product vectors. Correspondingly, the values ​​of the four cross product vectors are judged in turn. If the values ​​of the cross product vectors are different in sign and there is a value less than 0, that is, there is a connection vector located in the clockwise direction of the side length vector, then it is determined that the maximum projection area does not include the vertex coordinate A' of the projection area after the movement. If the values ​​of the four cross product vectors are the same in sign or 0, and are all greater than 0 or equal to 0, that is, the connection vector is located in the counterclockwise direction of the side length vector or the connection vector is collinear with the side length vector, then it is determined that the maximum projection area includes the vertex coordinate A' of the projection area after the movement.

[0333] In some embodiments, the vertex coordinates of the projection area after movement, that is, whether the target projection coordinates are all within the maximum projection area, are determined in sequence. If a vertex is determined to be outside the maximum projection area, a first prompt message is directly generated. For example, when the controller 23 determines in sequence whether the vertices A', B', C', and D' of the projection area after movement are within the maximum projection area, it determines whether the vertex A' is within the maximum projection area according to the above steps. If the vertex A' is within the maximum projection area, it continues to determine whether the vertex B' is within the maximum projection area. If the vertex B' is outside the maximum projection area, a first prompt message is directly generated.

[0334] S600: Based on the rotation matrix, the target projection coordinates are converted to the light-emitting surface to obtain the light-emitting projection coordinates, and the light-emitting component is controlled to project the projection content onto the projection surface according to the light-emitting projection coordinates.

[0335] If the vertex coordinates of the projection area after moving, that is, the target projection coordinates are all within the maximum projection area, the controller 23 can convert the vertex coordinates of the projection area after moving into the light-emitting projection coordinates in the light-emitting component coordinate system based on the acquired transformation matrix, and can obtain the actual position coordinate value required by the light-emitting component to project the projection picture after moving according to the picture movement instruction. It can also be understood that the light-emitting component can complete the invisible moving projection picture after projecting the playback content according to the light-emitting projection coordinates.

[0336] For example, as shown in Figure 39, the vertex coordinates of the moved projection area are A', B', C', and D', all of which are within the maximum projection area. Based on the rotation matrix between the projection surface and the light-emitting surface, the vertex coordinates of the moved projection area are converted to the light-emitting component coordinate system to obtain the light-emitting projection coordinates. The light-emitting component projects the projection content onto the projection surface according to the light-emitting projection coordinates, as shown in Figure 40, thus achieving the movement of the projection image.

[0337] Based on the same inventive concept, the embodiment of the present application also provides a projection image processing method, which is applied to a projection device, and the projection device includes: a light output component and a controller. As shown in Figure 41, the projection image processing method includes:

[0338] S100: Acquire a screen movement instruction input by a user;

[0339] The picture movement instruction includes a movement direction and a movement distance.

[0340] S200: In response to the image movement instruction, the vertex coordinates of the current light-emitting surface and the rotation matrix between the projection surface and the light-emitting surface are acquired.

[0341] S300: Calculating the current projection area according to the rotation matrix and the vertex coordinates of the current light-emitting surface.

[0342] S400: Calculating vertex moving distances according to vertex coordinates, moving directions, and moving distances in the current projection area.

[0343] S500: Calculating target projection coordinates according to vertex movement distances and vertex coordinates of the current projection area.

[0344] S600: Based on the rotation matrix, the target projection coordinates are converted to the light-emitting surface to obtain the light-emitting projection coordinates, and the light-emitting component is controlled to project the projection content onto the projection surface according to the light-emitting projection coordinates.

[0345] After receiving a screen movement command, the projection device can obtain the vertex coordinates of the current light-emitting surface and the rotation matrix between the projection surface and the light-emitting surface. The current projection area is calculated based on the rotation matrix and the vertex coordinates of the current light-emitting surface. The vertex movement distance is calculated based on the vertex coordinates, movement direction, and movement distance of the current projection area. The target projection coordinates are calculated based on the vertex movement distance and the vertex coordinates of the current projection area. Based on the rotation matrix, the target projection coordinates are converted to the light-emitting surface to obtain the light-emitting projection coordinates. The light-emitting component is controlled to project the projection content onto the projection surface according to the light-emitting projection coordinates. This solves the problem of deformation of the projection image when the projection device moves the image in side projection mode, thereby improving the user experience.

Claims

1. A projection device, comprising: A light emitting component configured to project projection content onto a projection surface; a camera configured to capture a sample image; The controller is configured as: In response to the obstacle avoidance instruction, obtaining a transformation matrix and an adjustment instruction input by a user, wherein the transformation matrix is ​​a homography matrix of coordinates between the camera and the light output component; the adjustment instruction includes a first obstacle target specified by the user; Identifying the first obstacle target in a sampled image, where the sampled image is an image captured by the camera when the light emitting component projects the corrected image; Delimiting a projectable area in the sampled image according to the first obstacle target, wherein the projectable area is a rectangular area with a maximum preset aspect ratio that is accommodated by an area in the sampled image excluding the first obstacle target; A target projection area is determined according to the projectable area and the transformation matrix, and the light output component is controlled to project the projection content onto the target projection area.

2. The projection device according to claim 1, wherein: The controller is further configured to: In response to an obstacle avoidance instruction, controlling the light emitting component to project a correction image, wherein the correction image includes a pure color image card and a feature image card; Controlling the camera to take pictures of the corrected image displayed on the projection surface when the light emitting component projects the pure color chart and the characteristic chart, respectively, to obtain a first sampling image and a second sampling image; Identifying the first obstacle target in the first sample image; The transformation matrix is ​​established according to the shape of the feature map in the second sampling image.

3. The projection device according to claim 2, wherein: The controller controls the light emitting component to project a corrected image, and is further configured to: Detecting the projection state of the light emitting component at the moment when the obstacle avoidance instruction is input; If the light emitting component is projecting a media image, inserting the pure color image card and the feature image card into the media data stream to be projected; Calculating the shooting time according to the insertion positions of the pure color image card and the feature image card in the media asset data stream; The shooting time is sent to the camera, so that the camera shoots and obtains the first sampling image and the second sampling image according to the shooting time.

4. The projection device according to claim 2, wherein: The controller executes obtaining the transformation matrix and is further configured to: Traversing color values ​​of pixels in the second sampled image, and identifying feature points in the second sampled image according to the color values; Extracting feature distribution information from a feature map corresponding to the second sampling image; Establishing a homography matrix of coordinates between the camera and the light output component according to the identified feature points and the feature distribution information; The homography matrix is ​​stored to obtain the transformation matrix.

5. The projection device according to claim 1, wherein: The controller performs identification of the first obstacle target in the sample image, and is further configured to: Calling a recognition model according to the type of the first obstacle target specified by the user, wherein the recognition model is a neural network model trained based on sample image data; Inputting the sampled image into the recognition model; Obtaining a recognition result output by the recognition model, wherein the recognition result is a classification probability that the sampled image contains the first obstacle target; If the recognition result includes the first obstacle target, executing the step of demarcating a projectable area in the sample image according to the first obstacle target; If the recognition result does not include the first obstacle target, first prompt information is generated, and the light emitting component is controlled to project the first prompt information.

6. The projection device according to claim 1, wherein: The adjustment instruction also includes projection parameters specified by the user, and the controller determines the target projection area according to the projectable area and the transformation matrix, and is further configured to: According to the transformation matrix, the projectable area is fitted into an effective projection range, where the effective projection range is the area range of the projectable area mapped in the light output component coordinate system; extracting the projection parameters from the adjustment instruction; The target projection area is defined in the effective projection range according to the projection parameters.

7. The projection device according to claim 6, wherein: The projection parameters include a specified screen size, and the controller is configured to define the target projection area in the effective projection range according to the projection parameters, and is further configured to: Obtaining the boundary size of the effective projection range; If the boundary size is greater than or equal to the specified screen size, demarcating the target projection area within the effective projection range according to the specified screen size; If the border size is smaller than the specified screen size, second prompt information is generated, and the light emitting component is controlled to project the second prompt information.

8. The projection device according to claim 7, wherein: The projection parameters further include a specified interval distance, and the controller is configured to define the target projection area in the effective projection range according to the projection parameters, and is further configured to: Calculating an extreme size according to the specified screen size and the specified interval distance, the extreme size including a minimum width and a minimum height, the minimum width being the sum of a specified screen width in the specified screen size and a specified horizontal distance in the specified interval distance; the minimum height being the sum of a specified screen height in the specified screen size and a specified vertical distance in the specified interval distance; Obtaining the effective width and effective height of the effective projection range; If the effective width is smaller than the minimum width, and / or the effective height is smaller than the minimum height, generating third prompt information, and controlling the light emitting component to project the third prompt information; If the effective width is greater than or equal to the minimum width, and the effective height is greater than or equal to the minimum height, the target projection area is delineated within the effective projection range according to the specified interval distance and the specified screen size.

9. The projection device according to claim 1, wherein: The controller is configured to define a projectable area in the sample image according to the first obstacle target, and is further configured to: Identify a second obstacle target in the projectable area, where the second obstacle target is a target other than the first obstacle target in the sampled image; If the second obstacle target exists in the projectable area, generating fourth prompt information, and / or generating a query instruction; controlling the light emitting component to project the fourth prompt information, and / or output the inquiry instruction; Obtaining a confirmation instruction input by the user based on the fourth prompt information and / or the inquiry instruction; In response to the confirmation instruction, the projectable area in the sample image is redefined according to the second obstacle target.

10. A projection image processing method, applied to a projection device, wherein the projection device includes a light output component, a camera, and a controller; the projection image processing method includes: In response to the obstacle avoidance instruction, obtaining a transformation matrix and an adjustment instruction input by a user, wherein the transformation matrix is ​​a homography matrix of coordinates between the camera and the light output component; the adjustment instruction includes a first obstacle target specified by the user; Identifying the first obstacle target in a sampled image, where the sampled image is an image captured by the camera when the light emitting component projects the corrected image; Delimiting a projectable area in the sampled image according to the first obstacle target, wherein the projectable area is a rectangular area with a maximum preset aspect ratio that is accommodated by an area in the sampled image excluding the first obstacle target; A target projection area is determined according to the projectable area and the transformation matrix, and the light output component is controlled to project the projection content onto the target projection area.

11. A projection device comprising: A light emitting component configured to project projection content onto a projection surface; a camera configured to capture a sample image; The controller is configured as: In response to a projection picture correction instruction, controlling the light output component to project a correction image, wherein the correction image includes a pure color image card and a feature image card; Acquire a first sampling image obtained by photographing the solid color chart with the camera, and a second sampling image obtained by photographing the feature chart with the camera; determining a characteristic contour region according to the first sampling image, wherein the characteristic contour region is a contour region with the largest area in the first sampling image or a contour region specified by a user; extracting feature points in the second sampling image according to the feature contour area; Calculate the angle between the projection surface and the light emitting component based on the feature point, and control the light emitting component to project according to the angle Project content onto the projection surface.

12. The projection device according to claim 11, wherein: The controller determines a feature contour area according to the first sampling image, and is further configured to: Extracting a contour area in the first sampling image; Detect the setting status of the obstacle avoidance switch; If the setting state is on, traversing the areas of the contour regions in the first sampling image to screen out a preset number of candidate contour regions, the candidate contour regions being used for the user to designate as the characteristic contour regions; If the setting state is off, the areas of the contour regions in the first sampling image are traversed to select the contour region with the largest area as the characteristic contour region.

13. The projection device according to claim 12, wherein: After the controller traverses the areas of the contour regions in the first sampling image to screen out a preset number of candidate contour regions, the controller is further configured to: Calculating the average depth of feature points in the second sampling image that are located within the same candidate contour area; Calculating an area ratio of the candidate contour region relative to the second sampling image; generating first prompt information according to the average depth and the area ratio, and controlling the light emitting component to project the first prompt information; Acquire a selection instruction input by the user based on the first prompt information; In response to the selection instruction, the candidate contour area specified in the selection instruction is marked as the characteristic contour area.

14. The projection device according to claim 13, wherein: The controller calculates the average depth of feature points in the second sample image that are located within the same candidate contour area, and is further configured to: Traversing the color values ​​of the feature points in the candidate contour area; Extract the projection shape according to the color values ​​of multiple feature points; Obtaining hardware parameters of the camera and the light output component, and obtaining a standard shape on a feature map card in the second sampled image; Calculating the distance between the feature point and the light emitting component according to the projection shape, the standard shape, and the hardware parameters of the camera; An average value of distances between a plurality of the feature points and the light output component is calculated to obtain the average depth.

15. The projection device according to claim 12, wherein: The controller is configured to extract a contour area from the first sample image and further: Traversing the color values ​​of pixels in the first sampling image; Identify a boundary pattern based on the color values ​​of the pixel points, where the boundary pattern is a pattern consisting of pixel points whose color difference values ​​with adjacent pixel points are greater than or equal to a color difference threshold; The contour area is delineated according to the boundary graphic and the edge of the first sampling image.

16. The projection device according to claim 11, wherein: The controller calculates the angle between the projection surface and the light output component based on the feature points, and is further configured to: Obtaining the coordinates of the feature points in the feature contour area in the camera coordinate system; According to the hardware parameters of the camera and the light emitting component, the feature point coordinates are converted into light emitting point coordinates in the light emitting component coordinate system; Fitting a projection surface in the light emitting component coordinate system according to the coordinates of the plurality of light emitting points; Calculate the angle between the projection surface and the light emitting surface corresponding to the light emitting component.

17. The projection device according to claim 11, wherein: The controller controls the light emitting component to project the projection content onto the projection surface according to the included angle, and is further configured to: Obtaining operating parameters of the light emitting component; Calculating a projectable area according to the operating parameters and the angle; Delimiting a target projection area in the projectable area, wherein the target projection area is the largest inscribed rectangular area in the projectable area, and the rectangular area has a preset aspect ratio; Inversely transforming the coordinates of the target projection area to the light-emitting surface of the light-emitting component according to the operating parameters and the angle; The light emitting component is controlled to project the projection content according to the inverse-transformed target projection area coordinates.

18. The projection device according to claim 17, wherein: The controller is configured to define a target projection area within the projectable area, and further configured to: Detecting an obstacle avoidance command input by a user for activating an obstacle avoidance function; If the obstacle avoidance instruction is detected, demarcating the target projection area according to the feature points; If the obstacle avoidance instruction is not detected, the target projection area is delineated according to the vertex coordinates of the light emitting surface of the light emitting component.

19. The projection device according to claim 11, wherein: The controller determines a feature contour area according to the first sampling image, and is further configured to: Inputting the first sample image into a recognition model, wherein the recognition model is a neural network model trained based on the sample image; Obtaining a recognition result output by the recognition model, wherein the recognition result is a classification probability that the first sampled image contains an obstacle target; If the recognition result indicates that an obstacle target is included, removing a contour region corresponding to the obstacle target from the first sample image, and determining a characteristic contour region in the first sample image after removing the obstacle target; If the recognition result shows that no obstacle target is included, a feature contour area is determined according to the first sampling image.

20. A projection image processing method, applied to a projection device, wherein the projection device includes a light output component, a camera, and a controller; the projection image processing method comprises: In response to a projection picture correction instruction, controlling the light output component to project a correction image, wherein the correction image includes a pure color image card and a feature image card; Acquire a first sampling image obtained by photographing the solid color chart with the camera, and a second sampling image obtained by photographing the feature chart with the camera; determining a characteristic contour region according to the first sampling image, wherein the characteristic contour region is a contour region with the largest area in the first sampling image or a contour region specified by a user; extracting feature points in the second sampling image according to the feature contour area; An included angle between the projection surface and the light emitting component is calculated based on the feature points, and the light emitting component is controlled to project the projection content onto the projection surface according to the included angle.

21. A projection device comprising: A light emitting component configured to project projection content onto a projection surface; The controller is configured as: Acquire a screen movement instruction input by a user, wherein the screen movement instruction includes a movement direction and a movement distance; In response to the image movement instruction, obtaining the vertex coordinates of the current light-emitting surface and the rotation matrix between the projection surface and the light-emitting surface; Calculating a current projection area according to the rotation matrix and the vertex coordinates of the current light-emitting surface; Calculating a vertex moving distance according to the vertex coordinates of the current projection area, the moving direction, and the moving distance; Calculate the target projection coordinates according to the vertex movement distance and the vertex coordinates of the current projection area; Based on the rotation matrix, the target projection coordinates are converted to the light-emitting surface to obtain light-emitting projection coordinates, and the light-emitting component is controlled to project the projection content onto the projection surface according to the light-emitting projection coordinates.

22. The projection device according to claim 21, further comprising a camera configured to capture an image of the projected content; and the controller is further configured to: Acquire a projection image captured by the camera, and identify coordinates of feature points in the projection image in a camera coordinate system; According to the hardware parameters of the light emitting component and the camera, the coordinates of the feature points are converted into a light emitting component coordinate system; Fitting a projection plane according to the coordinates of the feature points in the light emitting component coordinate system, and calculating an angle between the projection plane and the light emitting surface; The rotation matrix is ​​constructed according to the angle between the projection plane and the light emitting surface.

23. The projection device according to claim 21, wherein: The controller calculates the vertex moving distance according to the vertex coordinates of the current projection area, the moving direction, and the moving distance, and is further configured to: Obtaining the movement mode to which the movement direction belongs; If the movement mode is left-right movement, calculating a projection width ratio according to the vertex coordinates of the current projection area, and calculating the vertex movement distance according to the projection width ratio and the movement distance; If the movement mode is up and down movement, the projection height ratio is calculated according to the vertex coordinates of the current projection area, and the vertex movement distance is calculated according to the projection height ratio and the movement distance.

24. The projection device according to claim 23, wherein: The controller is further configured to: If the movement mode is left-right movement, calculate the first horizontal coordinate difference and the second horizontal coordinate difference of the current projection area, the first horizontal coordinate difference is the difference between the horizontal coordinates of the two vertices on the upper side of the current projection area, and the second horizontal coordinate difference is the difference between the horizontal coordinates of the two vertices on the upper side of the current projection area. The length of the lower side of the shadow area is the difference between the horizontal coordinates of the two vertices; Calculating a projection width ratio according to the first horizontal coordinate difference and the second horizontal coordinate difference, wherein the projection width ratio is an average of the first horizontal coordinate difference and the second horizontal coordinate difference; If the movement mode is up and down movement, calculate a first ordinate difference and a second ordinate difference of the current projection area, where the first ordinate difference is the difference between the ordinates of the two vertices on the left side of the current projection area, and the second ordinate difference is the difference between the ordinates of the two vertices on the right side of the current projection area; A projection height ratio is calculated according to the first ordinate difference and the second ordinate difference, where the projection height ratio is an average of the first ordinate difference and the second ordinate difference.

25. The projection device according to claim 23, wherein: The controller calculates target projection coordinates according to the vertex movement distance and the vertex coordinates of the current projection area, and is further configured to: If the movement mode is left-right movement, the horizontal coordinate of the vertex of the current projection area is moved by the vertex movement distance in the movement direction to obtain the target projection coordinate; If the movement mode is up and down movement, the vertical coordinates of the vertices of the current projection area are moved by the vertex movement distance in the movement direction to obtain target projection coordinates.

26. The projection device according to claim 25, wherein: If the movement mode is left-right movement, the controller is further configured to: Get the installation method of the projection equipment; If the installation method is orthographic projection, the horizontal coordinate of the vertex of the current projection area is moved by the vertex movement distance in the moving direction; If the installation method is back projection, the horizontal coordinates of the vertices of the current projection area are moved in the opposite direction of the moving direction by the vertex moving distance.

27. The projection device according to claim 21, wherein: The controller is further configured to: Calculating a maximum projection area according to the rotation matrix and the vertex coordinates of the maximum light-emitting surface; If the maximum projection area includes the target projection coordinates, performing a step of converting the target projection coordinates to the light exit surface; If the maximum projection area does not include the target projection coordinates, first prompt information is generated, and the light output component is controlled to project the first prompt information.

28. The projection device according to claim 27, wherein: The controller is further configured to: Determine a side length vector and a connection vector according to the vertex coordinates of the maximum projection area and the target projection coordinates, wherein the side length vector is a vector between the coordinates of two adjacent vertices of the maximum projection area, and the connection vector is a vector between the starting coordinates of each side length vector and the target projection coordinates; Perform vector cross multiplication on the side length vector and the connection vector respectively to obtain a cross product vector; If the values ​​of the cross product vectors have the same sign or are zero, determining that the maximum projection area includes the target projection coordinates; If the values ​​of the cross product vectors have different signs, it is determined that the maximum projection area does not include the target projection coordinates.

29. The projection device according to claim 21, wherein: The controller is further configured to: In response to the picture movement instruction, detecting a movement direction in the picture movement instruction; If the image movement instruction includes an opposite movement direction, second prompt information is generated, and the light emitting component is controlled to project the second prompt information.

30. A projection image processing method, applied to a projection device, wherein the projection device includes a light output component and a controller; the projection image processing method comprises: Acquire a screen movement instruction input by a user, wherein the screen movement instruction includes a movement direction and a movement distance; In response to the image movement instruction, obtaining the vertex coordinates of the current light-emitting surface and the rotation matrix between the projection surface and the light-emitting surface; Calculating a current projection area according to the rotation matrix and the vertex coordinates of the current light-emitting surface; Calculating a vertex moving distance according to the vertex coordinates of the current projection area, the moving direction, and the moving distance; Calculate the target projection coordinates according to the vertex movement distance and the vertex coordinates of the current projection area; Based on the rotation matrix, the target projection coordinates are converted to the light-emitting surface to obtain light-emitting projection coordinates, and the light-emitting component is controlled to project the projection content onto the projection surface according to the light-emitting projection coordinates.

Images