A projection device and a method for determining a projection area

Abstract

Embodiments of the present application disclose a projection device and a method for determining a projection area. The method comprises: a depth sensor configured to obtain an image of a projection surface; an optical machine configured to project a media resource; and a controller configured to: in response to a first instruction, obtain a first image of the projection surface by the depth sensor; use the first image as an input of a pre-trained first neural network model, and obtain a recognition result by the first neural network model; the first neural network model is used to identify whether a target object exists on the projection surface; determine an execution strategy for obtaining a target projection area based on the recognition result, the target projection area being an area for displaying a projection image; and obtain the target projection area in the projection surface based on the execution strategy. The method can reduce interaction with the projection device, automatically determine a projection mode for obtaining an optimal projection area, and use a neural network model to improve the efficiency of obtaining the optimal projection area and improve user experience.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of projection equipment, and in particular to a projection equipment and a projection area determination method. BACKGROUND

[0002] As a key tool in modern visual display technology, projection equipment has been widely used in various fields such as education, business, entertainment, and home theater. In these application scenarios, it is crucial to ensure that the content is clearly and completely presented to the audience. In order to provide the best viewing experience, accurately determining and setting the projection area becomes one of the basic steps to achieve high-quality projection effects. The projection area referred to here refers to the specific display range of the projected image. It defines the effective area on the screen that is actually used to display the content, and plays an important role in ensuring image quality, adapting to different scene requirements, and improving audience experience. By accurately setting the projection area, every detail can be displayed in the most ideal way, providing users with a more immersive visual experience.

[0003] Currently, when determining the projection area, the projection equipment usually needs to interact with the user multiple times to meet the specific operation requirements. For example, in order to realize the curtain-in function, the user often needs to start by pressing a specific button or selecting an option in the menu; similarly, obstacle avoidance and automatic correction functions also require the user to operate through similar interaction methods. After obtaining these instructions, the traditional method involves placing one or two full-screen picture cards and using these picture cards in combination with traditional image processing algorithms to calculate the projection area.

[0004] However, this method requires the user to judge and perform a series of steps on their own, resulting in frequent interactions and long waiting times, which affects the user experience. Not only does the user need to spend extra time waiting, but due to the slow processing speed of traditional image processing algorithms, the preparation time is further extended, reducing overall user satisfaction. SUMMARY

[0005] Some embodiments of the present application provide a projection equipment and a projection area determination method, which can reduce interaction with the projection equipment, automatically determine and obtain the best projection area projection method, and at the same time, use a neural network model to improve the efficiency of obtaining the best projection area, improving user experience.

[0006] In a first aspect, an embodiment of the present application provides a projection equipment, comprising:

[0007] a depth sensor configured to obtain an image of a projection surface;

[0008] an optical machine configured to project a media resource;

[0009] a controller configured to:

[0010] acquire, in response to the first instruction, a first image of the projection surface by the depth sensor;

[0011] input the first image into a first neural network model pre-trained, and obtain a recognition result by the first neural network model; the first neural network model is configured to recognize whether a target object exists on the projection surface;

[0012] determine an execution strategy for acquiring a target projection area based on the recognition result; the target projection area is an area for displaying a projection image; the execution strategy has a corresponding relationship with the recognition result;

[0013] acquire the target projection area in the projection surface based on the execution strategy.

[0014] The above technical solution has the following advantages or benefits: by responding to the first instruction, the image of the projection surface is acquired by the depth sensor, and the image is input into the first neural network model pre-trained for analysis. The model can recognize the recognition result on the projection surface, and based on the recognition result, different execution strategies are determined to define the target projection area. In this way, not only the degree of automation of the projection setting is improved, and the interaction steps with the user are reduced, but also the user's waiting time is significantly reduced by using the model processing, the processing efficiency of acquiring the target projection area is improved, and the user experience is improved.

[0015] In some embodiments, the target object includes at least one of a curtain or an obstacle; the controller executes the step of determining an execution strategy for acquiring a target projection area based on the recognition result, which is specifically configured as:

[0016] in the case that the recognition result includes a curtain, the execution strategy for acquiring a target projection area is determined as an entry-curtain strategy;

[0017] in the case that the recognition result includes the curtain and does not include an obstacle, the execution strategy for acquiring a target projection area is determined as an obstacle-avoiding strategy;

[0018] in the case that the recognition result includes the curtain and includes the obstacle, the execution strategy for acquiring a target projection area is determined as a non-sensing correction strategy.

[0019] The above technical solution has the following advantages or benefits: different execution strategies can be automatically selected based on different recognition results, the number of interactions between the user and the projection device is reduced, and the user experience is improved.

[0020] In some embodiments, the controller executes the step of acquiring the target projection area in the projection surface based on the execution strategy, which is specifically configured as:

[0021] determine a first target coordinate in the optical-mechanical coordinate system based on the execution strategy; the first target coordinate is a corner point coordinate of the target projection region in the projection surface in the optical-mechanical coordinate system;

[0022] convert the first target coordinate based on a first conversion relationship to obtain a second target coordinate in the physical coordinate system; the first conversion relationship is a mapping relationship between the optical-mechanical coordinate system and the physical coordinate system;

[0023] obtain the target projection region in the projection surface based on the second target coordinate.

[0024] The above technical solution has the following advantages or benefits: based on different execution strategies, a first target coordinate in the optical-mechanical coordinate system can be determined, then a first conversion relationship between the optical-mechanical coordinate system and the physical coordinate system is used to convert the first target coordinate into a second target coordinate in the physical coordinate system. Finally, based on the second target coordinate, the target projection region in the projection surface can be accurately determined. In this way, the accuracy of obtaining the target projection region can be improved through coordinates and conversion relationships, which facilitates the determination of the best projection region for displaying the projected image and improves the user experience.

[0025] In some embodiments, the controller executes determining a first target coordinate in the optical-mechanical coordinate system based on the execution strategy, which is specifically configured as:

[0026] In the case where the execution strategy is the curtain-in strategy, a second image corresponding to the projection surface is obtained; the second image includes a curtain object and a positioning card object, and the positioning card object is arranged in a preset region of the curtain object;

[0027] obtain a first curtain coordinate corresponding to the curtain object and a first card coordinate corresponding to the positioning card object in the second image; the first curtain coordinate is a corner point coordinate of the curtain object corresponding to the camera coordinate system, and the first card coordinate is a corner point coordinate of the positioning card object corresponding to the camera coordinate system;

[0028] perform an updating operation on the first curtain coordinate and the first card coordinate to determine the updated first curtain coordinate and the corresponding first weight, and the updated first card coordinate and the corresponding second weight; wherein the updated first curtain coordinate is a curtain coordinate obtained after the updating operation in the second image coordinate system; the updated first card coordinate is a card coordinate obtained after the updating operation in the second image coordinate system, and the updating operation includes at least one of moving, scaling, and rotating;

[0029] determine the first target coordinate corresponding to the target projection area in the optical-mechanical coordinate system based on the updated first curtain coordinate, the first weight, the updated first card coordinate, and the second weight.

[0030] The technical solution has the following beneficial effects or advantages: in the case of entering the curtain, by placing the positioning card in the preset area on the curtain, the corner point coordinates of the curtain and the positioning card can be obtained without affecting the user's viewing, the first target coordinate in the optical-mechanical coordinate system can be determined more accurately, and the accuracy of subsequent target projection area determination is significantly improved. In this way, not only the accurate alignment and display of the projection content are ensured, but also the user's viewing experience is not disturbed, and the user experience is improved.

[0031] In some embodiments, the camera is configured to capture the image of the projection surface, and the controller is configured to:

[0032] The camera is controlled to capture a first captured image, and the first captured image includes the curtain object and the positioning card object.

[0033] The first optical-mechanical coordinate is converted to the camera coordinate system based on a second conversion relationship to obtain a first camera coordinate; the first optical-mechanical coordinate is the corner point coordinate corresponding to the boundary region projected by the optical machine; and the second conversion relationship is a mapping relationship between the camera coordinate system and the optical-mechanical coordinate system.

[0034] The first captured image is cropped based on the first camera coordinate to obtain the second image; and the main body area of the second image is larger than the main body area of the first captured image.

[0035] The technical solution has the following beneficial effects or advantages: after the camera captures the second image containing the curtain and the card, since the image may contain other non-projection areas, the optimal projection area coordinate can be determined using the coordinates and the conversion relationship. Based on these accurate coordinates, image cropping can effectively reduce environmental interference of non-projection areas, ensure that subsequent processing focuses on the actual projection area, and thus improve the accuracy and effect of projection setting.

[0036] In some embodiments, the controller is configured to determine updated first curtain coordinates and corresponding first weights, and updated first card coordinates and corresponding second weights based on the first curtain coordinates and the first card coordinates, and specifically configured to:

[0037] crop the second image based on the first curtain coordinate and the first card coordinate to obtain at least one first cropped image corresponding to the first curtain coordinate and at least one second cropped image corresponding to the first card coordinate;

[0038] take the at least one first cropped image and the at least one second cropped image as inputs of a second neural network model, and obtain, by using the second neural network model, a second curtain coordinate and a corresponding first weight, and a second card coordinate and a corresponding second weight; the second curtain coordinate is a curtain coordinate corresponding to the first cropped image, and the second card coordinate is a card coordinate corresponding to the second cropped image;

[0039] convert the second curtain coordinate and the second card coordinate to a coordinate system of the second image to obtain the updated first curtain coordinate and the updated first card coordinate.

[0040] The above technical solution has the following beneficial effects or advantages: the first image is cropped based on the first curtain coordinate and the first card coordinate, and the cropped image is input into the second neural network model to update the first curtain coordinate and the first card coordinate, so that the accuracy of coordinate determination can be improved.

[0041] In some embodiments, the controller is configured to determine the first target coordinate corresponding to the target projection area in the optical-mechanical coordinate system based on the updated first curtain coordinate, the first weight, the updated first card coordinate, and the second weight, specifically by being configured to:

[0042] determine a third curtain coordinate of the second image based on the first weight; the third curtain coordinate is a curtain coordinate with the highest weight value in the first weight;

[0043] determine a third card coordinate of the second image based on the second weight; the third card coordinate is a card coordinate with the highest weight value in the second weight;

[0044] determine a third conversion relationship based on the third card coordinate and a preset card coordinate in the optical-mechanical coordinate system; the third conversion relationship is a mapping relationship between the camera coordinate system and the optical-mechanical coordinate system;

[0045] convert the third curtain coordinate based on the third conversion relationship to obtain the first target coordinate corresponding to the target projection area in the optical-mechanical coordinate system.

[0046] The above technical scheme has the following beneficial effects or advantages: by analyzing the plurality of first weights and the plurality of second weights, a group of second curtain coordinates corresponding to the highest weight in the first weight and a group of second picture card coordinates corresponding to the highest weight in the second weight are selected, and the coordinates are taken as the final curtain corner point coordinates and picture card corner point coordinates respectively. In this way, by selecting the most representative coordinate point, the accuracy of the first target coordinate is significantly improved, thereby ensuring the accurate positioning of the target projection area.

[0047] In some embodiments, the controller determines the first target coordinate in the optical-mechanical coordinate system based on the execution strategy, and is specifically configured to:

[0048] In the case where the execution strategy is the obstacle avoidance strategy, a binary image corresponding to the first image and a first pixel coordinate corresponding to the binary image are obtained; the first pixel coordinate includes a pixel coordinate corresponding to the obstacle and a pixel coordinate corresponding to a non-obstacle;

[0049] The first optical-mechanical coordinate is converted based on a fourth conversion relationship to obtain a first depth coordinate in a depth sensor coordinate system; the fourth conversion relationship is a mapping relationship between the optical-mechanical coordinate system and the depth sensor coordinate system;

[0050] Based on the first pixel coordinate and the first depth coordinate, a third target coordinate in the depth sensor coordinate system is determined, and the third target coordinate is a corner point coordinate of the target projection area in the projection surface in the depth sensor coordinate system;

[0051] The third target coordinate is converted based on the fourth conversion relationship to obtain the first target coordinate in the optical-mechanical coordinate system.

[0052] The above technical scheme has the following beneficial effects or advantages: in the case where there is no curtain but there is an obstacle, the specific pixel coordinates of the obstacle can be determined by analyzing the binary image. Then, a maximum possible projection area is defined by using the first optical-mechanical. In this area, by excluding those pixel positions marked as obstacles, a maximum available projection area that is not hindered, i.e., the target projection area, can be calculated, thereby ensuring the optimization of the projection area.

[0053] In some embodiments, the controller determines the first target coordinate in the optical-mechanical coordinate system based on the execution strategy, and is specifically configured to:

[0054] In the case where the execution strategy is the no-sense correction strategy, a first optical-mechanical coordinate is taken as the first target coordinate in the optical-mechanical coordinate system; wherein the first optical-mechanical coordinate is a corner point coordinate corresponding to a boundary region projected by the optical-mechanical.

[0055] The technical scheme has the following beneficial effects or advantages: in the absence of a curtain and an obstacle, the corner point coordinate corresponding to the first light machine coordinate can be taken as the first target coordinate, so as to determine the optimal projection area, and the somatic user experience is improved.

[0056] In a second aspect, the embodiments of the present application further provide a method for determining a projection area, comprising:

[0057] In response to the first instruction, the first image of the projection surface is acquired by the depth sensor;

[0058] The first image is taken as the input of the pre-trained first neural network model, and the first neural network model is used to obtain the recognition result; the first neural network model is used to identify whether there is a target object on the projection surface;

[0059] Based on the recognition result, an execution strategy for acquiring a target projection area is determined, the target projection area being a region for displaying a projection image; the execution strategy has a corresponding relationship with the recognition result;

[0060] Based on the execution strategy, the target projection area in the projection surface is acquired.

[0061] The technical scheme has the following beneficial effects or advantages: the technical scheme has the following beneficial effects or advantages: by responding to the first instruction, the image of the projection surface is acquired by the depth sensor, and the image is input into the pre-trained first neural network model for analysis. The model can identify the recognition result on the projection surface, and based on the recognition result, different execution strategies are determined to define the target projection area. In this way, not only the automation degree of the projection setting is improved, and the interaction steps with the user are reduced, but also the model processing significantly reduces the waiting time of the user, improves the processing efficiency of acquiring the target projection area, and improves the user experience. BRIEF DESCRIPTION OF DRAWINGS

[0062] Figure 1 The projection scene schematic diagram of the projection device provided by some embodiments of the present application is provided;

[0063] Figure 2 The optical path schematic diagram of the projection device provided by some embodiments of the present application is provided;

[0064] Figure 3 The circuit architecture schematic diagram provided by some embodiments of the present application is provided;

[0065] Figure 4 The projection device structure schematic diagram provided by some embodiments of the present application is provided;

[0066] Figure 5 The system framework schematic diagram for realizing display control of the projection device provided by some embodiments of the present application is provided;

[0067] Figure 6 A schematic diagram of conversion between coordinate systems provided for some embodiments of the present application;

[0068] Figure 7 A first flowchart of obtaining a target projection region provided for some embodiments of the present application;

[0069] Figure 8 A schematic diagram of a first image provided for some embodiments of the present application;

[0070] Figure 9 A flowchart of obtaining a recognition result provided for some embodiments of the present application;

[0071] Figure 10 A first flowchart of obtaining a first target coordinate provided for some embodiments of the present application

[0072] Figure 11 A second flowchart of obtaining a first target coordinate provided for some embodiments of the present application;

[0073] Figure 12 A schematic diagram of a first captured image and a second image provided for some embodiments of the present application;

[0074] Figure 13 A flowchart of cropping a second image provided for some embodiments of the present application;

[0075] Figure 14 A third flowchart of obtaining a first target coordinate provided for some embodiments of the present application;

[0076] Figure 15 A binary image provided for some embodiments of the present application;

[0077] Figure 16 A schematic diagram of determining a region corresponding to a third target coordinate provided for some embodiments of the present application;

[0078] Figure 17 A second method flowchart of obtaining a target projection region provided for some embodiments of the present application. DETAILED DESCRIPTION

[0079] In order to make the purpose and implementation of the present application more clear, the present application will be described clearly and completely in conjunction with the drawings of the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only some of the embodiments of the present application, but not all of the embodiments.

[0080] It should be noted that the brief description of the terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of the application. Unless otherwise specified, these terms should be understood in accordance with their ordinary and general meanings.

[0081] The terms "first", "second", "third", and the like in the specification and claims of this application and the above-described drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit the specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchanged under appropriate circumstances.

[0082] The terms "include" and "have" and any variations thereof are intended to cover but not exclusive inclusion, for example, a product or device including a series of components does not necessarily limit to all components clearly listed, but can include other components not clearly listed or inherent to these products or devices.

[0083] The projection device is a device that can project media data onto a projection medium. The projection device can be connected to a computer, a broadcast network, the Internet, a VCD (Video Compact Disc), a DVD (Digital Versatile Disc Recordable), a game console, a DV, etc. through different interfaces to receive media data that needs to be projected. The media data includes but is not limited to images, videos, texts, etc. The projection medium includes but is not limited to walls, curtains, screens, etc.

[0084] Figure 1 The projection scene diagram of the projection device provided by some embodiments of the application is shown.

[0085] In some embodiments, the projection device 100 can be a projector, a laser television, etc. with projection function. The type of the projection device is not limited in the application. The projection device is used to project a projection picture to a projection medium. Taking the projection device as a projector, referring to Figure 1 , the projector can include a projection host 2. The projection medium 1 is fixed at a first position, and the projection host 2 is placed at a second position. By adjusting the relationship between the first position and the second position, the projection picture of the projection host 2 is matched with the projection medium 1, that is, the second position is the best placement position of the projection host 2. The projection medium can be a curtain, a white wall, etc.

[0086] Figure 2 The light path diagram of the projection device provided by some embodiments of the application is shown.

[0087] The projection host 2 comprises a projection assembly, which comprises a light source 210, a light machine 220 and a lens 230. The light source 210 provides illumination for the light machine 220, the light machine 220 modulates the light beam of the light source and outputs to the lens 230, and the lens 230 images and projects to the projection medium 1, and the projection medium 1 presents the projection picture.

[0088] In some embodiments, the light source 210 can include a bulb assembly or an LED (Light Emitting Diode) light source, and the light beam emitted by the light source can be modulated and adjusted by the light machine 220, thereby providing the required light source for the projection picture. In this case, the light machine 220 is usually composed of different optical components, including color separation devices and adjustment devices, to process the light beam emitted by the light source and convert it into a projection picture.

[0089] In some embodiments, the light machine 220 can also include blue, green and red light machine modules, which generate laser or light sources suitable for displaying different colors to achieve clear and accurate projection pictures. In addition, the light machine 220 is usually equipped with a heat dissipation system and a circuit control system to ensure stable operation of the device.

[0090] In some embodiments, the light-emitting components of the projector can be realized by various means such as bulbs, LEDs or laser light sources, and different types of light sources determine the brightness, color accuracy and service life of the projector.

[0091] Figure 3 The circuit architecture schematic diagram provided for some embodiments of the present application.

[0092] In some embodiments, referring to Figure 3 , the projection host 2 can include a display control circuit 240, a laser light source 210, at least one laser driver assembly 250 and at least one brightness sensor 260. The laser light source 210 can include at least one laser corresponding to the at least one laser driver assembly one-to-one. Wherein, the at least one means one or more, and the plurality means two or more than two.

[0093] In some embodiments, the display control circuit 240 is used to output light control signals corresponding to different primary colors to the laser driver assembly 250 to drive the corresponding laser to emit light, for example, the light control signals include blue light control signals, red light control signals and green light control signals. Referring to Figure 3The display control circuit 240 is connected with the laser driver assembly 250, and is configured to output at least one light control signal corresponding to each of the three primary colors of the multi-frame display image, and transmit the at least one light control signal to the corresponding laser driver assembly 250. For example, the display control circuit 240 can be a microcontroller unit (MCU), also known as a single-chip microcomputer.

[0094] In some embodiments, the projector can realize adaptive adjustment. For example, by arranging a brightness sensor 260 in the light path of the laser light source 210, the brightness sensor 260 can detect a first brightness value of the laser light source 210 and send the first brightness value to the display control circuit 240. The display control circuit 240 can obtain a second brightness value corresponding to the driving current of each laser, and when the difference between the second brightness value of the laser and the first brightness value of the laser is greater than a difference threshold value, it is determined that the laser has a COD (Catastrophic optical damage) failure. Then the display control circuit 240 can adjust the current control signal of the laser driver assembly corresponding to the laser until the difference is less than or equal to the difference threshold value, thereby eliminating the COD failure of the laser, reducing the damage rate of the laser, and improving the image display effect of the projection device.

[0095] Figure 4 The projection device provided by some embodiments of the present application is shown in the structural schematic diagram.

[0096] In some embodiments, referring to Figure 4 The optical path structure includes a laser light source 210 and an optical assembly 214. The laser light source 210 can include independently arranged blue lasers 211, red lasers 212 and green lasers 213. The projection device can also be referred to as a three-color projection device. The blue lasers 211, the red lasers 212 and the green lasers 213 are all Mirai Console Loader (MCL) packaged lasers, which are small in size and conducive to compact arrangement of the optical path.

[0097] In some embodiments, the projection host 2 can include a controller including at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), a RAM (random access memory), a ROM (read-only memory), a first interface to an n-th interface for input / output, a communication bus, and the like. The controller is connected with the related hardware of the projection device, such as the display control circuit, the brightness sensor, the distance sensor, the image collector, and the like, for controlling the projection, the focusing, the correction, the calibration, the on-off screen state adjustment, and the like of the projection device.

[0098] In some embodiments, the projection device (for example, a laser television) can be provided with a plurality of types of interfaces on the body, such as a power interface, a USB interface, an HDMI (high definition multimedia interface) interface, a network cable interface, a VGA (video graphics array) interface, a DVI (digital visual interface), and the like, for connecting a signal source for transmitting media.

[0099] In some embodiments, the projection device can directly enter a display interface of a last selected signal source or a signal source selection interface after being started, where the signal source is, for example, a preset video on demand program, or one of a signal source such as an HDMI interface, a USB interface, and a live television interface. After a user selects a target signal source, the projection host 2 can acquire media data from the target signal source and project the media data on the projection medium 1 for display.

[0100] In some embodiments, the projection host 2 can be configured with a camera for cooperating with the projection host 2 to achieve related adjustment and control of the projection process. For example, the projection device can be configured with a 3D camera, a monocular camera, or a binocular camera. When the camera is implemented as a binocular camera, the binocular camera specifically includes a left camera and a right camera. The binocular camera can acquire an image and a playing content presented by a projection screen corresponding to the projection device, where the image or the playing content is projected by a light engine built in the projection device.

[0101] When the projection device is moved to a new position, the projection angle and the distance to the projection screen are changed, which can cause the projection image to be deformed and displayed as a trapezoidal image or other distorted image. The controller of the projection device can automatically enter the screen based on the image captured by the camera and by coupling the included angle between the light engine and the projection screen and the correct display of the projection image.

[0102] Figure 5 The projection device provided in some embodiments of the present application implements a system framework schematic diagram of display control.

[0103] In some embodiments, referring to Figure 5 , the system framework includes an application service layer, a process communication framework, an operation layer, a framework layer, a correction service, a camera service, a time of flight service, and hardware and its drivers, etc. The controller of the projection host 2 controls the overall system architecture, and realizes projection control of the projection device based on underlying program logic, including but not limited to automatic curtain entering, automatic obstacle avoidance, automatic focusing, anti-shooting eye, on-off screen control, automatic correction and fine correction of projection picture, etc.

[0104] In some embodiments, the projection host 2 is also configured with a distance sensor for detecting distance. The distance sensor can adopt a time of flight (TOF) sensor. The time of flight sensor measures the distance between nodes by using the flight time of signals between the transmitting end and the reflecting end. After the time of flight sensor collects distance data, the distance data is sent to the time of flight service. After the time of flight service obtains the distance data, the collected distance data is sent to the application service layer through the process communication framework. The distance data will be used for interactive use of data calling of the controller, user interface, program application, etc.

[0105] In some embodiments, the projection host 2 can also be configured with an image collector. The image collector can adopt a monocular camera, a binocular camera, a depth camera or a 3D camera, etc. The image collector sends the collected image data to the camera service, and then the camera service sends the image data to the process communication framework and / or the correction service. The process communication framework sends the image data to the application service layer. The image data will be used for interactive use of data calling of the controller, user interface, program application, etc.

[0106] In some embodiments, the projection device can be refocused after automatic correction is completed. The controller detects whether the automatic focusing function is turned on. If the automatic focusing function is not turned on, the controller will end the automatic focusing service. If the automatic focusing function is turned on, the controller performs focusing calculation according to the distance detection value of the time of flight sensor.

[0107] In some embodiments, the controller queries a preset mapping table based on the distance detection value from the time-of-flight sensor. This preset mapping table records the mapping relationship between distance and focal length, thereby obtaining the focal length of the projection device corresponding to the distance detection value. The middleware then sends the obtained focal length to the optical engine of the projection device. After the optical engine emits a laser according to the aforementioned focal length, at least one image acquisition device captures an image of the projected content. The controller performs a sharpness detection on the projected content image to determine if the current lens focal length is suitable. If the focal length is unsuitable, refocusing is required. The projection device locates the focus position with the highest sharpness by adjusting the lens position, taking another image, and comparing the change in sharpness of the projected content image before and after adjustment.

[0108] If the judgment result meets the preset completion conditions, the automatic focus adjustment process ends; if the judgment result does not meet the preset completion conditions, the middleware will fine-tune the focal length parameters of the projector's optical engine, for example, by gradually fine-tuning the focal length according to a preset step size, and then setting the adjusted focal length parameters back to the optical engine. Through multiple steps such as taking pictures and evaluating sharpness, the optimal focal length is finally locked by comparing the sharpness of the projected image, thereby completing the automatic focus adjustment.

[0109] In some embodiments, to enhance the viewing experience, the projection device needs to correct the light-emitting components when content is first projected onto the projection surface, or when the device shifts during projection. This ensures that the projected content is presented completely and stably on the projection surface. Within a projection device, there are several key components, each located in an independent coordinate system. For example, the projection surface has its own physical coordinate system, the optical engine of the projected light operates in its unique optical-mechanical coordinate system, the camera operates based on the camera coordinate system, and the depth sensor operates according to the depth sensor coordinate system. To ensure accurate display of the projected image and automatic correction of the light-emitting components during device movement or initial setup, precise conversion between these different coordinate systems is necessary.

[0110] For example, combined Figure 6 As shown, a transformation relationship 1 is defined between the optical-mechanical coordinate system and the physical coordinate system where the projection surface is located. Transformation relationship 1 corresponds to the first transformation relationship in this application. A transformation relationship 2 is established between the optical-mechanical coordinate system and the camera coordinate system corresponding to the camera. Transformation relationship 2 corresponds to the second transformation relationship in this application. A transformation relationship 3 is set between the optical-mechanical coordinate system and the depth sensor coordinate system corresponding to the depth sensor. Transformation relationship 3 corresponds to the fourth transformation relationship in this application. Based on the first transformation relationship, transformation relationship 2, and transformation relationship 3, the projection device can achieve accurate automatic correction and optimized projection effects under various conditions.

[0111] In one embodiment, in order to obtain the best target projection area, the user needs to interact multiple times to meet the specific operation requirements. For example, in order to realize the curtain-in function, the user often needs to start by pressing a specific button or selecting an option in the menu; similarly, the obstacle avoidance and automatic correction functions also require the user to operate through similar interaction methods. After obtaining these instructions, the traditional method also involves placing one or two full-screen picture cards and using these picture cards in combination with traditional image processing algorithms to calculate the projection area. However, this method requires the user to judge and perform a series of steps on their own, resulting in frequent interactions and long waiting times, which affects the user experience. Not only does the user need to spend extra time to set up, but the processing speed of the traditional image processing algorithm is also slow, further extending the preparation time and reducing overall user satisfaction.

[0112] To solve the above problems, the present application provides a projection device 200, wherein the projection device can be a long-focus projection device. The projection device 200 includes a depth sensor, an optical machine, and a controller. The depth sensor is configured to obtain an image of the projection surface. The optical machine is configured to project media resources. As Figure 7 shown, the controller is configured to:

[0113] S1: In response to a first instruction, obtain a first image of the projection surface through the depth sensor.

[0114] In one embodiment, the projection device 200 can provide a first key for the user, which includes a first key value, a second key value, and a third key value. The first key value can correspond to the curtain-in function, the second key value can correspond to the obstacle avoidance function, and the third key value can correspond to the no-sense function. That is, the first key is a multifunctional key. Illustratively, in response to the user's click operation on the first key, a first instruction is generated. The first instruction is an intelligent instruction that covers multiple functions.

[0115] Following the above example, after generating the first instruction, the projection device 200 controls the depth sensor to obtain a first image of the projection surface based on the first instruction.

[0116] As Figure 8 shown, Figure 8 some embodiments of the present application provide a schematic diagram of the first image.

[0117] The first image is an amplitude image, which is an image representing the reflection intensity or brightness information of each point in the scene, and is usually used to describe the light intensity distribution of the object surface. It can be understood as a traditional grayscale image or a color image, where each pixel value represents the light intensity of the corresponding position. Illustratively, the first light intensity is less than the second light intensity, that is, the area corresponding to the first light intensity has low brightness, and the area corresponding to the second light intensity has high brightness.

[0118] The amplitude image can be acquired by a depth sensor. The depth sensor is a device capable of capturing three-dimensional information of an object or a scene, which not only provides an amplitude image of brightness or color information, but also measures the distance (i.e., depth information) from the sensor to each point in the scene. Exemplarily, the depth sensor can be a Time of Flight (TOF) sensor. The projection surface refers to a plane in three-dimensional space for receiving a projected image or video, that is, the projection surface is a plane in a physical coordinate system, which is used to ensure that the projection content can be accurately displayed thereon.

[0119] Exemplarily, in combination with Figure 8 As shown in the first image, the amplitude image is acquired by the depth sensor.

[0120] S2: taking the first image as the input of the pre-trained first neural network model, using the first neural network model to obtain the recognition result.

[0121] The first neural network model is used to identify whether there is a target object on the projection surface. The target object includes at least one of a curtain or an obstacle.

[0122] The pre-trained neural network model refers to a neural network model that has been trained on a large-scale data set before use. In order to ensure the generalization ability of the neural network model, a large amount of training data needs to be used to train the neural network model.

[0123] In an embodiment, the training manner of the first neural network model is exemplarily described.

[0124] First, a training set is acquired.

[0125] The training set includes at least one training image, a target classification label corresponding to the at least one training image, and a target image corresponding to each target classification label. Each label explicitly indicates the category to which the image belongs. For example, the first classification label is 0 and a pure black image, 0 represents that the training image includes a curtain; the second classification label is 1 and a pure black image, 1 represents that the training image has no curtain and no obstacle; the second classification label is 2 and a binary image, 2 represents that the training image has an obstacle. The binary image is also called a mask image. The mask image includes pixel coordinates of pixels, for example, the place with an obstacle has a pixel coordinate of 255, and other places have a pixel coordinate of 0.

[0126] Then, the at least one training image is taken as the input of the first neural network model, and the target classification label and the target image corresponding to each target classification label are taken as the output of the first neural network model, to train the first neural network model.

[0127] The at least one training image is input into the initial neural network model, and a predicted classification label and a predicted image are output. Further, a preset loss function is used to determine a loss value between the predicted classification label and a target classification label and a loss value between the predicted image and a target image. When the loss values are less than a preset loss value, it is indicated that the initial neural network model is trained, and the initial neural network model is the trained first neural network model.

[0128] After the first neural network model is trained, the first neural network model is used to obtain a recognition result.

[0129] Exemplarily, in combination with Figure 9 As shown in the figure, the first image is input into the first neural network model, and 0 and a black image or 1 and a black image or 2 and a mask image are output.

[0130] In order to improve the accuracy of recognition, the first image can be cropped to obtain a cropped first image.

[0131] Exemplarily, based on the fourth conversion relationship, the first optical-mechanical coordinates are converted into a depth sensor coordinate system to obtain first depth coordinates, and based on the first depth coordinates, the first image is cropped to obtain a cropped first image.

[0132] Exemplarily, the fourth conversion relationship can be determined by formula (1):

[0133] T uv =M t *(R P2w *R P2t ) -1 *(T P2w (2)*R P2w *M P -1 *P uv +T P2t -T P2w ) Formula (1)

[0134] Wherein, T uv is the corner point coordinates under the depth sensor, M t is the internal parameter matrix of the depth sensor, R P2t is the rotation matrix of the optical-mechanical coordinate system to the depth sensor coordinate system, T P2w is the translation vector of the optical-mechanical coordinate system to the physical coordinate system, R P2w is the rotation matrix of the optical-mechanical coordinate system to the physical coordinate system, R P2c is the rotation matrix of the optical-mechanical coordinate system to the camera coordinate system, M P is the internal parameter matrix of the optical-mechanical coordinate system, and P uvis the first optical-mechanical coordinate in the optical-mechanical coordinate system, T P2t is the translation vector from the optical-mechanical coordinate system to the depth sensor coordinate system. In order to facilitate the subsequent description, the internal parameter matrix is described by using the internal parameter.

[0135] Next, according to the above example, P uv = P uv 1 is converted to T uv = T uv 1

[0136] S3: determining an execution strategy for obtaining the target projection area based on the recognition result.

[0137] The target projection area is the area for displaying the projection image. For example, the target projection area is usually set as a rectangle with a fixed aspect ratio. This setting ensures the visual coordination and accuracy of the projection content. For example, the 16:9 inscribed rectangle.

[0138] In an embodiment, the target object includes at least one of a curtain or an obstacle. In the case where the recognition result includes a curtain, the execution strategy for obtaining the target projection area is determined to be a curtain entry strategy; in the case where the recognition result does not include a curtain and includes an obstacle, the execution strategy for obtaining the target projection area is determined to be an obstacle avoidance strategy; in the case where the recognition result does not include a curtain and does not include an obstacle, the execution strategy for obtaining the target projection area is determined to be a no-sense correction strategy.

[0139] That is, the recognition result may contain multiple types of information, which determines the diversity of the execution strategy. That is, different recognition results will correspond to different execution strategies. For example, in the case where the recognition result is 0, that is, the target object existing on the projection surface includes a curtain, in the case where the recognition result includes a curtain, the execution strategy for obtaining the target projection area is determined to be a curtain entry strategy, in the case where the recognition result is 1, that is, the target object existing on the projection surface does not include a curtain and an obstacle, in the case where the recognition result does not include a curtain and includes an obstacle, the execution strategy for obtaining the target projection area is determined to be an obstacle avoidance strategy, in the case where the recognition result is 2, that is, the target object existing on the projection surface does not include a curtain and an obstacle, in the case where the recognition result does not include a curtain and does not include an obstacle, the execution strategy for obtaining the target projection area is determined to be a no-sense correction strategy.

[0140] S4: determining the target projection area in the projection surface based on the execution strategy.

[0141] The target projection area refers to the best range for displaying the projection image.

[0142] In an embodiment, S4 includes steps S41-S43.

[0143] Step S41: determining the first target coordinate in the optical-mechanical coordinate system based on the execution strategy.

[0144] The first target coordinate is the corner point coordinate of the target projection area in the projection surface in the optical-mechanical coordinate system.

[0145] The following describes the determination of the first target coordinate in the optical-mechanical coordinate system under different execution strategies.

[0146] The following describes the determination of the first target coordinate in the optical-mechanical coordinate system under different execution strategies. Figure 10 The first target coordinate is determined based on the in-curtain strategy.

[0147] Figure 10 The first target coordinate is determined based on the in-curtain strategy.

[0148] In an example, as shown in FIG. 4, the execution strategy is the in-curtain strategy, and the first target coordinate in the optical-mechanical coordinate system is determined as follows. Figure 10

[0149] S411: acquiring a second image corresponding to the projection surface when the execution strategy is the in-curtain strategy.

[0150] For example, S411 includes S4110-S4114.

[0151] S4110: projecting the image of the positioning card in the preset area of the curtain object by the optical machine when the execution strategy is the in-curtain strategy.

[0152] A preset area is set in the curtain for projecting the positioning card. The size of the preset area is not limited, but must be large enough to accommodate all key points of the positioning card and ensure that the key points meet the minimum distance requirement. For example, the minimum distance is 1 pixel. For example, if the specified minimum distance is 1 pixel, the coordinate difference between any two corner points (whether horizontal or vertical) must be greater than 1 pixel. At the same time, the preset area must be completely within the range of the curtain, i.e., not exceeding the actual size of the curtain.

[0153] It should be noted that the minimum distance is not limited.

[0154] ​Alignment charts or calibration targets are specialized images or patterns used for calibrating and adjusting optical systems, projectors, cameras, and other equipment. They typically contain a series of precisely designed geometric shapes, lines, dots, or other features that help ensure that the equipment can accurately capture or project images, and can be used to measure and correct various types of distortion or error. Illustratively, at a first focal length, an image of an alignment chart is projected by the light engine within a pre-set region in the screen. The pre-set region may, for example, be a central region of the screen.

[0155] Illustratively, the alignment chart is projected in the screen on the projection surface, and the alignment chart is disposed within the pre-set region.

[0156] S4111: Control the camera to obtain a first captured image corresponding to the projection surface.

[0157] In combination Figure 11 As shown in (a), the first captured image includes a screen object 90 and an alignment chart object 91. The alignment chart object 91 is disposed within a pre-set region of the screen object 90.

[0158] S4112: Convert the first light engine coordinates to the camera coordinate system based on a second conversion relationship to obtain first camera coordinates.

[0159] The first light engine coordinates are the corner point coordinates of the boundary region projected by the light engine, that is, the first light engine coordinates are the corner point coordinates in the light engine coordinate system; and the first camera coordinates are the corner point coordinates of the first light engine coordinates in the camera coordinate system.

[0160] The second conversion relationship is an approximate affine relationship between the camera coordinate system and the light engine coordinate system.

[0161] The second conversion relationship is an approximate affine relationship between the camera coordinate system and the light engine coordinate system.

[0162] The projection device 200 can determine a first conversion relationship between the light engine and the projection surface based on first parameter information.

[0163] The first parameter information includes the intrinsic parameters of the light engine, the depth information of the depth sensor, and the extrinsic parameter matrix between the depth sensor and the light engine. Hereinafter, the extrinsic parameters are described. The intrinsic parameters Mp of the light engine, also known as the intrinsic parameter matrix of the light engine, refer to the parameters that describe the internal optical system of the light engine (such as the focal length of the light engine, the principal point coordinates, etc.), which are used to determine the mapping relationship from three-dimensional space to two-dimensional projection picture. The extrinsic parameters between the TOF sensor and the light engine include a rotation matrix R P2w and a translation vector T P2w , the rotation matrix R P2w and the translation vector T P2wis the relationship between the optical-mechanical coordinate system and the TOF sensor coordinate system.

[0164] Exemplarily, the first conversion relationship can be determined by using formula (2):

[0165] W xyz =T P2w (2)*R P2w *M P -1* P uv formula (2)

[0166] wherein, W xyz is the corner point coordinate T P2w of the physical coordinate system, is the translation vector of the optical-mechanical coordinate system to the physical coordinate system, M P is the intrinsic matrix of the optical-mechanical system, used to describe the internal parameters in the optical-mechanical coordinate system, P uv is the first optical-mechanical coordinate in the optical-mechanical coordinate system.

[0167] The projection device 200 determines the second conversion relationship between the optical-mechanical system and the camera based on the first conversion relationship and the second parameter information.

[0168] The second parameter information includes the intrinsic M c of the camera, the first extrinsic R P2c and the second extrinsic T P2c between the camera coordinate system and the optical-mechanical coordinate system. Wherein, the intrinsic M c of the camera refers to the parameters describing the internal optical system of the camera, used to convert the three-dimensional world coordinate into the two-dimensional image coordinate. The intrinsic M c of the camera usually includes the focal length, principal point coordinate, etc. of the camera. The first extrinsic R P2c between the camera coordinate system and the optical-mechanical coordinate system is a rotation matrix, used to describe the rotation relationship between the optical-mechanical coordinate system and the camera coordinate system. The first extrinsic T P2c between the camera coordinate system and the optical-mechanical coordinate system is a translation vector, used to describe the translation relationship between the origin of the optical-mechanical coordinate system and the origin of the camera coordinate system, representing the relative displacement between the two coordinate systems.

[0169] Exemplarily, the second conversion relationship can be determined by using formula (3):

[0170] C uv =M c *(R P2w *R P2c ) -1 *(T P2w (2)*R P2w *M P -1 *P uv +T P2c-T P2w ) Formula (3)

[0171] Among them, C uv M represents the corner coordinates in the camera coordinate system. c Let R be the intrinsic parameter matrix of the camera. P2w It is the rotation matrix from the optomechanical coordinate system to the physical coordinate system, R P2c It is the rotation matrix from the optical-mechanical coordinate system to the camera coordinate system, T P2w It is the translation vector from the optomechanical coordinate system to the physical coordinate system, T P2c M is the translation vector from the optical-mechanical coordinate system to the camera coordinate system. P It is the intrinsic parameter matrix of the optomechanic, used to describe the internal parameters in the optomechanical coordinate system, P uv It is the first optomechanical coordinate in the optomechanical coordinate system.

[0172] For example, the first optomechanical coordinate P uv 1 converted to C uv 1.

[0173] S4113: Based on the coordinates of the first camera, crop the first captured image to obtain the second image.

[0174] Combination Figure 11 As shown in (b), the second image includes a backdrop object 90 and a positioning card object 91, with the positioning card object 90 positioned within a preset area of ​​the backdrop object. The main body area of ​​the second image is larger than the main body area of ​​the first image.

[0175] S412: Obtain the coordinates of the first screen object and the coordinates of the first map object in the second image.

[0176] The first screen coordinates are the corner coordinates of the screen object in the camera coordinate system, and the first image card coordinates are the corner coordinates of the positioning image card object in the camera coordinate system.

[0177] In one implementation, the first image is used as input to a pre-trained third neural network model, and the third neural network model is used to obtain the first curtain coordinates of the curtain object and the first map coordinates of the positioning map object in the second image.

[0178] Optionally, the training method for the third neural network model can refer to the third neural network model in step S2 above, and will not be repeated here.

[0179] For ease of explanation, the coordinates of the first curtain and the first card are shown in two-dimensional coordinates, not homogeneous coordinates.

[0180] Continuing with the example above, the second image is input into a pre-trained third neural network model to obtain the first screen coordinates of the screen object and the first map coordinates corresponding to the positioning map object. For example, if the first screen coordinates are A1(x... 1’ y 1’ ), B1(x 2’ y 2’ C1(x) 3’ y 3’ ), D1(x 4’ y 4’ The first graph coordinates are E1(x). 80’ y 80’ ), F1(x 81’ y 82’ ), G1(x 83’ y 83’ H1(x) 84’ y 84’ ), I1(x 85’ y 85’ J1(x) 86’ y 86’ ), K1(x 87’ y 87’ L1(x) 88’ y 88’ ).

[0181] To improve the accuracy of the first coordinate of the screen object and the second coordinate of the positioning card object, the acquisition of the first coordinate of the screen object and the second coordinate of the positioning card object can be optimized, as shown in S413.

[0182] S413: Perform an update operation on the coordinates of the first curtain and the coordinates of the first card to determine the updated coordinates of the first curtain and the corresponding first weight, as well as the updated coordinates of the first card and the corresponding second weight.

[0183] Wherein, the updated first screen coordinates are the screen coordinates obtained after the update operation in the second image coordinate system; the updated first screen coordinates are the card coordinates obtained after the update operation in the second image coordinate system, and the update operation includes at least one of movement, scaling, and rotation.

[0184] For example, combined Figure 12 As shown, S413 includes S4131-S4133.

[0185] S4131: Based on the coordinates of the first screen and the coordinates of the first image card, crop the second image to obtain at least one cropped image corresponding to the coordinates of the first screen and at least one second cropped image corresponding to the coordinates of the first image card.

[0186] Combination Figure 13An example is described.

[0187] Figure 13 A flowchart for cropping the second image is provided for some embodiments of the present application.

[0188] As Figure 13 shown, each coordinate point of the first curtain coordinates and the first card coordinates is taken as a center point, and cropping is performed based on the center point and the offset to obtain at least one first cropped image and a second cropped image.

[0189] For example, a 96*96 image is cropped on the second image with the center point as the center to obtain at least one first cropped image and a second cropped image.

[0190] The ratio of the cropped image can be 96*96 as described above or other ratios, which are not specifically limited here.

[0191] Following the above example, the first curtain coordinates and the first card coordinates in the second image are cropped. Taking one of the second images as an example, the first curtain coordinates are A1(x 1’ , y 1’ ), B(x 2’ , y 2’ ), C1(x 3’ , y 3’ ), and D1(x 4’ , y 4’ ), and A1(x 1’ , y 1’ ) is taken as the center point, and cropping is performed with (x 1’ -48, y 1’ -48), (x 1’ +48, y 1’ +48) as the vertices to obtain four first cropped images corresponding to the four corner points of the curtain object. Similarly, taking the first card coordinates as an example, eight second cropped images corresponding to the eight corner points of the positioning card object can be obtained.

[0192] S4132: Taking at least one first cropped image and at least one second cropped image as input of a second neural network model, and using the second neural network model to obtain second curtain coordinates corresponding to the first cropped image and a corresponding first weight, and second card coordinates of the second cropped image and a corresponding second weight.

[0193] The second curtain coordinates are the curtain coordinates corresponding to the first cropped image, and the second card coordinates are the card coordinates corresponding to the second cropped image; the second neural network model is used to obtain the corner point coordinates in the image and the weight ratio of each corner point coordinate.

[0194] The training method of the second neural network model can refer to step S2 described above, which will not be repeated here.

[0195] wherein the weight is used to reflect the accuracy of the current coordinate point.

[0196] Then, according to the above example, the four first cropped images corresponding to the four corner points and the eight second cropped images corresponding to the eight corner points are input into the second neural network model to obtain the second curtain coordinates A1(x 1” , y 1” ), B1(x 2” , y 2” ), C1(x 3” , y 3” ), D1(x 4” , y 4” ), and the first weight a1 corresponding to A1, the first weight b1 corresponding to B1, the first weight c1 corresponding to C1, and the first weight d1 corresponding to D1, and similarly, the second card coordinates E1(x 80” , y 80” ), F1(x 81” , y 82” ), G1(x 83” , y 83” ), H1(x 84” , y 84” ), I1(x 85” , y 85” ), J1(x 86” , y 86” ), K1(x 87” , y 87” ), L1(x 88” , y 88” ) are obtained, and the second weight e1 corresponding to E1, the second weight f1 corresponding to F1, the second weight g1 corresponding to G1, the second weight h1 corresponding to H1, the second weight i1 corresponding to I1, the second weight j1 corresponding to J1, the second weight k1 corresponding to K1, and the second weight l1 corresponding to L1.

[0197] S4133: converting the second curtain coordinates and the second card coordinates into the coordinate system of the second image to obtain the updated first curtain coordinates and the updated first curtain coordinates.

[0198] The second curtain coordinates and the second card coordinates are coordinates in the camera coordinate system, and the coordinates of the same point in the cropped image are not the same as the coordinates in the first image, so coordinate conversion is needed to obtain the coordinates corresponding to the first image. That is, the second curtain coordinates and the second card coordinates are the coordinates corresponding to the 96*96 image coordinate system after cropping.

[0199] Then, according to the above example, the second curtain coordinates A1(x1” 1” 2” 2” 3” 3” 4” 4” 80” 80” 81” 82” 83” 83” 84” 84” 85” 85” 86” 86” 87” 87” 88” 88” 1”’ 1”’ 2”’ 2”’ 3”’ 3”’ 4”’ 4”’ 80”’ 80”’ 81”’ 82”’ 83”’ 83”’ 84”’ 84”’ 85”’ 85”’ 86”’ 86”’ 87”’ 87”’ 88”’ 88”’

[0200] In order to improve the accuracy of the updated first curtain coordinates and the updated first picture card coordinates again, the steps S412-S413 are repeated. The number of repetitions is not specifically limited.

[0201] ​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​Exemplarily, the number of repetitions can be set to 5 times. Then the above steps S411-S413 can be repeatedly executed, that is, 6 first shooting images can be acquired, 6 second images can be obtained, and then 6 first curtain coordinates and first graph coordinates can be obtained. After cropping, 24 first cropped images and 48 second cropped images are obtained. The 24 first cropped images and 48 second cropped images cropped for 6 times are input into the second neural network model to obtain 6 updated first curtain coordinates and first weights corresponding to each corner point in each first curtain coordinate, and 6 updated first graph coordinates and second weights corresponding to each corner point in each first graph coordinate.

[0202] It should be noted that the number of repetitions is not specifically limited.

[0203] S414: determining a first target coordinate corresponding to a target projection area in an optical-mechanical coordinate system based on the updated first curtain coordinate, the first weight, the updated first graph coordinate, and the second weight.

[0204] Exemplarily, continuing to combine Figure 12 As shown in the figure, step S414 includes S4141-S4144.

[0205] S4141: determining a third curtain coordinate of the second image based on the first weight.

[0206] The third curtain coordinate is a corner point coordinate of the curtain object in the first image in the camera coordinate system. The third curtain coordinate is the curtain coordinate corresponding to the highest weight value in the updated first curtain coordinate in the 6 second images.

[0207] Next, according to the above example, for the convenience of description, taking one corner point coordinate in the first curtain coordinate in the 6 second images as an example, for example, A1(x 1”’ , y 1”’ ) in the first second image, the corresponding first weight is a1; A2(x 1”’ , y 1”’ ) in the second second image, the corresponding first weight is a2; A3(x 1”’ , y 1”’ ) in the third second image, the corresponding first weight is a3; A4(x 1”’ , y 1”’ ) in the fourth second image, the corresponding first weight is a4; A5(x 1”’ , y 1”’ ) in the fifth second image, the corresponding first weight is a5; and A6(x 1”’ , y 1”’), and the corresponding first weight is a6. If a1=0.3, a2=0.5, a3=0.3, a4=0.7, a5=0.7, and a6=0.6, the first curtain coordinate corresponding to the fourth second image is determined as the final first curtain coordinate.

[0208] S4142: determining the third card coordinate of the second image based on the second weight.

[0209] The third card coordinate is the card coordinate with the highest weight value in the second weight. That is, the third curtain coordinate is the card coordinate with the highest weight value in the updated first card coordinate of the six second images.

[0210] The specific content of S4142 can refer to the above S4141, which will not be repeated here.

[0211] S4143: determining a third conversion relationship based on the third card coordinate and the preset card coordinate in the light machine coordinate system.

[0212] The third conversion relationship is the accurate mapping relationship between the camera coordinate system and the light machine coordinate system.

[0213] The corresponding relationship established based on the third card coordinate and the preset card coordinate in the light machine is used to represent the accurate mapping relationship between the camera and the light machine. The mapping relationship can be referred to as a homography matrix H2.

[0214] Exemplarily, the third conversion relationship can be determined based on formula (3):

[0215] P uv = H2*C uv Formula (3)

[0216] Wherein, P uv is the coordinate in the light machine coordinate system; C uv is the coordinate in the camera coordinate system.

[0217] S4144: converting the third curtain coordinate based on the third conversion relationship to obtain the first target coordinate corresponding to the target projection area in the light machine coordinate system.

[0218] Based on the above example, the first target coordinate corresponding to the target projection area in the light machine coordinate system is determined based on formula (3).

[0219] The following will be combined Figure 14 to illustrate the first target coordinate obtained by the obstacle avoidance strategy.

[0220] Figure 14 A second flowchart for obtaining the first target coordinate provided by some embodiments of the present application is shown.

[0221] In another example, as shown in Figure 14 Step S41: based on the execution strategy, determine the specific execution strategy of the first target coordinate in the optical-mechanical coordinate system as an obstacle avoidance strategy, that is, in the case where the execution strategy is an obstacle avoidance strategy, obtaining the first target coordinate in the optical-mechanical coordinate system includes the following steps:

[0222] S415: In the case where the execution strategy is an obstacle avoidance strategy, obtain the binary image corresponding to the first image and the first pixel coordinate corresponding to the binary image.

[0223] The first pixel coordinate includes pixel coordinates corresponding to obstacles and pixel coordinates corresponding to non-obstacles.

[0224] Exemplarily, as shown in Figure 15 the binary image, the place with obstacles in the binary image is pixel coordinates 255, and other places are pixel coordinates 0. S416: based on the fourth conversion relationship, convert the first optical-mechanical coordinate to obtain the first depth coordinate in the depth sensor coordinate system.

[0225] Wherein, the fourth conversion relationship is the mapping relationship between the optical-mechanical coordinate system and the depth sensor coordinate system.

[0226] Exemplarily, the fourth conversion relationship can be determined by formula (4):

[0227] T uv =M t *(R P2w *R P2t ) -1 *(T P2w (2)*R P2w *M P -1 *P uv +T P2t -T P2w ) Formula (4)

[0228] Wherein, T uz is the corner point coordinate under the depth sensor, M t is the intrinsic parameter of the depth sensor, R P2t is the rotation matrix of the optical-mechanical coordinate system to the depth sensor coordinate system T P2w is the translation vector of the optical-mechanical coordinate system to the physical coordinate system, R P2w is the rotation matrix of the optical-mechanical coordinate system to the physical coordinate system, R P2c is the rotation matrix of the optical-mechanical coordinate system to the camera coordinate system, M P is the intrinsic matrix of the optical-mechanical system, which is used to describe the internal parameters in the optical-mechanical coordinate system, P uv is the first optical-mechanical coordinate in the optical-mechanical coordinate system, T P2ta translation vector of the optical-mechanical coordinate system to the depth sensor coordinate system.

[0229] Then, according to the above example, P uv 1 is converted into T uv 1.

[0230] S417: determining a third target coordinate in the depth sensor coordinate system based on the first pixel coordinate and the first depth coordinate.

[0231] The third target coordinate is a corner point coordinate of the target projection area in the projection surface in the depth sensor coordinate system.

[0232] S417 can determine the third target coordinate by the water-filling method.

[0233] In an example, the projection device 200 determines a fourth target coordinate in the depth sensor coordinate system based on the first pixel coordinate, the first depth coordinate, and a first speed; after determining the fourth target coordinate in the depth sensor coordinate system, determines a third target coordinate in the depth sensor coordinate system based on the first pixel coordinate, the first depth coordinate, and a second speed; wherein the first speed is greater than the second speed.

[0234] The fourth target coordinate is an intermediate coordinate of the corner point of the projection area to be determined.

[0235] In combination with (a) in FIG. Figure 16 In the range of the first depth coordinate, a smaller 16:9 rectangle is placed, and the rectangle is pulled in the arrow direction, while maintaining the shape of 16:9, and the pulling step is a distributed pulling manner, such as first pulling at a first speed, and then pulling at a second speed, thereby reducing the number of pulling times, until the four corners of the 16:9 rectangle meet the first condition, and then the size of the 16:9 rectangle is further determined, thereby determining the fifth target coordinate. The first condition is that the corner point coordinates corresponding to the 16:9 rectangle do not exceed the range of the first depth coordinate, and do not overlap with the obstacle part.

[0236] The third target coordinate obtained above may not be the best projection area, so it needs to be judged.

[0237] For example, if the rectangular area corresponding to the third target coordinate does not meet the second condition, that is, the second condition is that the 16:9 rectangle can move without exceeding the range of the first depth coordinate and without overlapping with the obstacle part. In the case where the preset condition is not met, the third target coordinate is updated based on the offset of the movement to obtain the final third target coordinate.

[0238] For example, in combination with (b) in FIG. Figure 16 Figure 16 ​(b) in the (b) in the is the region corresponding to the final third target coordinate.

[0239] Exemplarily, the third target coordinate is determined as T uv 1.

[0240] S418: converting the third target coordinate based on a fourth conversion relationship to obtain a first target coordinate in the optical-mechanical coordinate system.

[0241] Exemplarily, the third target coordinate T uv 1 is converted to P uv 1.

[0242] The first target coordinate obtained by the non-inductive correction strategy is exemplarily described below.

[0243] In another example, step S41: based on the execution strategy, the specific execution strategy of the first target coordinate in the optical-mechanical coordinate system is determined to be the non-inductive correction strategy, that is, in the case where the execution strategy is the non-inductive correction strategy, obtaining the first target coordinate in the optical-mechanical coordinate system includes the following steps:

[0244] S419: in the case where the execution strategy is the non-inductive correction strategy, the first optical-mechanical coordinate is taken as the first target coordinate in the optical-mechanical coordinate system.

[0245] The first optical-mechanical coordinate is the corner point coordinate corresponding to the boundary region projected by the optical machine.

[0246] Step S42: converting the first target coordinate based on a first conversion relationship to obtain a second target coordinate in the physical coordinate system.

[0247] The first conversion relationship is the mapping relationship between the optical-mechanical coordinate system and the physical coordinate system.

[0248] Exemplarily, P uv 1 is converted to obtain the second target coordinate in the physical coordinate system, that is, W xyz 1.

[0249] Step S43: determining a target projection region in the projection surface based on the second target coordinate.

[0250] The target projection region is the range region of the second target coordinate.

[0251] The above technical solution has the following beneficial effects or advantages: by responding to the first instruction, the image of the projection surface is acquired by using the depth sensor, and the image is input into the pre-trained first neural network model for analysis. The model can identify the recognition result on the projection surface, and determine different execution strategies based on the recognition result to define the target projection area. In this way, not only the automation degree of the projection setting is improved, and the interaction steps with the user are reduced, but also the user's waiting time is significantly reduced by using the model processing, the processing efficiency of acquiring the target projection area is improved, and the user experience is improved.

[0252] The following will be described in detail Figure 17 The method for acquiring the target projection area will be described in detail.

[0253] Figure 17 A second method flowchart for acquiring the target projection area is provided for some embodiments of the present application.

[0254] The projection device 200 acquires the first image by using the depth sensor;

[0255] The projection device 200 inputs the first image into the first neural network model to obtain the recognition result;

[0256] The projection device 200 determines whether the curtain is included based on the recognition result;

[0257] The projection device 200 punches the positioning card in the preset area of the curtain on the projection surface in the case of including the curtain;

[0258] The projection device 200 photographs the projection surface to obtain the first photographed image;

[0259] The projection device 200 crops the first photographed image to obtain the second image;

[0260] The projection device 200 inputs the second image into the third neural network model to obtain the four corner coordinates of the curtain and the eight corner coordinates of the card (the first curtain coordinates and the second curtain coordinates);

[0261] The projection device 200 crops based on the four corner coordinates of the curtain and the eight corner coordinates of the card to obtain the first cropped image and the second cropped image;

[0262] The projection device 200 inputs the first cropped image and the second cropped image into the second neural network model to obtain the updated first curtain coordinates and the first card coordinates;

[0263] The projection device 200 determines the homography relationship H2 according to the preset card coordinates and the first card coordinates;

[0264] The projection device 200 converts the first curtain image to the light engine coordinate system based on the homography to obtain first target coordinates;

[0265] The projection device 200 obtains a target projection area based on the first target coordinates;

[0266] If the curtain is not included, the projection device 200 determines whether there is an obstacle;

[0267] If there is no obstacle, the projection device 200 takes the first light engine coordinates as the first target coordinates;

[0268] The projection device 200 obtains a target projection area based on the first target coordinates;

[0269] If there is an obstacle, the projection device 200 obtains the pixels corresponding to the obstacle in the binary image;

[0270] The projection device 200 selects the largest 16:9 rectangle in the binary image;

[0271] It should be noted that the 16:9 rectangle can also be selected in the physical coordinate system, which is not limited here.

[0272] The projection device 200 converts the corner point coordinates corresponding to the 16:9 rectangle to the light engine coordinate system to obtain the first target coordinates;

[0273] The projection device 200 obtains a target projection area based on the first target coordinates.

[0274] Based on the above projection device 200, some embodiments of the present application further provide a method for determining a projection area, comprising:

[0275] In response to a first instruction, a first image of the projection surface is obtained by the depth sensor;

[0276] The first image is taken as the input of a pre-trained first neural network model, and the first neural network model is used to obtain a recognition result; the first neural network model is used to identify whether there is a target object on the projection surface;

[0277] Based on the recognition result, an execution strategy for obtaining a target projection area is determined, the target projection area being a region for displaying a projection image; the execution strategy has a corresponding relationship with the recognition result;

[0278] Based on the execution strategy, the target projection area in the projection surface is obtained.

[0279] The above technical solution has the following beneficial effects or advantages: by responding to the first instruction, the image of the projection surface is acquired by using the depth sensor, and the image is input into the pre-trained first neural network model for analysis. The model can identify the recognition result on the projection surface, and determine different execution strategies based on the recognition result to define the target projection area. In this way, not only the automation degree of the projection setting is improved, and the interaction steps with the user are reduced, but also the waiting time of the user is significantly reduced by using the model processing, the processing efficiency of obtaining the target projection area is improved, and the user experience is improved.

[0280] The same or similar parts among various embodiments in the specification can be referred to each other, and will not be described here again.

[0281] Those skilled in the art can clearly understand that the technology in the embodiments of the present application can be realized by means of software and necessary general hardware platform. Based on such understanding, the technical solutions in the embodiments of the present application can be embodied in the form of software product, which can be stored in storage medium such as ROM / RAM, magnetic disk, optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, server, or network device, etc.) execute the method of various embodiments or some parts of the embodiments of the present application.

[0282] Finally, it should be pointed out that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application. In order to facilitate explanation, the above description has been made in combination with specific embodiments. However, the above exemplary discussion is not intended to exhaust or limit the embodiments to the specific forms disclosed above. According to the above teaching, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different modified embodiments suitable for specific use considerations.

Claims

1. A projection device, characterized in that, include: A depth sensor is configured to acquire an image of the projection surface; The optical engine is configured to project media assets. The controller is configured as follows: In response to a first command, a first image of the projection surface is acquired using the depth sensor; Using the first image as input to a pre-trained first neural network model, a recognition result is obtained using the first neural network model; the first neural network model is used to identify whether a target object exists on the projection surface. The target object includes at least one of a curtain and an obstacle; Based on the recognition results, an execution strategy for obtaining the target projection area is determined, wherein the target projection area is the area where the projected image is displayed; The execution strategy and the identification result have a corresponding relationship; Based on the execution strategy, the target projection area in the projection plane is obtained; The controller executes an execution strategy for acquiring the target projection region based on the recognition result, specifically configured as follows: If the recognition result includes a screen, the execution strategy for obtaining the target projection area is determined to be an entry screen strategy; If the recognition result does not include the curtain but includes obstacles, the execution strategy for obtaining the target projection area is determined to be an obstacle avoidance strategy. If the recognition result does not include the screen or the obstacle, the execution strategy for obtaining the target projection area is determined to be a non-sensory correction strategy.

2. The projection device according to claim 1, characterized in that, The controller executes the execution strategy to obtain the target projection area in the projection plane, specifically configured as follows: Based on the execution strategy, the coordinates of a first target in the optomechanical coordinate system are determined; the first target coordinates are the corner coordinates of the target projection area in the projection plane in the optomechanical coordinate system. Based on the first transformation relationship, the coordinates of the first target are transformed to obtain the coordinates of the second target in the physical coordinate system; the first transformation relationship is the mapping relationship between the optomechanical coordinate system and the physical coordinate system; Based on the second target coordinates, the target projection area in the projection plane is obtained.

3. The projection device according to claim 2, characterized in that, The controller executes the execution strategy to determine the coordinates of the first target in the optomechanical coordinate system, specifically configured as follows: When the execution strategy is the screen entry strategy, a second image corresponding to the projection surface is obtained; the second image includes a screen object and a positioning card object, and the positioning card object is set in a preset area of ​​the screen object; Obtain the first screen coordinates corresponding to the screen object in the second image and the first map coordinates corresponding to the positioning map object; the first screen coordinates are the corner coordinates corresponding to the screen object in the camera coordinate system, and the first map coordinates are the corner coordinates corresponding to the positioning map object in the camera coordinate system. An update operation is performed on the first screen coordinates and the first image coordinates to determine the updated first screen coordinates and their corresponding first weights, as well as the updated first image coordinates and their corresponding second weights; wherein, the updated first screen coordinates are the screen coordinates obtained after the update operation is performed in the second image coordinate system; the updated first image coordinates are the image coordinates obtained after the update operation is performed in the second image coordinate system, and the update operation includes at least one of movement, scaling, and rotation; Based on the updated first screen coordinates, the first weight, the updated first map coordinates, and the second weight, the first target coordinates corresponding to the target projection area in the optical-mechanical coordinate system are determined.

4. The projection device according to claim 3, characterized in that, Also includes: The camera is configured to capture an image of the projection surface; the controller is specifically configured to acquire a second image corresponding to the projection surface. The camera is controlled to acquire a first captured image; the first captured image includes the backdrop object and the positioning card object. Based on the second transformation relationship, the first optical engine coordinates are transformed to the camera coordinate system to obtain the first camera coordinates; wherein, the first optical engine coordinates are the corner coordinates corresponding to the boundary area projected by the optical engine; the second transformation relationship is the mapping relationship between the camera coordinate system and the optical engine coordinate system; Based on the first camera coordinates, the first captured image is cropped to obtain the second image; the main body area of ​​the second image is larger than the main body area of ​​the first captured image.

5. The projection device according to claim 3, characterized in that, The controller executes an update of the first screen coordinates and its corresponding first weight, and an update of the first image coordinates and its corresponding second weight, based on the first screen coordinates and the first image coordinates. Specifically, this is configured as follows: Based on the first screen coordinates and the first image coordinates, the second image is cropped to obtain at least one first cropped image corresponding to the first screen coordinates and at least one second cropped image corresponding to the first image coordinates. The at least one first cropped image and the at least one second cropped image are used as inputs to the second neural network model. Using the second neural network model, the second screen coordinates and the corresponding first weights, as well as the second card coordinates and the corresponding second weights, are obtained. The second screen coordinates are the screen coordinates corresponding to the first cropped image, and the second card coordinates are the card coordinates corresponding to the second cropped image. The second screen coordinates and the second image coordinates are transformed into the coordinate system of the second image to obtain the updated first screen coordinates and the updated first screen coordinates.

6. The projection device according to any one of claims 3-5, characterized in that, The controller performs the following operation: determining the first target coordinates corresponding to the target projection area in the optical-mechanical coordinate system based on the updated first screen coordinates, the first weight, the updated first map coordinates, and the second weight. Specifically, this is configured as follows: Based on the first weight, the third screen coordinates of the second image are determined; the third screen coordinates are the screen coordinates with the highest weight value among the first weights. Based on the second weight, the third map coordinates of the second image are determined; the third map coordinates are the map coordinates with the highest weight value in the second weight; based on the third map coordinates and the preset map coordinates in the optical-mechanical coordinate system, a third transformation relationship is determined; the third transformation relationship is the mapping relationship between the camera coordinate system and the optical-mechanical coordinate system. Based on the third transformation relationship, the coordinates of the third screen are transformed to obtain the first target coordinates corresponding to the target projection area in the optomechanical coordinate system.

7. The projection device according to claim 2, characterized in that, The controller executes the execution strategy to determine the coordinates of the first target in the optomechanical coordinate system, specifically configured as follows: When the execution strategy is the obstacle avoidance strategy, the binary image corresponding to the first image and the first pixel coordinates corresponding to the binary image are obtained; the first pixel coordinates include the pixel coordinates corresponding to the obstacle and the pixel coordinates corresponding to non-obstacles; Based on the fourth transformation relationship, the first optomechanical coordinates are transformed to obtain the first depth coordinates in the depth sensor coordinate system; the fourth transformation relationship is the mapping relationship between the optomechanical coordinate system and the depth sensor coordinate system. Based on the first pixel coordinates and the first depth coordinates, the third target coordinates in the depth sensor coordinate system are determined. The third target coordinates are the corner coordinates of the target projection area in the projection plane in the depth sensor coordinate system. Based on the fourth transformation relationship, the third target coordinates are transformed to obtain the first target coordinates in the optomechanical coordinate system.

8. The projection device according to claim 2, characterized in that, The controller executes the execution strategy to determine the coordinates of the first target in the optomechanical coordinate system, specifically configured as follows: When the execution strategy is the non-sensory correction strategy, the first optomechanical coordinates are used as the first target coordinates in the optomechanical coordinate system; wherein, the first optomechanical coordinates are the corner coordinates corresponding to the boundary region projected by the optomechanical system.

9. A method for determining a projection area, characterized in that, include: In response to the first command, a first image of the projection surface is acquired using a depth sensor; Using the first image as input to a pre-trained first neural network model, a recognition result is obtained using the first neural network model; the first neural network model is used to identify whether a target object exists on the projection surface. The target object includes at least one of a curtain and an obstacle; Based on the recognition results, an execution strategy for obtaining the target projection area is determined, wherein the target projection area is the area where the projected image is displayed; The execution strategy and the identification result have a corresponding relationship; Based on the execution strategy, the target projection area in the projection plane is obtained; Based on the recognition results, the execution strategy for obtaining the target projection region is determined, and the specific configuration is as follows: If the recognition result includes a screen, the execution strategy for obtaining the target projection area is determined to be an entry screen strategy; If the recognition result does not include the curtain but includes obstacles, the execution strategy for obtaining the target projection area is determined to be an obstacle avoidance strategy. If the recognition result does not include the screen or the obstacle, the execution strategy for obtaining the target projection area is determined to be a non-sensory correction strategy.

Images