Projection equipment and projection area determination method

Abstract

The embodiment of the invention discloses projection equipment and a projection area determination method, and the method comprises a depth sensor which is configured to be used for obtaining an image of a projection surface; the optical machine is configured to be used for projecting the media asset resources; a controller configured to: acquire a first image of the projection surface through the depth sensor in response to a first instruction; taking the first image as the input of a pre-trained first neural network model, and obtaining an identification result by using the first neural network model; the first neural network model is used for identifying whether a target object exists on the projection surface; based on the recognition result, determining an execution strategy for acquiring a target projection area, the target projection area being an area for displaying the projection image; and obtaining a target projection area in the projection plane based on the execution strategy. According to the method, interaction with the projection equipment can be reduced, the projection mode of the optimal projection area can be automatically judged and obtained, meanwhile, the efficiency of obtaining the optimal projection area can be improved by adopting the neural network model, and the user experience is improved.

Description

Technical Field

[0001] The present application relates to the technical field of projection equipment, and in particular to a projection equipment and a method for determining a projection area. Background Art

[0002] As a key tool of modern visual display technology, projection equipment has been widely used in many fields such as education, business, entertainment and home theater. In these applications, it is crucial to ensure that the content is presented to the audience clearly and completely. In order to provide the best viewing experience, accurately determining and setting the projection area has become one of the basic steps to achieve high-quality projection effects. The projection area here refers to the specific display range of the projected image. It defines the effective area on the screen that is actually used to display content, which plays an important role in ensuring image quality, adapting to different scene requirements and improving the audience experience. By accurately setting the projection area, it can be ensured that every detail is presented in the most ideal way, thus providing users with a more immersive visual enjoyment.

[0003] Currently, projection devices usually require multiple interactions with users to meet specific operational requirements when determining the projection area. For example, in order to achieve the entry function, users often need to start by pressing a specific button or selecting an option in a menu; similarly, functions such as obstacle avoidance and automatic correction also require users to operate through similar interactions. After obtaining these instructions, the traditional method also involves placing one or two full-screen image cards and using these image cards in combination with traditional image processing algorithms to calculate the projection area.

[0004] However, this approach requires users to make their own judgments and perform a series of steps, resulting in frequent interactions and long waiting times, which affects the user experience. Not only do users need to spend extra time waiting, but the slow processing speed of traditional image processing algorithms further prolongs the preparation time and reduces overall user satisfaction. Summary of the invention

[0005] Some embodiments of the present application provide a projection device and a method for determining a projection area, which can reduce the interaction with the projection device and independently determine the projection method for obtaining the best projection area. At the same time, the use of a neural network model can improve the efficiency of obtaining the best projection area and improve the user experience.

[0006] In a first aspect, an embodiment of the present application provides a projection device, including:

[0007] a depth sensor configured to acquire an image of the projection surface;

[0008] An optical machine is configured to project media resources;

[0009] The controller is configured as:

[0010] In response to a first instruction, acquiring a first image of the projection surface through the depth sensor;

[0011] The first image is used as an input of a pre-trained first neural network model, and a recognition result is obtained using the first neural network model; the first neural network model is used to identify whether there is a target object on the projection surface;

[0012] Based on the recognition result, determining an execution strategy for acquiring a target projection area, the target projection area being an area where a projection image is displayed; the execution strategy having a corresponding relationship with the recognition result;

[0013] Based on the execution strategy, the target projection area in the projection surface is acquired.

[0014] The above technical solution has the following beneficial effects or advantages: by responding to the first instruction, the image of the projection surface is obtained by using the depth sensor, and the image is input into the pre-trained first neural network model for analysis. The model can identify the recognition result on the projection surface, and determine different execution strategies based on the recognition result to define the target projection area. In this way, not only the automation of the projection setting is improved, and the interaction steps with the user are reduced, but also the user's waiting time is significantly reduced by using the model processing, the processing efficiency of obtaining the target projection area is improved, and the user experience is improved.

[0015] In some embodiments, the target object includes at least one of a curtain or an obstacle; the controller determines an execution strategy for acquiring a target projection area based on the recognition result, and the specific configuration is:

[0016] In the case where the recognition result includes a curtain, determining that the execution strategy for acquiring the target projection area is an entry curtain strategy;

[0017] When the recognition result includes the curtain but does not include an obstacle, determining that the execution strategy for acquiring the target projection area is an obstacle avoidance strategy;

[0018] In a case where the recognition result includes the curtain and the obstacle, the execution strategy for acquiring the target projection area is determined to be a non-sensing correction strategy.

[0019] The above technical solution has the following beneficial effects or advantages: different recognition results can be used to implement different execution strategies, thereby reducing the number of interactions between the user and the projection device and improving the user experience.

[0020] In some embodiments, the controller executes based on the execution strategy to obtain the target projection area in the projection surface, and the specific configuration is:

[0021] Based on the execution strategy, determining first target coordinates in an optical-mechanical coordinate system; the first target coordinates are the coordinates of a corner point of the target projection area in the projection plane in the optical-mechanical coordinate system;

[0022] Based on a first conversion relationship, the first target coordinates are converted to obtain second target coordinates of a physical coordinate system; the first conversion relationship is a mapping relationship between the optical-mechanical coordinate system and the physical coordinate system;

[0023] Based on the second target coordinates, the target projection area in the projection plane is acquired.

[0024] The above technical solution has the following beneficial effects or advantages: based on different execution strategies, the first target coordinates in the optical-mechanical coordinate system can be determined, and then, the first target coordinates can be converted into the second target coordinates in the physical coordinate system using the first conversion relationship between the optical-mechanical coordinate system and the physical coordinate system. Finally, based on the second target coordinates, the target projection area in the projection surface can be accurately determined. In this way, the accuracy of obtaining the target projection area can be improved through the coordinates and conversion relationship, which facilitates the determination of the best projection area for displaying the projected image and improves the user experience.

[0025] In some embodiments, the controller executes based on the execution strategy to determine the first target coordinates in the optical-mechanical coordinate system, and the specific configuration is:

[0026] When the execution strategy is the entry strategy, a second image corresponding to the projection surface is acquired; the second image includes a curtain object and a positioning card object, and the positioning card object is set in a preset area of ​​the curtain object;

[0027] Acquire the first curtain coordinates corresponding to the curtain object and the first card coordinates corresponding to the positioning card object in the second image; the first curtain coordinates are the corner coordinates corresponding to the curtain object in the camera coordinate system, and the first card coordinates are the corner coordinates corresponding to the positioning card object in the camera coordinate system;

[0028] An update operation is performed on the first screen coordinates and the first image card coordinates to determine the updated first screen coordinates and the corresponding first weight, and the updated first image card coordinates and the corresponding second weight; wherein the updated first screen coordinates are screen coordinates obtained after the update operation is performed in the second image coordinate system; the updated first screen coordinates are image card coordinates obtained after the update operation is performed in the second image coordinate system, and the update operation includes at least one of moving, scaling, and rotating;

[0029] Based on the updated first curtain coordinates, the first weight, the updated first image card coordinates, and the second weight, the first target coordinates corresponding to the target projection area in the optical machine coordinate system are determined.

[0030] The above technical solution has the following beneficial effects or advantages: in the case of entering the screen, by placing a positioning card in the preset area on the screen, the coordinates of the corner points of the screen and the positioning card can be obtained without affecting the user's viewing, and the first target coordinates in the optical machine coordinate system can be determined more accurately, thereby significantly improving the accuracy of the subsequent target projection area determination. In this way, not only the precise alignment and display of the projected content is ensured, but also the user's viewing experience is kept undisturbed, thereby improving the user experience.

[0031] In some embodiments, the method further includes: a camera configured to capture the image of the projection surface; and the controller executes the acquisition of the second image corresponding to the projection surface, which is specifically configured as follows:

[0032] Controlling the camera to obtain a first captured image; the first captured image includes the curtain object and the positioning card object;

[0033] Based on the second conversion relationship, the first optical-mechanical coordinates are converted into the camera coordinate system to obtain the first camera coordinates; wherein the first optical-mechanical coordinates are the corner point coordinates corresponding to the boundary area projected by the optical machine; and the second conversion relationship is the mapping relationship between the camera coordinate system and the optical-mechanical coordinate system;

[0034] Based on the first camera coordinates, the first captured image is cropped to obtain the second image; a main body area of ​​the second image is larger than a main body area of ​​the first captured image.

[0035] The above technical solution has the following beneficial effects or advantages: after the second image including the screen and the picture card is captured by the camera, since the image may contain other non-projection areas, the coordinates corresponding to the optimal projection area can be determined by using the coordinate and conversion relationship. Image cropping based on these precise coordinates can effectively reduce environmental interference in the non-projection area, ensuring that subsequent processing focuses on the actual projection area, thereby improving the accuracy and effect of the projection setting.

[0036] In some embodiments, the controller determines the updated first curtain coordinates and the corresponding first weight, and the updated first image card coordinates and the corresponding second weight based on the first curtain coordinates and the first image card coordinates, and is specifically configured as follows:

[0037] Based on the first curtain coordinates and the first image card coordinates, the second image is cropped to obtain at least one first cropped image corresponding to the first curtain coordinates and at least one second cropped image corresponding to the first image card coordinates;

[0038] Using the at least one first cropped image and the at least one second cropped image as inputs of a second neural network model, and using the second neural network model to obtain second curtain coordinates and the corresponding first weights, and second image card coordinates and the corresponding second weights; the second curtain coordinates are curtain coordinates corresponding to the first cropped image, and the second image card coordinates are image card coordinates corresponding to the second cropped image;

[0039] The second curtain coordinates and the second image card coordinates are converted into the coordinate system of the second image to obtain the updated first curtain coordinates and the updated first curtain coordinates.

[0040] The above technical solution has the following beneficial effects or advantages: the first image is cropped by the first curtain coordinates and the first image card coordinates, and the cropped image is input into the second neural network model to update the first curtain coordinates and the first image card coordinates, so that the accuracy of coordinate determination can be improved.

[0041] In some embodiments, the controller determines the first target coordinates corresponding to the target projection area in the optical machine coordinate system based on the updated first curtain coordinates, the first weight, the updated first image card coordinates, and the second weight, and is specifically configured as follows:

[0042] Based on the first weights, determining a third curtain coordinate of the second image; the third curtain coordinate is the curtain coordinate with the highest weight value among the first weights;

[0043] Determine a third card coordinate of the second image based on the second weight; the third card coordinate is the card coordinate with the highest weight value among the second weights;

[0044] Based on the third chart coordinates and the preset chart coordinates in the optical-mechanical coordinate system, a third conversion relationship is determined; the third conversion relationship is a mapping relationship between the camera coordinate system and the optical-mechanical coordinate system;

[0045] Based on the third conversion relationship, the third screen coordinates are converted to obtain the first target coordinates corresponding to the target projection area in the optical machine coordinate system.

[0046] The above technical solution has the following beneficial effects or advantages: by analyzing multiple first weights and multiple second weights, a set of second screen coordinates corresponding to the highest weight among the first weights and a set of second chart coordinates corresponding to the highest weight among the second weights are selected, and these coordinates are used as the final screen corner coordinates and chart corner coordinates, respectively. In this way, by selecting the most representative coordinate points, the accuracy of the first target coordinates is significantly improved, thereby ensuring the accurate positioning of the target projection area.

[0047] In some embodiments, the controller executes based on the execution strategy to determine the first target coordinates in the optical-mechanical coordinate system, and the specific configuration is:

[0048] When the execution strategy is the obstacle avoidance strategy, a binary image corresponding to the first image and a first pixel coordinate corresponding to the binary image are obtained; the first pixel coordinate includes a pixel coordinate corresponding to the obstacle and a pixel coordinate corresponding to the non-obstacle;

[0049] Based on a fourth conversion relationship, the first optical-mechanical coordinate is converted to obtain a first depth coordinate in a depth sensor coordinate system; the fourth conversion relationship is a mapping relationship between the optical-mechanical coordinate system and the depth sensor coordinate system;

[0050] Determine, based on the first pixel coordinates and the first depth coordinates, a third target coordinate in the depth sensor coordinate system, wherein the third target coordinate is a corner point coordinate of the target projection area in the projection plane in the depth sensor coordinate system;

[0051] Based on the fourth conversion relationship, the third target coordinates are converted to obtain the first target coordinates in the optical-mechanical coordinate system.

[0052] The above technical solution has the following beneficial effects or advantages: when there is no screen but there is an obstacle, the specific pixel coordinates of the obstacle can be determined by analyzing the binary image. Then, a maximum possible projection area is defined by using the first optical machine. In this area, by excluding the pixel positions identified as obstacles, an unobstructed maximum available projection area, i.e., the target projection area, can be calculated, thereby ensuring the optimization of the projection area.

[0053] In some embodiments, the controller executes based on the execution strategy to determine the first target coordinates in the optical-mechanical coordinate system, and the specific configuration is:

[0054] When the execution strategy is the sensorless correction strategy, the first optomechanical coordinates are used as the first target coordinates in the optomechanical coordinate system; wherein the first optomechanical coordinates are the corner point coordinates corresponding to the boundary area projected by the optomechanical system.

[0055] The above technical solution has the following beneficial effects or advantages: in the absence of a screen or obstacles, the corner point coordinates corresponding to the first optical-mechanical coordinates can be used as the first target coordinates to determine the optimal projection area and provide a somatosensory user experience.

[0056] In a second aspect, an embodiment of the present application further provides a method for determining a projection area, including:

[0057] In response to a first instruction, acquiring a first image of the projection surface through the depth sensor;

[0058] The first image is used as an input of a pre-trained first neural network model, and a recognition result is obtained using the first neural network model; the first neural network model is used to identify whether there is a target object on the projection surface;

[0059] Based on the recognition result, determining an execution strategy for acquiring a target projection area, the target projection area being an area where a projection image is displayed; the execution strategy having a corresponding relationship with the recognition result;

[0060] Based on the execution strategy, the target projection area in the projection surface is acquired.

[0061] The above technical solution has the following beneficial effects or advantages: The above technical solution has the following beneficial effects or advantages: By responding to the first instruction, the image of the projection surface is obtained by using the depth sensor, and the image is input into the pre-trained first neural network model for analysis. The model can identify the recognition result on the projection surface, and determine different execution strategies based on the recognition result to define the target projection area. In this way, not only the automation of the projection setting is improved, and the interaction steps with the user are reduced, but also the user's waiting time is significantly reduced by using the model processing, the processing efficiency of obtaining the target projection area is improved, and the user experience is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 A schematic diagram of a projection scene of a projection device provided in some embodiments of the present application;

[0063] Figure 2 A schematic diagram of the optical path of a projection device provided in some embodiments of the present application;

[0064] Figure 3 A schematic diagram of a circuit architecture provided for some embodiments of the present application;

[0065] Figure 4 A schematic diagram of the structure of a projection device provided in some embodiments of the present application;

[0066] Figure 5 A schematic diagram of a system framework for implementing display control for a projection device provided in some embodiments of the present application;

[0067] Figure 6 A schematic diagram of conversion between coordinate systems provided in some embodiments of the present application;

[0068] Figure 7 A first flow chart for obtaining a target projection area provided in some embodiments of the present application;

[0069] Figure 8 A schematic diagram of a first image is provided for some embodiments of the present application;

[0070] Fig. 9 A schematic diagram of a process for obtaining a recognition result is provided for some embodiments of the present application;

[0071] Fig.10 A first flow chart of obtaining first target coordinates provided in some embodiments of the present application

[0072] Fig.11 A second flow chart of obtaining first target coordinates provided in some embodiments of the present application;

[0073] Fig.12 A schematic diagram of a first captured image and a second image provided in some embodiments of the present application;

[0074] Fig.13 A schematic diagram of a process of cropping a second image provided in some embodiments of the present application;

[0075] Fig.14 A third flow chart of obtaining first target coordinates provided in some embodiments of the present application;

[0076] Fig.15 A binary image provided in some embodiments of the present application;

[0077] Fig.16 A schematic diagram of determining an area corresponding to a third target coordinate provided in some embodiments of the present application;

[0078] Fig.17 A second method flow chart for obtaining a target projection area is provided in some embodiments of the present application. DETAILED DESCRIPTION

[0079] In order to make the purpose and implementation method of the present application clearer, the exemplary implementation method of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0080] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0081] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0082] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0083] Projection equipment is a device that can project media data onto a projection medium. Projection equipment can be connected to computers, broadcasting networks, the Internet, VCD (Video Compact Disc), DVD (Digital Versatile Disc Recordable), game consoles, DV and other devices through different interfaces to receive media data to be projected. Among them, media data includes but is not limited to images, videos, texts and other types, and projection media includes but is not limited to physical forms such as walls, curtains, and screens.

[0084] Figure 1 A schematic diagram of a projection scene of a projection device provided in some embodiments of the present application.

[0085] In some embodiments, the projection device 100 may be a device with a projection function such as a projector or a laser TV. The present application does not limit the type of the projection device. The projection device is used to project a projection image onto a projection medium. For example, the projection device is a projector. Figure 1 The projector may include a projection host 2. The projection medium 1 is fixed at a first position, and the projection host 2 is placed at a second position. By adjusting the relationship between the first position and the second position, the projection image of the projection host 2 is made to match the projection medium 1, that is, the second position is the best placement position of the projection host 2. The projection medium may be a screen, a white wall, or the like.

[0086] Figure 2 A schematic diagram of the optical path of a projection device provided in some embodiments of the present application.

[0087] The projection host 2 includes a projection component, which includes a light source 210, an optical machine 220 and a lens 230. The light source 210 provides illumination for the optical machine 220, the optical machine 220 modulates the light beam of the light source and outputs it to the lens 230, the lens 230 forms an image and projects it to the projection medium 1, and the projection medium 1 presents a projection picture.

[0088] In some embodiments, the light source 210 may include a bulb assembly or an LED (Light Emitting Diode) light source, and the light beam emitted by the light source may be modulated and adjusted by the optical engine 220 to provide the required light source for the projection image. In this case, the optical engine 220 is usually composed of different optical components, including a color separation device and an adjustment device, to process the light beam emitted by the light source and convert it into a projection image.

[0089] In some embodiments, the optical engine 220 may also include blue, green and red optical engine modules, through which lasers or light sources suitable for displaying different colors are generated to achieve clear and accurate projection images. In addition, the optical engine 220 is usually equipped with a heat dissipation system and a circuit control system to ensure stable operation of the device.

[0090] In some embodiments, the light-emitting component of the projector can be realized in a variety of ways such as a bulb, LED or laser light source. Different light source types determine the brightness, color accuracy and service life of the projector.

[0091] Figure 3 A schematic diagram of a circuit architecture provided for some embodiments of the present application.

[0092] In some embodiments, see Figure 3 The projection host 2 may include a display control circuit 240, a laser light source 210, at least one laser driving component 250, and at least one brightness sensor 260. The laser light source 210 may include at least one laser corresponding to the at least one laser driving component. The at least one refers to one or more, and the multiple refers to two or more.

[0093] In some embodiments, the display control circuit 240 is used to output light control signals corresponding to different primary colors to the laser driving component 250 to drive the corresponding laser to emit light. For example, the light control signal includes a blue light control signal, a red light control signal, and a green light control signal. Figure 3The display control circuit 240 is connected to the laser driving component 250, and is used to output at least one light control signal corresponding to the three primary colors of each frame of the multi-frame display image, and transmit the at least one light control signal to the corresponding laser driving component 250. For example, the display control circuit 240 can be a microcontroller unit (MCU), also known as a single chip microcomputer.

[0094] In some embodiments, the projector can achieve adaptive adjustment. For example, by setting a brightness sensor 260 in the light output path of the laser light source 210, the brightness sensor 260 can detect the first brightness value of the laser light source 210 and send the first brightness value to the display control circuit 240. The display control circuit 240 can obtain the second brightness value corresponding to the driving current of each laser, and when the difference between the second brightness value of the laser and the first brightness value of the laser is greater than the difference threshold, it is determined that the laser has a COD (Catastrophic optical damage) failure, and the display control circuit 240 can adjust the current control signal of the laser driving component corresponding to the laser until the aforementioned difference is less than or equal to the difference threshold, thereby eliminating the COD failure of the laser, reducing the damage rate of the laser, and improving the image display effect of the projection device.

[0095] Figure 4 A schematic diagram of the structure of a projection device provided in some embodiments of the present application.

[0096] In some embodiments, see Figure 4 The optical path structure includes a laser light source 210 and an optical component 214. The laser light source 210 may include an independently arranged blue laser 211, a red laser 212, and a green laser 213. The projection device may also be referred to as a three-color projection device. The blue laser 211, the red laser 212, and the green laser 213 are all module lightweight (Mirai Console Loader, MCL) packaged lasers with a small size, which is conducive to the compact arrangement of the optical path.

[0097] In some embodiments, the projection host 2 may include a controller, which includes at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), a RAM Random Access Memory (RAM), a ROM (Read-Only Memory, ROM), a first interface to an nth interface for input / output, and a communication bus (Bus), etc. The controller is connected to the relevant hardware of the projection device, such as the display control circuit, the brightness sensor, the distance sensor, the image collector, etc., and is used to control the projection, focusing, correction, calibration, switch screen state adjustment, etc. of the projection device.

[0098] In some embodiments, several types of interfaces may be provided on the body of the projection device (e.g., a laser TV), such as a power interface, a USB interface, an HDMI (High Definition Multimedia Interface) interface, a network cable interface, a VGA (Video Graphics Array) interface, a DVI (Digital Visual Interface), etc., to connect a signal source for transmitting media.

[0099] In some embodiments, after the projection device is started, it can directly enter the display interface of the signal source selected last time, or the signal source selection interface, where the signal source is, for example, a preset video-on-demand program, or can be one of the signal sources such as an HDMI interface, a USB interface, a live TV interface, etc. After the user selects the target signal source, the projection host 2 can obtain media data from the target signal source and project the media data onto the projection medium 1 for display.

[0100] In some embodiments, the projection host 2 may be configured with a camera for cooperating with the projection host 2 to achieve relevant adjustment and control of the projection process. For example, the projection device may be configured with a 3D camera, a monocular camera, or a binocular camera. When the camera is implemented as a binocular camera, it specifically includes a left camera and a right camera; the binocular camera can obtain the screen corresponding to the projection device, that is, the image and playback content presented by the projection surface, and the image or playback content is projected by the built-in optical machine of the projection device.

[0101] When the projection device moves, its projection angle and distance to the projection surface change, which will cause the projected image to deform. The projected image will appear as a trapezoidal image or other deformed image. The projection device controller can automatically enter the screen based on the image taken by the camera by coupling the angle between the optical and mechanical projection surfaces and the correct display of the projection image.

[0102] Figure 5 A schematic diagram of a system framework for implementing display control for a projection device provided in some embodiments of the present application.

[0103] In some embodiments, see Figure 5 The system framework includes application service layer, process communication framework, operation layer, framework layer, correction service, camera service, flight time service, hardware and its driver, etc. The controller of the projection host 2 controls the overall system architecture and implements projection control of the projection device based on the underlying program logic, including but not limited to automatic screen entry, automatic obstacle avoidance, automatic focus, anti-eye shooting, screen switch control, automatic correction of the projection picture and fine-tuning correction, etc.

[0104] In some embodiments, the projection host 2 is also configured with a distance sensor for detecting distance. The distance sensor can be a time of flight (TOF) sensor. The time of flight sensor uses the flight time of a signal between a transmitting end and a reflecting end to measure the distance between nodes. After the time of flight sensor collects the distance data, it sends the distance data to the time of flight service; after the time of flight service obtains the distance data, it sends the collected distance data to the application service layer through the process communication framework. The distance data will be used for interactive use such as controller data calls, user interfaces, and program applications.

[0105] In some embodiments, the projection host 2 may also be configured with an image collector, which may be a monocular camera, a binocular camera, a depth camera or a 3D camera, etc.; the image collector sends the collected image data to the camera service, and then the camera service sends the image data to the process communication framework and / or the correction service; the process communication framework sends the image data to the application service layer, and the image data will be used for interactive use such as the controller's data call, user interface, and program application.

[0106] In some embodiments, the projection device can be refocused after automatic calibration is completed, and the controller detects whether the automatic focusing function is turned on; if the automatic focusing function is not turned on, the controller will end the automatic focusing service; if the automatic focusing function is turned on, the controller performs focusing calculations based on the distance detection value of the time of flight sensor.

[0107] In some embodiments, the controller queries a preset mapping table based on the distance detection value of the time-of-flight sensor, and the preset mapping table records the mapping relationship between distance and focal length, so as to obtain the focal length of the projection device corresponding to the distance detection value; then the middleware sends the acquired focal length to the optical machine of the projection device; after the optical machine performs laser emission according to the above focal length, at least one image collector captures the projection content image, and the controller performs clarity detection on the projection content image to determine whether the current lens focal length is appropriate. If the focal length is not appropriate, focusing processing is required. The projection device locates the focus position with the highest clarity by adjusting the lens position and shooting, and comparing the clarity changes of the projection content image before and after the adjustment.

[0108] If the judgment result meets the preset completion conditions, the control of the automatic focusing process ends; if the judgment result does not meet the preset completion conditions, the middleware will fine-tune the focal length parameters of the projection device optical machine, such as gradually fine-tuning the focal length according to the preset step length, and setting the adjusted focal length parameters to the optical machine again, through multiple photos, clarity evaluation and other steps, and finally through the clarity comparison of the projection picture, the optimal focal length is locked, thereby completing the automatic focusing.

[0109] In some embodiments, in order to enhance the viewing experience, the projection device needs to calibrate the light-emitting component when projecting content onto the projection surface for the first time, or when the device moves during the projection process. This ensures that the projected content can be presented completely and stably on the projection surface. In a projection device, there are multiple key components, each located in an independent coordinate system. For example, the projection surface has its own physical coordinate system, the optical machine of the projected light runs in its unique optical machine coordinate system, the camera operation is based on the camera coordinate system, and the depth sensor works according to the depth sensor coordinate system. In order to ensure that the projected image can be displayed accurately and the light-emitting component is automatically calibrated when the device is moved or set up for the first time, precise conversion must be performed between these different coordinate systems.

[0110] For example, in combination Figure 6 As shown, a conversion relationship 1 is defined between the optical-mechanical coordinate system and the physical coordinate system where the projection surface is located, wherein conversion relationship 1 corresponds to the first conversion relationship in this application, a conversion relationship 2 is established between the optical-mechanical coordinate system and the camera coordinate system corresponding to the camera, wherein conversion relationship 2 corresponds to the second conversion relationship in this application, and a conversion relationship 3 is set between the optical-mechanical coordinate system and the depth sensor coordinate system corresponding to the depth sensor, wherein conversion relationship 3 corresponds to the fourth conversion relationship in this application. Based on the first conversion relationship, conversion relationship 2 and conversion relationship 3, the projection device can achieve accurate automatic correction and optimized projection effect in various situations.

[0111] In one embodiment, in order to obtain the best target projection area, the user is required to interact multiple times to meet specific operational requirements. For example, in order to realize the entry function, the user often needs to start by pressing a specific button or selecting an option in the menu; similarly, functions such as obstacle avoidance and automatic correction also require the user to operate through similar interactive methods. After obtaining these instructions, the traditional method also involves placing one or two full-screen image cards, and using these image cards in combination with traditional image processing algorithms to calculate the projection area. However, this method requires the user to make their own judgments and perform a series of steps, resulting in frequent interactions and long waiting times, which affects the user experience. Not only does the user need to spend extra time to set up, but the slow processing speed of the traditional image processing algorithm further prolongs the preparation time and reduces the overall user satisfaction.

[0112] In order to solve the above problems, the present application provides a projection device 200, wherein the projection device can be a telephoto projection device. The projection device 200 includes a depth sensor, an optical machine, and a controller. The depth sensor is configured to acquire an image of a projection surface. The optical machine is configured to project a media resource. Figure 7 As shown, the controller is configured as:

[0113] S1: In response to a first instruction, acquiring a first image of a projection surface through a depth sensor.

[0114] In one embodiment, the projection device 200 may provide a first button for the user, and the first button includes a first key value, a second key value, and a third key value. The first key value may correspond to the screen entry function, the second key value may correspond to the obstacle avoidance function, and the third key value may correspond to the non-sensing function. In other words, the first button is a multi-function button. Exemplarily, in response to the user's click operation on the first button, a first instruction is generated. The first instruction is an intelligent instruction covering multiple functions.

[0115] Continuing with the above example, after generating the first instruction, the projection device 200 controls the depth sensor to acquire the first image of the projection surface based on the first instruction.

[0116] like Figure 8 As shown, Figure 8 A schematic diagram of a first image is provided for some embodiments of the present application.

[0117] The first image is an amplitude image, which is an image that represents the reflection intensity or brightness information of each point in the scene. It is usually used to describe the light intensity distribution on the surface of an object. It can be understood as a traditional grayscale image or color image, in which each pixel value represents the light intensity at the corresponding position. Exemplarily, the first light intensity is less than the second light intensity, that is, the area corresponding to the first light intensity has low brightness, and the area corresponding to the second light intensity has high brightness.

[0118] The amplitude image can be obtained by a depth sensor. Among them, the depth sensor is a device that can capture three-dimensional information of an object or scene. It not only provides an amplitude image of brightness or color information, but also measures the distance from the sensor to each point in the scene (ie, depth information). Exemplarily, the depth sensor can be a time of flight (TOF) sensor. The projection surface refers to a plane in three-dimensional space for receiving projected images or videos, that is, the projection surface is a plane in a physical coordinate system, which is used to ensure that the projected content can be accurately displayed on it.

[0119] For example, in combination Figure 8 As shown, the first image is an amplitude image acquired by a depth sensor.

[0120] S2: Using the first image as input of a pre-trained first neural network model, and using the first neural network model to obtain a recognition result.

[0121] The first neural network model is used to identify whether there is a target object on the projection surface. The target object includes at least one of a curtain or an obstacle.

[0122] A pre-trained neural network model refers to a neural network model that has been trained on a large-scale data set before use. In order to ensure the generalization ability of the neural network model, the neural network model needs to be trained based on a large amount of training data.

[0123] In one implementation, the training method of the first neural network model is used as an example for illustration.

[0124] First, get the training set.

[0125] The training set includes at least one training image, a target classification label corresponding to at least one training image, and a target image corresponding to each target classification label. Each label clearly indicates the category to which the image belongs. For example, the first classification label is 0 and a pure black image, where 0 represents that the training image includes a curtain; the second classification label is 1 and a pure black image, where 1 represents that there is no curtain or obstacle in the training image; the second classification label is 2 and a binary image, where 2 represents that there is an obstacle in the training image. The binary image is also called a mask image. The mask image includes the pixel coordinates of the pixel, such as the pixel coordinates of 255 where there is an obstacle and the pixel coordinates of 0 in other places.

[0126] Next, the first neural network model is trained by taking at least one training image as input of the first neural network model, and taking the target classification label and the target image corresponding to each target classification label as output of the first neural network model.

[0127] Input at least one training image into the initial neural network model, output the predicted classification label and the predicted image, further, use the preset loss function to determine the loss value between the predicted classification label and the target classification label, and the loss value between the predicted image and the target image, respectively. When the above loss value is less than the preset loss value, it means that the training of the initial neural network model is completed, and the initial neural network model is the trained first neural network model.

[0128] After the first neural network model is trained, the first neural network model is used to obtain a recognition result.

[0129] For example, in combination Fig. 9 As shown, the first image is input into the first neural network model, and outputs 0 and a pure black image, or outputs 1 and a pure black image, or outputs 2 and a mask image.

[0130] In order to improve the recognition accuracy, the first image may be cropped to obtain a cropped first image.

[0131] Exemplarily, based on the fourth conversion relationship, the first optical-mechanical coordinates are converted into the depth sensor coordinate system to obtain a first depth coordinate, and based on the first depth coordinate, the first image is cropped to obtain a cropped first image.

[0132] Exemplarily, the fourth conversion relationship may be determined by formula (1):

[0133] T uv =M t *(R P2w *R P2t ) -1 *(T P2w (2)*R P2w *M P -1 *P uv +T P2t -T P2w ) Formula (1)

[0134] Among them, T uv is the corner coordinate under the depth sensor, M t is the internal parameter matrix of the depth sensor, R P2t is the rotation matrix from the optical machine coordinate system to the depth sensor coordinate system, T P2w is the translation vector from the optomechanical coordinate system to the physical coordinate system, R P2w is the rotation matrix from the optomechanical coordinate system to the physical coordinate system, R P2c is the rotation matrix from the optical machine coordinate system to the camera coordinate system, M P is the internal parameter matrix of the optical machine, P uvis the first optomechanical coordinate in the optomechanical coordinate system, T P2t is the translation vector from the optical machine coordinate system to the depth sensor coordinate system. In order to facilitate the following description, the internal parameter matrix is ​​described using internal parameters.

[0135] Continuing with the above example, uv =P uv 1 to T uv =T uv 1

[0136] S3: Based on the recognition result, determine an execution strategy for acquiring the target projection area.

[0137] The target projection area is the area where the projected image is displayed. For example, the target projection area is usually set as a rectangle with a fixed aspect ratio. This setting ensures the visual coordination and accuracy of the projected content. For example, an inscribed rectangle of 16:9.

[0138] In one embodiment, the target object includes at least one of a curtain or an obstacle. When the recognition result includes the curtain, the execution strategy for obtaining the target projection area is determined to be the curtain entry strategy; when the recognition result does not include the curtain but includes the obstacle, the execution strategy for obtaining the target projection area is determined to be the obstacle avoidance strategy; when the recognition result does not include the curtain and does not include the obstacle, the execution strategy for obtaining the target projection area is determined to be the non-sensing correction strategy.

[0139] That is to say, the recognition result may contain multiple types of information, which determines the diversity of the execution strategy. That is to say, different recognition results will correspond to different execution strategies. For example, when the recognition result is 0, that is, the target object on the projection surface includes the curtain, and when the recognition result includes the curtain, the execution strategy for obtaining the target projection area is determined to be the curtain entry strategy. When the recognition result is 1, that is, the target object on the projection surface does not include the curtain and obstacles, and when the recognition result does not include the curtain and includes obstacles, the execution strategy for obtaining the target projection area is determined to be the obstacle avoidance strategy. When the recognition result is 2, that is, the target object on the projection surface does not include the curtain and obstacles, and when the recognition result does not include the curtain and obstacles, the execution strategy for obtaining the target projection area is determined to be the non-sensing correction strategy.

[0140] S4: Based on the execution strategy, determine the target projection area in the projection surface.

[0141] The target projection area refers to the optimal range for displaying the projected image.

[0142] In one embodiment, S4 includes steps S41 to S43.

[0143] Step S41: Based on the execution strategy, determine the first target coordinates in the optical-mechanical coordinate system.

[0144] The first target coordinates are the coordinates of the corner points of the target projection area in the projection plane in the optical-mechanical coordinate system.

[0145] The following describes different execution strategies for determining the first target coordinates of the optical-mechanical coordinate system.

[0146] Combine the following Fig.10 An exemplary description is given of obtaining the first target coordinates based on the entry strategy.

[0147] Fig.10 A first flowchart for obtaining first target coordinates is provided in some embodiments of the present application.

[0148] In one example, if Fig.10 As shown, when the execution strategy is the entry-screen strategy, obtaining the first target coordinates in the optical-mechanical coordinate system includes the following steps:

[0149] S411: When the execution strategy is the entry-to-screen strategy, a second image corresponding to the projection surface is acquired.

[0150] Exemplarily, S411 includes S4110-S4114.

[0151] S4110: When the execution strategy is the entry-to-screen strategy, an image of a positioning card is projected into a preset area of ​​the screen object through an optical machine.

[0152] A preset area is set in the screen for projecting the positioning card. The size of the preset area is not specifically limited, but it must be large enough to accommodate all the key points of the positioning card and ensure that the minimum spacing requirements are met between these key points. For example, the minimum spacing is 1 pixel. For example, if the specified minimum spacing is 1 pixel, then the coordinate difference between any two corner points (either horizontally or vertically) must be greater than 1 pixel. At the same time, the preset area must be completely within the range of the screen, that is, it must not exceed the actual size of the screen.

[0153] It should be noted that there is no specific limitation on the setting of the minimum spacing.

[0154] Alignment Chart or Calibration Target is a special image or pattern used to calibrate and adjust optical systems, projectors, cameras and other equipment. They usually contain a series of precisely designed geometric shapes, lines, points or other features that help ensure that the equipment can accurately capture or project images, and can be used to measure and correct various types of distortion or errors. Exemplarily, at a first focal length, the image of the alignment chart is projected into a preset area in the screen by an optical machine. For example, the preset area can be the center area of ​​the screen.

[0155] Exemplarily, a positioning map card is projected onto a curtain on a projection surface, and the positioning map card is arranged within a preset area.

[0156] S4111: Control the camera to obtain a first captured image corresponding to the projection surface.

[0157] Combination Fig.11 As shown in (a) of FIG. 1 , the first captured image includes a curtain object 90 and a positioning card object 91 . The positioning card object 91 is set in a preset area of ​​the curtain object 90 .

[0158] S4112: Based on the second conversion relationship, the first optical-mechanical coordinates are converted into the camera coordinate system to obtain first camera coordinates.

[0159] Among them, the first optical coordinates are the corner point coordinates corresponding to the boundary area projected by the optical machine, that is, the first optical coordinates are the corner point coordinates in the optical coordinate system; the first camera coordinates are the corner point coordinates of the first optical coordinates in the camera coordinate system.

[0160] The second transformation relationship is an approximate affine relationship between the camera coordinate system and the optomechanical coordinate system.

[0161] The following is an exemplary description of obtaining the second conversion relationship.

[0162] The projection device 200 can determine a first conversion relationship between the optical machine and the projection surface based on the first parameter information.

[0163] The first parameter information includes the internal parameters of the optical machine, the depth information of the depth sensor, and the external parameter matrix between the depth sensor and the optical machine. The following is an explanation of the external parameters. Among them, the internal parameters Mp of the optical machine, also known as the internal parameter matrix of the optical machine, refers to the parameters that describe the internal optical system of the optical machine (such as the focal length of the optical machine, the coordinates of the principal point, etc.), which are used to determine the mapping relationship from the three-dimensional space to the two-dimensional projection screen. The external parameters between the TOF sensor and the optical machine include the rotation matrix R P2w and the translation vector T P2w , the rotation matrix R P2w and the translation vector T P2wIt is the relationship between the optomechanical coordinate system and the TOF sensor coordinate system.

[0164] Exemplarily, the first conversion relationship may be determined using formula (2):

[0165] W xyz =T P2w (2)*R P2w *M P -1* P uv Formula (2)

[0166] Among them, W xyz is the corner point coordinate T of the physical coordinate system P2w , is the translation vector from the optomechanical coordinate system to the physical coordinate system, M P is the internal parameter matrix of the optomechanical system, which is used to describe the internal parameters in the optomechanical coordinate system. uv is the first optomechanical coordinate in the optomechanical coordinate system.

[0167] The projection device 200 determines a second conversion relationship between the optical machine and the camera based on the first conversion relationship and the second parameter information.

[0168] The second parameter information includes the camera's internal parameter M c , the first external reference R between the camera coordinate system and the optical machine coordinate system P2c and the second external parameter T P2c Among them, the camera's internal parameter M c , refers to the parameters describing the internal optical system of the camera, which is used to convert the three-dimensional world coordinates into two-dimensional image coordinates. The internal parameters of the camera M c Usually includes the focal length of the camera, the coordinates of the principal point, etc. The first external parameter R between the camera coordinate system and the optical machine coordinate system P2c It is a rotation matrix used to describe the rotation relationship between the optical-mechanical coordinate system and the camera coordinate system. The first external parameter T between the camera coordinate system and the optical-mechanical coordinate system P2c It is a translation vector, which is used to describe the translation relationship from the origin of the optical-mechanical coordinate system to the origin of the camera coordinate system, and represents the relative displacement between the two coordinate systems.

[0169] Exemplarily, the second conversion relationship may be determined using formula (3):

[0170] C uv =M c *(R P2w *R P2c ) -1 *(T P2w (2)*R P2w *M P -1 *P uv +T P2c-T P2w ) Formula (3)

[0171] Among them, C uv is the corner point coordinate in the camera coordinate system, M c is the camera’s intrinsic parameter matrix, R P2w is the rotation matrix from the optomechanical coordinate system to the physical coordinate system, R P2c is the rotation matrix from the optical machine coordinate system to the camera coordinate system, T P2w is the translation vector from the optomechanical coordinate system to the physical coordinate system, T P2c is the translation vector from the optical-mechanical coordinate system to the camera coordinate system, M P is the internal parameter matrix of the optomechanical system, which is used to describe the internal parameters in the optomechanical coordinate system. uv is the first optomechanical coordinate in the optomechanical coordinate system.

[0172] Exemplarily, the first optical coordinate P uv 1 to C uv 1.

[0173] S4113: Based on the first camera coordinates, crop the first captured image to obtain a second image.

[0174] Combination Fig.11 As shown in (b), the second image includes a curtain object 90 and a positioning card object 91, and the positioning card object 91 is set in a preset area of ​​the curtain object. The main area of ​​the second image is larger than the main area of ​​the first captured image.

[0175] S412: Acquire first curtain coordinates corresponding to the curtain object and first image card coordinates corresponding to the positioning image card object in the second image.

[0176] The first curtain coordinates are the corner point coordinates corresponding to the curtain object in the camera coordinate system, and the first image card coordinates are the corner point coordinates corresponding to the positioning image card object in the camera coordinate system.

[0177] In one embodiment, the first image is used as the input of the pre-trained third neural network model, and the third neural network model is used to obtain the first curtain coordinates of the curtain object in the second image and the first image card coordinates of the positioning image card object.

[0178] Optionally, the training method of the third neural network model can refer to the third neural network model in step S2 above, which will not be repeated here.

[0179] For the sake of explanation, the first curtain coordinates and the first image card coordinates are taken as two-dimensional coordinates, and the homogeneous coordinates are not used.

[0180] Following the above example, the second image is input into the pre-trained third neural network model to obtain the first curtain coordinates of the curtain object and the first card coordinates corresponding to the positioning card object. 1’ ,y 1’ ), B1(x 2’ ,y 2’ ), C1(x 3’ ,y 3’ ), D1(x 4’ ,y 4’ ), the first chart coordinate E1(x 80’ ,y 80’ ), F1(x 81’ ,y 82’ ), G1(x 83’ ,y 83’ ), H1(x 84’ ,y 84’ ), I1(x 85’ ,y 85’ ), J1(x 86’ ,y 86’ ), K1(x 87’ ,y 87’ ), L1(x 88’ ,y 88’ ).

[0181] In order to improve the accuracy of the first coordinate of the curtain object and the second coordinate of the positioning card object, the acquisition of the first coordinate of the curtain object and the second coordinate of the positioning card object may be optimized, as shown in S413.

[0182] S413: performing an update operation on the first curtain coordinates and the first image card coordinates to determine the updated first curtain coordinates and the corresponding first weights, and the updated first image card coordinates and the corresponding second weights.

[0183] Among them, the updated first screen coordinates are the screen coordinates obtained after the update operation is performed in the second image coordinate system; the updated first screen coordinates are the image card coordinates obtained after the update operation is performed in the second image coordinate system, and the update operation includes at least one of moving, scaling, and rotating.

[0184] For example, in combination Fig.12 As shown, S413 includes S4131-S4133.

[0185] S4131: Based on the first screen coordinates and the first image card coordinates, the second image is cropped to obtain at least one cropped image corresponding to the first screen coordinates and at least one second cropped image corresponding to the first image card coordinates.

[0186] Combination Fig.13An exemplary description is given.

[0187] Fig.13 A schematic diagram of a process of cropping a second image provided in some embodiments of the present application.

[0188] like Fig.13 As shown, each coordinate point of the first curtain coordinates and the first image card coordinates is used as a center point, and cropping is performed based on the center point and the offset to obtain at least one first cropped image and a second cropped image.

[0189] For example, a 96*96 image is cropped with the center point as the center on the second image to obtain at least one first cropped image and a second cropped image.

[0190] The ratio of the cropped image may be the above-mentioned 96*96, or may be other ratios, which are not specifically limited here.

[0191] Following the above example, the first curtain coordinates and the first image card coordinates in the second image are cropped. Taking one of the second images as an example, the first curtain coordinates are A1(x 1’ ,y 1’ ), B(x 2’ ,y 2’ ), C1(x 3’ ,y 3’ ), D1(x 4’ ,y 4’ ), the above A1(x 1’ ,y 1’ ) as the center point, with (x 1’ -48,y 1’ -48)、(x 1’ +48,y 1’ +48) as the vertex to obtain four first cropped images corresponding to the four corner points corresponding to the curtain object. Similarly, taking the first card coordinates as an example, eight second cropped images corresponding to the eight corner points corresponding to the positioning card object can be obtained.

[0192] S4132: Using at least one first cropped image and at least one second cropped image as inputs of a second neural network model, and using the second neural network model to obtain second curtain coordinates and first weights corresponding to the first cropped image, and second image card coordinates and second weights corresponding to the second cropped image.

[0193] Among them, the second curtain coordinates are the curtain coordinates corresponding to the first cropped image, and the second card coordinates are the card coordinates corresponding to the second cropped image; the second neural network model is used to obtain the corner point coordinates in the image and the weight ratio of each corner point coordinate.

[0194] The training method of the second neural network model can refer to the above step S2 and will not be repeated here.

[0195] The weight is used to reflect the accuracy of the current coordinate point.

[0196] Following the above example, the four first cropped images corresponding to the four corner points and the eight second cropped images corresponding to the eight corner points are input into the second neural network model to obtain the second curtain coordinates A1(x 1” ,y 1” ), B1(x 2” ,y 2” ), C1(x 3” ,y 3” ), D1(x 4” ,y 4” ), and the first weight a1 corresponding to A1, the first weight b1 corresponding to B1, the first weight c1 corresponding to C1, and the first weight d1 corresponding to D1. Similarly, the second chart coordinate E1 (x 80” ,y 80” ), F1(x 81” ,y 82” ), G1(x 83” ,y 83” ), H1(x 84” ,y 84” ), I1(x 85” ,y 85” ), J1(x 86” ,y 86” ), K1(x 87” ,y 87” ), L1(x 88” ,y 88” ), and the second weight e1 corresponding to E1, the second weight f1 corresponding to F1, the second weight g1 corresponding to G1, the second weight h1 corresponding to H1, the second weight i1 corresponding to I1, the second weight j1 corresponding to J1, the second weight k1 corresponding to K1, and the second weight l1 corresponding to L1.

[0197] S4133: Convert the second curtain coordinates and the second image card coordinates into the coordinate system of the second image to obtain updated first curtain coordinates and updated first curtain coordinates.

[0198] The coordinates of the second curtain and the second image card are the coordinates in the camera coordinate system. The coordinates of the same point in the cropped image are different from those in the first image, so coordinate conversion is required to obtain the coordinates corresponding to the first image. In other words, the coordinates of the second curtain and the second image card are the coordinates corresponding to the 96*96 image coordinate system after cropping.

[0199] Continuing with the above example, the coordinates of the second curtain A1 (x1” ,y 1” ), B1(x 2” ,y 2” ), C1(x 3” ,y 3” ), D1(x 4” ,y 4” ) and the second chart coordinate E1(x 80” ,y 80” ), F1(x 81” ,y 82” ), G1(x 83” ,y 83” ), H1(x 84” ,y 84” ), I1(x 85” ,y 85” ), J1(x 86” ,y 86” ), K1(x 87” ,y 87” ), L1(x 88” ,y 88” ) to obtain the updated first curtain coordinate A1(x 1”’ ,y 1”’ ), B1(x 2”’ ,y 2”’ ), C1(x 3”’ ,y 3”’ ), D1(x 4”’ ,y 4”’ ) and the updated first card coordinates E1(x 80”’ ,y 80”’ ), F1(x 81”’ ,y 82”’ ), G1(x 83”’ ,y 83”’ ), H1(x 84”’ ,y 84”’ ), I1(x 85”’ ,y 85”’ ), J1(x 86”’ ,y 86”’ ), K1(x 87”’ ,y 87”’ ), L1(x 88”’ ,y 88”’ ).

[0200] In order to further improve the accuracy of the updated first curtain coordinates and the updated first image card coordinates, the above steps S412-S413 are repeated. The number of repetitions is not specifically limited.

[0201] Exemplarily, the number of repetitions can be set to 5 times. Then the above steps S411-S413 can be repeatedly performed, that is, 6 first captured images can be acquired to obtain 6 second images, and then 6 first curtain coordinates and first image card coordinates can be obtained. After cropping, 24 first cropped images and 48 second cropped images are obtained. The 24 first cropped images and 48 second cropped images cropped 6 times are input into the second neural network model to obtain 6 updated first curtain coordinates and the first weight corresponding to each corner point in each first curtain coordinate, as well as 6 updated first image card coordinates and the second weight corresponding to each corner point in each first image card coordinate.

[0202] It should be noted that the number of repetitions is not specifically limited.

[0203] S414: Determine the first target coordinates corresponding to the target projection area in the optical-mechanical coordinate system based on the updated first screen coordinates, the first weight, the updated first image card coordinates, and the second weight.

[0204] For example, continue to combine Fig.12 As shown, step S414 includes S4141-S4144.

[0205] S4141: Determine the third curtain coordinates of the second image based on the first weight.

[0206] The third curtain coordinates are the corner coordinates corresponding to the curtain object in the first image in the camera coordinate system. The third curtain coordinates are the curtain coordinates corresponding to the highest weight value among the updated first curtain coordinates in the six second images.

[0207] Following the above example, for the sake of convenience, take the coordinates of a corner point in the first curtain coordinates in the six second images as an example, such as A1(x 1”’ ,y 1”’ ), the corresponding first weight is a1; A2(x 1”’ ,y 1”’ ), the corresponding first weight is a2; A3(x 1”’ ,y 1”’ ), the corresponding first weight is a3; A4(x 1”’ ,y 1”’ ), the corresponding first weight is a4; A5(x 1”’ ,y 1”’ ), the corresponding first weight is a5; A6(x 1”’ ,y 1”’), the corresponding first weight is a6. If a1=0.3, a2=0.5, a3=0.3, a4=0.7, a5=0.7, a6=0.6, then the first curtain coordinates corresponding to the fourth second image are determined as the final first curtain coordinates.

[0208] S4142: Determine the third card coordinates of the second image based on the second weight.

[0209] The third card coordinate is the card coordinate with the highest weight value in the second weights. That is, the third curtain coordinate is the card coordinate corresponding to the highest weight value in the updated first card coordinates in the six second images.

[0210] The specific content of S4142 can refer to the above S4141 and will not be repeated here.

[0211] S4143: Determine a third conversion relationship based on the third chart coordinates and the preset chart coordinates in the optical-mechanical coordinate system.

[0212] The third transformation relationship is the precise mapping relationship between the camera coordinate system and the optical-mechanical coordinate system.

[0213] Based on the correspondence between the third image card coordinates and the preset image card coordinates in the optical machine, the precise mapping relationship between the camera and the optical machine is represented. The mapping relationship can be called the homography matrix H2.

[0214] Exemplarily, the third conversion relationship may be determined based on formula (3):

[0215] P uv =H2*C uv Formula (3)

[0216] Among them, P uv is the coordinate in the optical-mechanical coordinate system; C uv is the coordinate in the camera coordinate system.

[0217] S4144: Based on the third conversion relationship, the third screen coordinates are converted to obtain the first target coordinates corresponding to the target projection area in the optical-mechanical coordinate system.

[0218] Following the above example, the first target coordinates corresponding to the target projection area in the optical-mechanical coordinate system are determined based on formula (3).

[0219] Combine the following Fig.14 The obstacle avoidance strategy for obtaining the first target coordinates is exemplified.

[0220] Fig.14 A second flow chart of obtaining first target coordinates is provided in some embodiments of the present application.

[0221] In another example, Fig.14 As shown, step S41: based on the execution strategy, determining that the specific execution strategy of the first target coordinates in the optical-mechanical coordinate system is an obstacle avoidance strategy, that is, when the execution strategy is the obstacle avoidance strategy, obtaining the first target coordinates in the optical-mechanical coordinate system includes the following steps:

[0222] S415: When the execution strategy is the obstacle avoidance strategy, a binary image corresponding to the first image and a first pixel coordinate corresponding to the binary image are obtained.

[0223] The first pixel coordinates include pixel coordinates corresponding to obstacles and pixel coordinates corresponding to non-obstacles.

[0224] For example, Fig.15 In the binary image shown, the pixel coordinates of the places where there are obstacles in the binary image are 255, and the pixel coordinates of other places are 0. S416: Based on the fourth conversion relationship, the first optical-mechanical coordinates are converted to obtain a first depth coordinate in the depth sensor coordinate system.

[0225] Among them, the fourth conversion relationship is a mapping relationship between the optical-mechanical coordinate system and the depth sensor coordinate system.

[0226] Exemplarily, the fourth conversion relationship can be determined by formula (4):

[0227] T uv =M t *(R P2w *R P2t ) -1 *(T P2w (2)*R P2w *M P -1 *P uv +T P2t -T P2w ) Formula (4)

[0228] Among them, T uz is the corner coordinate under the depth sensor, M t is the internal parameter of the depth sensor, R P2t is the rotation matrix T from the optical machine coordinate system to the depth sensor coordinate system P2w , is the translation vector from the optomechanical coordinate system to the physical coordinate system, R P2w is the rotation matrix from the optomechanical coordinate system to the physical coordinate system, R P2c is the rotation matrix from the optical machine coordinate system to the camera coordinate system, M P is the internal parameter matrix of the optomechanical system, which is used to describe the internal parameters in the optomechanical coordinate system. uv is the first optomechanical coordinate in the optomechanical coordinate system, T P2tis the translation vector from the optical-mechanical coordinate system to the depth sensor coordinate system.

[0229] Continuing with the above example, uv 1 to T uv 1.

[0230] S417: Determine a third target coordinate in the depth sensor coordinate system based on the first pixel coordinate and the first depth coordinate.

[0231] The third target coordinates are the coordinates of the corner points of the target projection area in the projection plane in the depth sensor coordinate system.

[0232] S417 can determine the third target coordinates by using a water flooding method.

[0233] In one example, the projection device 200 determines a fourth target coordinate in the depth sensor coordinate system based on the first pixel coordinate, the first depth coordinate and the first speed; after determining the fourth target coordinate in the depth sensor coordinate system, determines a third target coordinate in the depth sensor coordinate system based on the first pixel coordinate, the first depth coordinate and the second speed; wherein the first speed is greater than the second speed.

[0234] The fourth target coordinates are the middle coordinates of the corner points of the projection area to be determined.

[0235] Combination Fig.16 As shown in (a), a smaller 16:9 rectangle is placed within the range of the first depth coordinate, and the rectangle is pulled in the direction of the arrow in sequence, and the 16:9 shape is always maintained during the pulling process. The pulling step length is a distributed pulling method, such as pulling in large steps at the first speed, and then pulling in small steps at the second speed, thereby reducing the number of pulling times, until the four vertices of the 16:9 rectangle meet the first condition, and then the size of the 16:9 rectangle is further determined, thereby determining the fifth target coordinate. Among them, the first condition is that the corner point coordinates corresponding to the 16:9 rectangle do not exceed the range of the first depth coordinate and do not overlap with the obstacle.

[0236] The rectangular area corresponding to the third target coordinates obtained above is not necessarily the best projection area, so a judgment is required.

[0237] Exemplarily, if the rectangular area corresponding to the third target coordinates does not meet the second condition, that is, the second condition is that the 16:9 rectangle can move without exceeding the range of the first depth coordinates and without overlapping with the obstacle. If the preset condition is not met, the third target coordinates are updated based on the offset of the movement to obtain the final third target coordinates.

[0238] For example, in combination Fig.16 As shown in (b), Fig.16(b) in FIG. 1 is the area corresponding to the final third target coordinates.

[0239] Exemplarily, the third target coordinate is determined as T uv 1.

[0240] S418: Based on the fourth transformation relationship, the third target coordinates are transformed to obtain the first target coordinates in the optical-mechanical coordinate system.

[0241] For example, the third target coordinate is T uv 1 to P uv 1.

[0242] The following is an exemplary description of how the sensorless correction strategy is used to obtain the first target coordinates.

[0243] In another example, step S41: based on the execution strategy, determining that the specific execution strategy of the first target coordinates in the optical-mechanical coordinate system is a non-sensing correction strategy, that is, when the execution strategy is the non-sensing correction strategy, obtaining the first target coordinates in the optical-mechanical coordinate system includes the following steps:

[0244] S419: When the execution strategy is the sensorless correction strategy, the first optomechanical coordinate is used as the first target coordinate in the optomechanical coordinate system.

[0245] The first optical-mechanical coordinates are the coordinates of the corner points corresponding to the boundary area projected by the optical machine.

[0246] Step S42: Based on the first conversion relationship, the first target coordinates are converted to obtain second target coordinates in the physical coordinate system.

[0247] The first conversion relationship is a mapping relationship between the optical-mechanical coordinate system and the physical coordinate system.

[0248] For example, P uv 1 is converted to obtain the second target coordinate of the physical coordinate system, namely W xyz 1.

[0249] Step S43: Determine the target projection area in the projection plane based on the second target coordinates.

[0250] The target projection area is the range area of ​​the second target coordinates.

[0251] The above technical solution has the following beneficial effects or advantages: by responding to the first instruction, the image of the projection surface is obtained by using the depth sensor, and the image is input into the pre-trained first neural network model for analysis. The model can identify the recognition result on the projection surface, and determine different execution strategies based on the recognition result to define the target projection area. In this way, not only the automation of the projection setting is improved, and the interaction steps with the user are reduced, but also the user's waiting time is significantly reduced by using the model processing, the processing efficiency of obtaining the target projection area is improved, and the user experience is improved.

[0252] Combine the following Fig.17 The method for obtaining the target projection area is described in detail.

[0253] Fig.17 A second method flow chart for obtaining a target projection area is provided in some embodiments of the present application.

[0254] The projection device 200 acquires a first image through a depth sensor;

[0255] The projection device 200 inputs the first image into the first neural network model to obtain a recognition result;

[0256] The projection device 200 determines whether a screen is included based on the recognition result;

[0257] When the projection device 200 includes a screen, a positioning map card is placed in a preset area of ​​the screen on the projection surface;

[0258] The projection device 200 captures the projection surface to obtain a first captured image;

[0259] The projection device 200 crops the first captured image to obtain a second image;

[0260] The projection device 200 inputs the second image into the third neural network model to obtain the coordinates of the four corner points of the screen and the coordinates of the eight corner points of the chart (the first screen coordinates and the second screen coordinates);

[0261] The projection device 200 performs cropping based on the coordinates of the four corner points of the screen and the coordinates of the eight corner points of the image card to obtain a first cropped image and a second cropped image;

[0262] The projection device 200 inputs the first cropped image and the second cropped image into the second neural network model to obtain updated first curtain coordinates and first image card coordinates;

[0263] The projection device 200 determines a homography relationship H2 according to the preset card coordinates and the first card coordinates;

[0264] The projection device 200 converts the first screen image into an optical-mechanical coordinate system based on the homography relationship to obtain a first target coordinate;

[0265] The projection device 200 acquires a target projection area based on the first target coordinates;

[0266] If a screen is not included, the projection device 200 determines whether an obstacle exists;

[0267] If there is no obstacle, the projection device 200 uses the first optical-mechanical coordinate as the first target coordinate;

[0268] The projection device 200 acquires a target projection area based on the first target coordinates;

[0269] If there is an obstacle, the projection device 200 obtains pixels corresponding to the obstacle in the binary image;

[0270] The projection device 200 selects the largest 16:9 rectangle in the binary image;

[0271] It should be noted that the 16:9 rectangle can also be selected in the physical coordinate system, and is not specifically limited here.

[0272] The projection device 200 converts the corner point coordinates corresponding to the 16:9 rectangle into an optical-mechanical coordinate system to obtain a first target coordinate;

[0273] The projection device 200 acquires the target projection area based on the first target coordinates.

[0274] Based on the above-mentioned projection device 200, some embodiments of the present application further provide a method for determining a projection area, including:

[0275] In response to a first instruction, acquiring a first image of the projection surface through the depth sensor;

[0276] The first image is used as an input of a pre-trained first neural network model, and a recognition result is obtained using the first neural network model; the first neural network model is used to identify whether there is a target object on the projection surface;

[0277] Based on the recognition result, determining an execution strategy for acquiring a target projection area, the target projection area being an area where a projection image is displayed; the execution strategy having a corresponding relationship with the recognition result;

[0278] Based on the execution strategy, the target projection area in the projection surface is acquired.

[0279] The above technical solution has the following beneficial effects or advantages: by responding to the first instruction, the image of the projection surface is obtained by using the depth sensor, and the image is input into the pre-trained first neural network model for analysis. The model can identify the recognition result on the projection surface, and determine different execution strategies based on the recognition result to define the target projection area. In this way, not only the automation of the projection setting is improved, and the interaction steps with the user are reduced, but also the user's waiting time is significantly reduced by using the model processing, the processing efficiency of obtaining the target projection area is improved, and the user experience is improved.

[0280] The same and similar parts between the various embodiments in this specification can be referenced to each other and will not be described again here.

[0281] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of various embodiments of the present invention or certain parts of the embodiments.

[0282] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application is described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application. For the convenience of explanation, the above description has been made in conjunction with specific implementation methods. However, the above exemplary discussion is not intended to be exhaustive or to limit the implementation methods to the specific forms disclosed above. According to the above teachings, a variety of modifications and variations can be obtained. The selection and description of the above implementation methods are intended to better explain the principles and practical applications, so that those skilled in the art can better use the implementation methods and various different variations of implementation methods suitable for specific use considerations.

Claims

1. A projection device, characterized in that: include: a depth sensor configured to acquire an image of the projection surface; An optical machine is configured to project media resources; The controller is configured as: In response to a first instruction, acquiring a first image of the projection surface through the depth sensor; The first image is used as an input of a pre-trained first neural network model, and a recognition result is obtained using the first neural network model; the first neural network model is used to identify whether there is a target object on the projection surface; Based on the recognition result, determining an execution strategy for acquiring a target projection area, wherein the target projection area is an area where a projection image is displayed; The execution strategy has a corresponding relationship with the recognition result; Based on the execution strategy, the target projection area in the projection surface is acquired.

2. The projection device according to claim 1, characterized in that: The target object includes at least one of a curtain or an obstacle; the controller determines an execution strategy for acquiring a target projection area based on the recognition result, and the specific configuration is: In the case where the recognition result includes a curtain, determining that the execution strategy for acquiring the target projection area is an entry curtain strategy; In a case where the recognition result does not include the curtain but includes an obstacle, determining that the execution strategy for acquiring the target projection area is an obstacle avoidance strategy; When the recognition result does not include the curtain and the obstacle, it is determined that the execution strategy for acquiring the target projection area is a non-sensing correction strategy.

3. The projection device according to claim 2, characterized in that: The controller executes, based on the execution strategy, to obtain the target projection area in the projection surface, and is specifically configured as follows: Based on the execution strategy, determining first target coordinates in an optical-mechanical coordinate system; the first target coordinates are the coordinates of a corner point of the target projection area in the projection plane in the optical-mechanical coordinate system; Based on a first conversion relationship, the first target coordinates are converted to obtain second target coordinates in a physical coordinate system; the first conversion relationship is a mapping relationship between the optical-mechanical coordinate system and the physical coordinate system; Based on the second target coordinates, the target projection area in the projection plane is acquired.

4. The projection device according to claim 3, characterized in that: The controller executes based on the execution strategy to determine the first target coordinates in the optical-mechanical coordinate system, and the specific configuration is: When the execution strategy is the entry strategy, a second image corresponding to the projection surface is acquired; the second image includes a curtain object and a positioning card object, and the positioning card object is set in a preset area of ​​the curtain object; Acquire the first curtain coordinates corresponding to the curtain object and the first card coordinates corresponding to the positioning card object in the second image; the first curtain coordinates are the corner coordinates corresponding to the curtain object in the camera coordinate system, and the first card coordinates are the corner coordinates corresponding to the positioning card object in the camera coordinate system; An update operation is performed on the first screen coordinates and the first image card coordinates to determine the updated first screen coordinates and the corresponding first weight, and the updated first image card coordinates and the corresponding second weight; wherein the updated first screen coordinates are screen coordinates obtained after the update operation is performed in the second image coordinate system; the updated first screen coordinates are image card coordinates obtained after the update operation is performed in the second image coordinate system, and the update operation includes at least one of moving, scaling, and rotating; Based on the updated first curtain coordinates, the first weight, the updated first image card coordinates, and the second weight, the first target coordinates corresponding to the target projection area in the optical machine coordinate system are determined.

5. The projection device according to claim 4, characterized in that: Also includes: The camera is configured to capture the image of the projection surface; the controller executes acquisition of a second image corresponding to the projection surface, and is specifically configured to: Controlling the camera to obtain a first captured image; the first captured image includes the curtain object and the positioning card object; Based on the second conversion relationship, the first optical-mechanical coordinates are converted into the camera coordinate system to obtain the first camera coordinates; wherein the first optical-mechanical coordinates are the corner point coordinates corresponding to the boundary area projected by the optical machine; and the second conversion relationship is the mapping relationship between the camera coordinate system and the optical-mechanical coordinate system; Based on the first camera coordinates, the first captured image is cropped to obtain the second image; a main body area of ​​the second image is larger than a main body area of ​​the first captured image.

6. The projection device according to claim 4, characterized in that: The controller determines the updated first curtain coordinates and the corresponding first weight, and the updated first image card coordinates and the corresponding second weight based on the first curtain coordinates and the first image card coordinates, and is specifically configured as follows: Based on the first curtain coordinates and the first image card coordinates, the second image is cropped to obtain at least one first cropped image corresponding to the first curtain coordinates and at least one second cropped image corresponding to the first image card coordinates; Using the at least one first cropped image and the at least one second cropped image as inputs of a second neural network model, and using the second neural network model to obtain second curtain coordinates and the corresponding first weights, and second image card coordinates and the corresponding second weights; the second curtain coordinates are curtain coordinates corresponding to the first cropped image, and the second image card coordinates are image card coordinates corresponding to the second cropped image; The second curtain coordinates and the second image card coordinates are converted into the coordinate system of the second image to obtain the updated first curtain coordinates and the updated first curtain coordinates.

7. The projection device according to any one of claims 4 to 6, characterized in that: The controller determines the first target coordinates corresponding to the target projection area in the optical machine coordinate system based on the updated first curtain coordinates, the first weight, the updated first image card coordinates and the second weight, and is specifically configured as follows: Based on the first weights, determining a third curtain coordinate of the second image; the third curtain coordinate is the curtain coordinate with the highest weight value among the first weights; Based on the second weight, determine a third card coordinate of the second image; the third card coordinate is the card coordinate with the highest weight value in the second weight; based on the third card coordinate and the preset card coordinate in the optical-mechanical coordinate system, determine a third conversion relationship; the third conversion relationship is a mapping relationship between the camera coordinate system and the optical-mechanical coordinate system; Based on the third conversion relationship, the third screen coordinates are converted to obtain the first target coordinates corresponding to the target projection area in the optical machine coordinate system.

8. The projection device according to claim 3, characterized in that: The controller executes based on the execution strategy to determine the first target coordinates in the optical-mechanical coordinate system, and the specific configuration is: When the execution strategy is the obstacle avoidance strategy, a binary image corresponding to the first image and a first pixel coordinate corresponding to the binary image are obtained; the first pixel coordinate includes a pixel coordinate corresponding to the obstacle and a pixel coordinate corresponding to the non-obstacle; Based on a fourth conversion relationship, the first optical-mechanical coordinate is converted to obtain a first depth coordinate in a depth sensor coordinate system; the fourth conversion relationship is a mapping relationship between the optical-mechanical coordinate system and the depth sensor coordinate system; Determine, based on the first pixel coordinates and the first depth coordinates, a third target coordinate in the depth sensor coordinate system, wherein the third target coordinate is a corner point coordinate of the target projection area in the projection plane in the depth sensor coordinate system; Based on the fourth conversion relationship, the third target coordinates are converted to obtain the first target coordinates in the optical-mechanical coordinate system.

9. The projection device according to claim 3, characterized in that: The controller executes based on the execution strategy to determine the first target coordinates in the optical-mechanical coordinate system, and the specific configuration is: When the execution strategy is the sensorless correction strategy, the first optomechanical coordinates are used as the first target coordinates in the optomechanical coordinate system; wherein the first optomechanical coordinates are the corner point coordinates corresponding to the boundary area projected by the optomechanical system.

10. A method for determining a projection area, characterized in that: include: In response to a first instruction, acquiring a first image of the projection surface through the depth sensor; The first image is used as an input of a pre-trained first neural network model, and a recognition result is obtained using the first neural network model; the first neural network model is used to identify whether there is a target object on the projection surface; Based on the recognition result, determining an execution strategy for acquiring a target projection area, wherein the target projection area is an area where a projection image is displayed; The execution strategy has a corresponding relationship with the recognition result; Based on the execution strategy, the target projection area in the projection surface is acquired.

Images