Projection device and curtain-in method

Abstract

Embodiments of the present application disclose a projection device and a curtain entering method. The method comprises: a camera configured to capture an image of a projection surface; an optical machine configured to project a media resource; and a controller configured to: in response to a curtain entering instruction, project an image of a positioning card in a preset area of a curtain area by the optical machine; acquire a first image corresponding to the projection surface by the camera; take the first image as an input of a pre-trained first neural network model, and obtain first coordinates of a curtain area object and second coordinates of a positioning card object in the first image by the first neural network model; convert the first coordinates based on a first corresponding relationship to determine third coordinates; and perform a projection operation based on the third coordinates. The method can shorten the display time of the positioning card when the projection device enters the curtain, and the positioning card is displayed only in the preset area, so that the user's viewing experience is not affected in the projection scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of projection equipment, and particularly relates to a projection equipment and a curtain entering method. BACKGROUND

[0002] As a visual display device, projection equipment plays an important role in many fields. With the development of technology, the functions of projection equipment are more intelligent, and the operation becomes simpler and simpler, so that projection equipment gradually enters thousands of households. In order to improve the viewing experience, the curtain entering operation becomes particularly important. The curtain entering refers to adjusting the projection picture, so that the light emitting area of the light engine in the projection equipment can be accurately projected on the curtain area. In this process, the position coordinates of the curtain area in the physical coordinate system need to be converted to the light engine coordinate system, so as to ensure that the projection area of the light engine is accurately connected with the curtain area.

[0003] At present, the curtain entering process of the projection equipment is usually to display a white card and a full-screen chessboard card, focus, and calculate the coordinates of the curtain area in the light engine coordinate system through a traditional image processing algorithm. However, in the whole curtain entering process, the existence time of the card is relatively long, such as 10-20 seconds, which leads to a long waiting time of the user and a poor experience of the user. SUMMARY

[0004] Some embodiments of the present application provide a projection equipment and a curtain entering method, which can set a positioning card in a preset area in the curtain area, obtain the curtain area by using a neural network model, further obtain the coordinates of the curtain area in the light engine coordinate system, so as to shorten the display time of the positioning card, and only set the positioning card in the preset area, which does not affect the viewing experience of the user if the projection scene is being projected.

[0005] In a first aspect, some embodiments of the present application provide a projection equipment, comprising:

[0006] a camera configured to capture an image of a projection surface;

[0007] a light engine configured to project a media resource;

[0008] a controller configured to:

[0009] in response to a curtain entering instruction, project an image of a positioning card in a preset area of a curtain area by the light engine; the image of the positioning card comprises a first card area and a second card area; the first card area is an area surrounded by vertices of the image of the positioning card, and the second card area is arranged in the interior of the first card area;

[0010] acquire a first image corresponding to the projection surface through the camera; the first image includes a curtain area object corresponding to the curtain area and a positioning map card object corresponding to the positioning map card;

[0011] take the first image as an input of a first neural network model pre-trained, and obtain first coordinates of the curtain area object and second coordinates of the positioning map card object in the first image by using the first neural network model; wherein the first neural network model is used to identify corner point coordinates of a curtain and a map card in an image; the first coordinates are corner point coordinates corresponding to the curtain area object; and the second coordinates include corner point coordinates corresponding to the first map card area and corner point coordinates corresponding to the second map card area;

[0012] convert the first coordinates based on a first corresponding relationship to determine third coordinates, the third coordinates being corner point coordinates corresponding to the curtain area object in the light machine; the first corresponding relationship is a corresponding relationship established based on the second coordinates and preset map card coordinates in the light machine to represent a mapping relationship between the camera and the light machine;

[0013] perform a projection operation based on the third coordinates.

[0014] The above technical solution has the following beneficial effects or advantages: in response to a curtain entering instruction, a projection device projects an image of a positioning map card in a preset area of a curtain area, and an image is captured by a camera, a neural network model pre-trained is used to quickly obtain coordinates of the curtain area and the positioning map card, based on these coordinates, a coordinate conversion is performed in combination with a corresponding relationship between a light machine and the camera, corner point coordinates of the curtain area in the light machine can be accurately determined, and then an accurate projection operation is realized. In this way, in the process of obtaining the corner point coordinates of the curtain area in the light machine by using the first neural network model, the map card display time can be shortened, the waiting time of a user can be reduced, and the positioning map card is displayed only in the preset area, which does not affect the viewing experience of the user if a projection scene is being projected.

[0015] In some embodiments, the controller performs, in response to a curtain entering instruction, projecting, by the light machine, an image of a positioning map card in a preset area of the curtain area, and is specifically configured to:

[0016] in response to the curtain entering instruction, acquire a second image corresponding to an initial focal length through the camera; the initial focal length is determined when a projection ratio of the light machine is a first value;

[0017] if it is detected that the second image includes the curtain area object, acquire initial coordinates corresponding to the curtain area object; wherein the initial coordinates are corner point coordinates corresponding to the curtain area object;

[0018] obtain a first focal length based on the initial coordinates corresponding to the curtain area object;

[0019] project an image of a positioning card in a preset area of the curtain area based on the first focal length.

[0020] The above technical solution has the following beneficial effects or advantages: focusing is performed before setting the positioning card, so that the projection device can be in the best imaging state when performing coordinate conversion, thereby avoiding the occurrence of blur or unclear conditions when the card is projected, which helps to improve the accuracy of subsequent coordinate conversion and optimizes the user experience.

[0021] In some embodiments, the controller is configured to, in a case where it is detected that the second image includes the curtain area object, obtain initial coordinates corresponding to the curtain area object in the second image, and is specifically configured to:

[0022] use the second image as an input of a pre-trained second neural network model, use the second neural network model to output a first probability and output coordinates; the first probability is used to measure the probability that the curtain area object is included in the second image;

[0023] In a case where the first probability is greater than a preset probability, it is determined that the second image includes the curtain area object.

[0024] In a case where the second image includes the curtain area object, the output coordinates are used as the initial coordinates corresponding to the curtain area object.

[0025] The above technical solution has the following beneficial effects or advantages: the pre-trained second neural network model can accurately identify the initial coordinates corresponding to the curtain area object and the curtain area, thereby optimizing the efficiency and user experience of the curtain entering process.

[0026] In some embodiments, the controller is configured to, based on the initial coordinates corresponding to the curtain area object, obtain a first focal length, and is specifically configured to:

[0027] convert a first light machine coordinate of the light machine to obtain a first camera coordinate corresponding to the camera based on a second conversion relationship; wherein the first light machine coordinate is a corner point coordinate corresponding to a physical boundary region projected by the light machine under the initial focal length; and the second conversion relationship is an approximate mapping relationship between the light machine and the camera.

[0028] determine a zoom factor based on the initial coordinates of the curtain area object and the first camera coordinate;

[0029] determine the first focal length based on the zoom factor.

[0030] The above technical solution has the following beneficial effects or advantages: by obtaining the most suitable first focal length, the projection ratio can be effectively adjusted to avoid exceeding the screen area due to a too large projection ratio or affecting user experience due to a too small projection ratio. This method ensures that the projection device can automatically adapt to the actual size of the screen area, thereby providing the best projection effect.

[0031] In some embodiments,

[0032] Further comprising: a depth sensor configured to obtain depth information; the controller performs conversion of the first optical engine coordinates of the optical engine based on a second conversion relationship to obtain first camera coordinates corresponding to the camera, specifically configured to:

[0033] determine a first conversion relationship between the optical engine and the projection surface based on first parameter information, the first parameter information including internal parameters of the optical engine, depth information of the depth sensor, and external parameters between the depth sensor and the optical engine;

[0034] determine the second conversion relationship between the optical engine and the camera based on the first conversion relationship and second parameter information, the second parameter information including first and second external parameters between the camera coordinate system and the optical engine coordinate system, and internal parameters of the camera;

[0035] convert the first optical engine coordinates based on the second conversion relationship to obtain the first camera coordinates.

[0036] The above technical solution has the following beneficial effects or advantages: by obtaining accurate depth information through the depth sensor, and combining the internal and external parameters of the optical engine and the camera, the conversion relationship between the optical engine coordinate system and the camera coordinate system can be accurately calculated, the corresponding position of the maximum coordinate in the optical engine in the camera coordinate system can be determined, thereby providing accurate basis for subsequent focal length adjustment, ensuring that the projection device can adapt to the focal length within the maximum range, and improving the accuracy and clarity of the projection effect.

[0037] In some embodiments, the controller determines a zoom factor based on the initial coordinates of the screen area object and the first camera coordinates, specifically configured to:

[0038] if the initial coordinates are within the range corresponding to the first camera coordinates, adjusting the projection ratio to a second value;

[0039] determining a first adjusted focal length corresponding to the second value of the projection ratio;

[0040] determining the first camera coordinates corresponding to the first adjusted focal length;

[0041] if the initial coordinate is in the range corresponding to the first camera coordinate, continue to adjust the projection ratio;

[0042] if the initial coordinate is not in the range corresponding to the first camera coordinate, determine the zoom factor based on the current projection ratio and the preset step value.

[0043] The above technical solution has the following beneficial effects or advantages: by dynamically adjusting the projection ratio according to the relationship between the initial coordinate and the first camera coordinate, it is ensured that the projection picture is always within the field of view of the camera, and the situation of picture deviation or insufficient coverage is avoided, thereby ensuring that the projection device realizes the best projection effect in the maximum range.

[0044] In some embodiments,

[0045] The controller is configured to perform the steps of using the first image as input of a pre-trained second neural network model, using the first neural network model to obtain first coordinates of the curtain area object in the first image and second coordinates of the image of the positioning card, and the specific configuration is:

[0046] using the first image as input of a pre-trained first neural network model, using the first neural network model to obtain first coordinates of the curtain area object in the first image and second coordinates of the positioning card object;

[0047] based on the first coordinates and the second coordinates, cropping the first image to obtain at least one first cropped image corresponding to the first coordinates, and at least one second cropped image corresponding to the second coordinates;

[0048] using the at least one first cropped image and the at least one second cropped image as input of a pre-trained third neural network model, using the third neural network model to obtain fourth coordinates of the first cropped image and fifth coordinates of the second cropped image;

[0049] based on the coordinate system of the first image, converting the fourth coordinates and the fifth coordinates to obtain calibrated first coordinates of the curtain area object and calibrated second coordinates of the positioning card object.

[0050] The above technical solution has the following beneficial effects or advantages: by cropping the image, the key area can be extracted for optimization processing, and the coordinate precision is improved. Using the third neural network model to analyze the cropped image can effectively obtain the first coordinates and the second coordinates of the first image, so as to ensure that the projection device can perform more accurate image projection, and improve the overall projection effect and user experience.

[0051] In some embodiments, before the step of executing the response to the curtain-in instruction, the controller is further configured to: generate the curtain-in instruction in response to detection of movement of the projection device, or in response to a click operation of a user on a curtain-in switch.

[0052] The above technical solution has the following beneficial effects or advantages: whether the projection device is moving or the curtain-in switch is turned on, the curtain-in instruction can be generated in real time before or during projection, effectively capturing and adapting to changes in user behavior, thereby improving user experience.

[0053] In some embodiments, the controller executes the second image as an input to a pre-trained second neural network model, uses the second neural network model to output a first probability and output coordinates; after the step of measuring the probability of the curtain area object included in the second image, the controller is further configured to:

[0054] In the case where the first probability is less than a preset probability, it is determined that the second image does not include the curtain area object; based on the second image, it is determined whether an obstacle avoidance switch is turned on;

[0055] In the case where the obstacle avoidance switch is turned on, based on the second light machine coordinates, projection is performed in a first target area corresponding to the light machine, the first target area being a region in the region corresponding to the second light machine coordinates that does not include an obstacle corresponding region, and the second light machine coordinates being a vertex coordinate corresponding to a physical boundary region projected by the light machine;

[0056] In the case where the obstacle avoidance switch is not turned on, based on the second light machine coordinates, projection is performed in a second target area corresponding to the light machine, the second target area being a region in the region corresponding to the second light machine coordinates that includes an obstacle corresponding region.

[0057] The above technical solution has the following beneficial effects or advantages: by using the camera to capture images and obtaining the operation of the obstacle avoidance switch, the projection area can be intelligently adjusted to improve the accuracy and flexibility of subsequent projection.

[0058] In a second aspect, the embodiments of the present application also provide a curtain-in method, comprising:

[0059] In response to the curtain-in instruction, the light machine projects an image of a positioning card in a preset region of the curtain area; the image of the positioning card includes a first card region and a second card region; the first card region is a region enclosed by a vertex of the image of the positioning card, and the second card region is arranged inside the first card region;

[0060] The camera obtains a first image corresponding to the projection surface; the first image includes a curtain area object corresponding to the curtain area and a positioning card object corresponding to the positioning card.

[0061] taking the first image as an input of a pre-trained first neural network model, and using the first neural network model to obtain first coordinates of the curtain area object and second coordinates of the positioning card object in the first image; wherein the first neural network model is used to identify corner point coordinates of a curtain and a card in an image; the first coordinates are corner point coordinates corresponding to the curtain area object; and the second coordinates include corner point coordinates corresponding to the first card area and corner point coordinates corresponding to the second card area;

[0062] based on the first correspondence relationship, converting the first coordinates to determine third coordinates, the third coordinates being corner point coordinates corresponding to the curtain area object in the light machine; and the first correspondence relationship is a correspondence relationship established based on the second coordinates and preset card coordinates in the light machine, to represent a mapping relationship between the camera and the light machine;

[0063] performing a projection operation based on the third coordinates.

[0064] The above technical solution has the following beneficial effects or advantages: in response to an entering curtain instruction, the projection device projects a positioning card image in a preset area of a curtain area, and captures an image through a camera, and uses a pre-trained neural network model to quickly obtain coordinates of the curtain area and the positioning card. Based on these coordinates, and in combination with a correspondence relationship between the light machine and the camera, coordinate conversion is performed to accurately determine corner point coordinates of the curtain area in the light machine, and thus accurate projection operation is realized. In this way, the display time of the card is shortened and the automatic calibration process is realized, which not only improves the entering curtain efficiency, but also enhances the accuracy of projection to a certain extent, and optimizes the user experience. BRIEF DESCRIPTION OF DRAWINGS

[0065] Figure 1 a projection scene schematic diagram of a projection device provided for some embodiments of the present application;

[0066] Figure 2 a light path schematic diagram of a projection device provided for some embodiments of the present application;

[0067] Figure 3 a circuit architecture schematic diagram provided for some embodiments of the present application;

[0068] Figure 4 a projection device structure schematic diagram provided for some embodiments of the present application;

[0069] Figure 5 a system framework schematic diagram for realizing display control of a projection device provided for some embodiments of the present application;

[0070] Figure 6 a coordinate system conversion relationship schematic diagram provided for some embodiments of the present application;

[0071] Figure 7 A first flowchart of a method for entering a scene according to some embodiments of the present application;

[0072] Figure 8 A flowchart of a method for obtaining a zooming coefficient according to some embodiments of the present application;

[0073] Figure 9 A schematic diagram of a first image according to some embodiments of the present application;

[0074] Figure 10 A flowchart of a method for cropping a first image according to some embodiments of the present application;

[0075] Figure 11 A second flowchart of a method for entering a scene according to some embodiments of the present application;

[0076] Figure 12 A schematic diagram of a projection according to some embodiments of the present application.

[0077] Figure 13 A flowchart of a method for entering a scene according to some embodiments of the present application; DETAILED DESCRIPTION

[0078] For the purpose of clarity, the description of the exemplary embodiments of the present application will be described with reference to the accompanying drawings. It will be obvious, however, to those having ordinary skill in the art that the exemplary embodiments described herein are only a part of the embodiments of the present application and that they are not all-inclusive of all technical concepts of the present application.

[0079] It should be noted that the brief description of the terms in the present application is only for the convenience of understanding the following described embodiments, and is not intended to limit the embodiments of the present application. Unless otherwise specified, these terms should be understood according to their ordinary and general meanings.

[0080] The terms "first", "second", "third", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar or identical objects or entities, and do not necessarily mean a specific order or sequence,

[0081] Unless otherwise noted, it should be understood that the terms used in this way can be interchanged as appropriate.

[0082] Unless otherwise noted, it should be understood that the terms used in this way can be interchanged as appropriate.

[0083] The terms "include" and "have" and any variations thereof, are intended to cover but not exclusively include, for example, a product or device that includes a list of components, and is not necessarily limited to all the components clearly listed, but can include other components not clearly listed or inherent to such products or devices.

[0084] The projection device is a device capable of projecting media data onto a projection medium. The projection device can be connected to a computer, a broadcast network, the Internet, a VCD (Video Compact Disc), a DVD (Digital Versatile Disc Recordable), a game console, a DV, etc. through different interfaces to receive media data to be projected. The media data includes, but is not limited to, images, videos, texts, etc. The projection medium includes, but is not limited to, a wall, a curtain, a screen, etc.

[0085] Figure 1 A projection scene schematic diagram of the projection device provided by some embodiments of the present application is shown.

[0086] In some embodiments, the projection device 100 can be a projector, a laser television, etc. having a projection function. The type of the projection device is not limited in the present application. The projection device is used to project a projection picture onto a projection medium. Taking the projection device as a projector for example, referring to FIG. 1, the projector can include a projection host 2. The projection medium 1 is fixed at a first position, and the projection host 2 is placed at a second position. The relationship between the first position and the second position is adjusted so that the projection picture of the projection host 2 matches the projection medium 1, that is, the second position is the best placement position of the projection host 2. The projection medium can be a curtain, a white wall, etc. Figure 1

[0087] A light path schematic diagram of the projection device provided by some embodiments of the present application is shown. Figure 2 The projection host 2 includes a projection assembly, which includes a light source 210, a light machine 220, and a lens 230. The light source 210 provides illumination for the light machine 220. The light machine 220 modulates the light beam of the light source and outputs it to the lens 230. The lens 230 performs imaging and projects it to the projection medium 1, so that the projection picture is presented by the projection medium 1.

[0088] In some embodiments, the light source 210 can include a bulb assembly or an LED (Light Emitting Diode) light source. The light beam emitted by the light source can be modulated and adjusted by the light machine 220, thereby providing the required light source for the projection picture. In this case, the light machine 220 is usually composed of different optical components, including a color separation device and an adjusting device, to process the light beam emitted by the light source and convert it into the projection picture.

[0089]

[0090] ​In some embodiments, the light engine 220 can further include blue, green and red light engine modules, through which laser or light sources suitable for displaying different colors are generated to achieve clear and accurate projection images. In addition, the light engine 220 is usually equipped with a heat dissipation system and a circuit control system to ensure stable operation of the device.

[0091] In some embodiments, the light emitting components of the projector can be implemented in various ways such as bulbs, LEDs or laser light sources, etc. Different types of light sources determine the brightness, color accuracy and service life of the projector.

[0092] Figure 3 The circuit architecture schematic diagram provided for some embodiments of the present application.

[0093] In some embodiments, referring to Figure 3 , the projection host 2 can include a display control circuit 240, a laser light source 210, at least one laser driver assembly 250 and at least one brightness sensor 260. The laser light source 210 can include at least one laser corresponding to the at least one laser driver assembly one-to-one. Wherein, the at least one refers to one or more, and the plurality refers to two or more than two.

[0094] In some embodiments, the display control circuit 240 is configured to output light control signals corresponding to different primary colors to the laser driver assembly 250 to drive the corresponding laser to emit light, for example, the light control signals include blue light control signals, red light control signals and green light control signals. Referring to Figure 3 , the display control circuit 240 is connected with the laser driver assembly 250, configured to output at least one light control signal corresponding to three primary colors of each frame of image in a plurality of frames of display image, and transmit the at least one light control signal to the corresponding laser driver assembly 250 respectively. For example, the display control circuit 240 can be a microcontroller unit (MCU), also known as a single-chip microcomputer.

[0095] In some embodiments, the projector can achieve adaptive adjustment. For example, by arranging the brightness sensor 260 in the light path of the laser light source 210, the brightness sensor 260 can detect the first brightness value of the laser light source 210 and send the first brightness value to the display control circuit 240. The display control circuit 240 can obtain the second brightness value corresponding to the driving current of each laser, and when the difference between the second brightness value of the laser and the first brightness value of the laser is greater than the difference threshold value, it is determined that the laser has a COD (Catastrophic optical damage, optical catastrophic damage) fault. Then the display control circuit 240 can adjust the current control signal of the laser driver assembly corresponding to the laser until the difference is less than or equal to the difference threshold value, thereby eliminating the COD fault of the laser, reducing the damage rate of the laser, and improving the image display effect of the projection device.

[0096] Figure 4 The projection device structure schematic diagram provided for some embodiments of the present application.

[0097] In some embodiments, referring to Figure 4 The optical path structure includes a laser light source 210 and an optical assembly 214. The laser light source 210 can include independently arranged blue lasers 211, red lasers 212 and green lasers 213. The projection device can also be referred to as a three-color projection device. The blue lasers 211, the red lasers 212 and the green lasers 213 are all Mirai Console Loader (MCL) packaged lasers, which are small in size and conducive to compact arrangement of the optical path.

[0098] In some embodiments, the projection host 2 can include a controller, which includes at least one of a Central Processing Unit (CPU), a video processor, an audio processor, a Graphics Processing Unit (GPU), a RAM (Random Access Memory), a ROM (Read-Only Memory), a first interface to an n-th interface for input / output, a communication bus (Bus), etc. The controller is connected with related hardware of the projection device, such as display control circuit, brightness sensor, distance sensor, image collector, etc., for controlling the functions of the projection device, such as projection, focusing, correction, calibration, on-off screen state adjustment, etc.

[0099] In some embodiments, the projection device (e.g., a laser TV) can be provided with several types of interfaces on the body of the device, such as a power interface, a USB interface, an HDMI (High Definition Multimedia Interface) interface, a network cable interface, a VGA (Video Graphics Array) interface, a DVI (Digital Visual Interface) interface, etc., to connect a signal source for transmitting media.

[0100] In some embodiments, the projection device can directly enter a display interface of a last selected signal source, or a signal source selection interface, after being started, where the signal source can be, for example, a preset video on demand program, or one of signal sources such as an HDMI interface, a USB interface, a live TV interface, etc. After a user selects a target signal source, the projection host 2 can acquire media data from the target signal source, and project the media data on the projection medium 1 for display.

[0101] In some embodiments, the projection host 2 can be configured with a camera for cooperating with the projection host 2 to achieve relevant adjustment and control of the projection process. For example, the projection device can be configured with a 3D camera, a monocular camera, or a binocular camera. When the camera is implemented as a binocular camera, it specifically includes a left camera and a right camera. The binocular camera can acquire an image and a playing content presented by a projection screen corresponding to the projection device, which are projected by a light engine built in the projection device.

[0102] When the projection device is moved to a new position, the projection angle and the distance to the projection screen change, which can cause the projection image to be deformed, and the projection image can be displayed as a trapezoidal image or other distorted image. The controller of the projection device can achieve automatic in-screen based on the image captured by the camera, by coupling the angle between the projection screen and the correct display of the projection image.

[0103] Figure 5 A system framework schematic diagram for implementing display control of the projection device provided in some embodiments of the present application is shown.

[0104] In some embodiments, referring to Figure 5 , the system framework includes an application service layer, a process communication framework, an operation layer, a framework layer, a correction service, a camera service, a time-of-flight service, and hardware and its drivers, etc. The controller of the projection host 2 controls the overall system architecture, and implements projection control of the projection device based on the underlying program logic, including but not limited to automatic in-screen, automatic obstacle avoidance, automatic focusing, anti-eye shooting, on-off screen control, automatic correction and fine-tuning correction of the projection screen, etc.

[0105] In some embodiments, the projection host 2 is also configured with a distance sensor for detecting distance, which can adopt a Time of Flight (TOF) sensor. The TOF sensor measures the distance between nodes by using the time of flight of a signal between the transmitting end and the reflecting end. After the TOF sensor collects distance data, the distance data is sent to a TOF service. After the TOF service obtains the distance data, the collected distance data is sent to an application service layer through a process communication framework, which will be used for data calling of the controller, user interface, program application, and other interactive uses.

[0106] In some embodiments, the projection host 2 can also be configured with an image collector, which can adopt a monocular camera, a binocular camera, a depth camera, or a 3D camera, etc. The image collector sends the collected image data to a camera service, which then sends the image data to a process communication framework and / or a correction service. The process communication framework sends the image data to an application service layer, which will be used for data calling of the controller, user interface, program application, and other interactive uses.

[0107] In some embodiments, the projection device can be refocused after automatic correction is completed. The controller detects whether the automatic focusing function is turned on. If the automatic focusing function is not turned on, the controller ends the automatic focusing service. If the automatic focusing function is turned on, the controller performs focusing calculation according to the distance detection value of the TOF sensor.

[0108] In some embodiments, the controller queries a preset mapping table according to the distance detection value of the TOF sensor, which records the mapping relationship between distance and focal length, to obtain the focal length of the projection device corresponding to the distance detection value. Then, the middleware sends the obtained focal length to the optical engine of the projection device. After the optical engine emits laser according to the above focal length, at least one image collector captures the projection content image. The controller detects the sharpness of the projection content image to determine whether the current lens focal length is appropriate. If the focal length is not appropriate, focusing adjustment needs to be performed. The projection device locates the focusing position with the highest sharpness by adjusting the lens position and capturing, and comparing the sharpness changes of the projection content images before and after adjustment.

[0109] If the determination result meets the preset completion condition, the automatic focusing process is ended. If the determination result does not meet the preset completion condition, the middleware adjusts the focal length parameter of the optical engine of the projection device, for example, gradually adjusts the focal length according to a preset step size, and sets the adjusted focal length parameter to the optical engine again. Through multiple steps of photographing, sharpness evaluation, etc., the optimal focal length is finally locked through the sharpness comparison of the projection image, thereby completing the automatic focusing.

[0110] In some embodiments, in order to improve the viewing experience, the projection device needs to correct the light-emitting assembly when the projection content is first projected to the projection surface or when displacement occurs during projection. This can ensure that the projection content is displayed completely and smoothly on the projection surface. This process is called "entering the curtain", that is, adjusting the projection picture so that the light-emitting area of the light machine can be accurately projected onto the curtain area. In this process, the position coordinates of the curtain area in the physical coordinate system need to be converted to the light machine coordinate system to ensure that the projection area of the light machine accurately connects with the curtain area. When converting the position coordinates of the curtain area in the physical coordinate system to the light machine coordinate system, zooming and "punching the picture card" operations are generally required. The purpose of zooming is to adjust the scale of the image so that the projection image better fits the curtain area, avoiding the image being too small to affect the viewing experience, or being too large to cause light source waste. The "punching the picture card" operation is used to determine the correspondence between the physical coordinate system and the light machine coordinate system, facilitating subsequent coordinate conversion.

[0111] In an embodiment, in combination with Figure 6 As shown, the coordinate system of the camera can serve as an intermediate coordinate system between the physical coordinate system and the light machine coordinate system, helping to complete the coordinate conversion process between the physical coordinate system and the light machine coordinate system. The physical coordinate system generally refers to a coordinate system corresponding to the actual physical environment, and in the scene of the projection device, the specific position of the curtain area in the three-dimensional space can be determined. The camera coordinate system is the coordinate system related to the internal imaging system of the camera, especially the position and direction on the image sensor (such as CMOS or CCD sensor). The light machine coordinate system is the coordinate system related to the internal optical system of the projection device (such as laser projector, projector, etc.), which is used to describe the position and direction of the optical elements (such as light source, light machine, etc.) in the projection device. Exemplarily, the correspondence between different coordinate systems is determined, such as the conversion matrix (for example, homography matrix H2) between the light machine coordinate system and the camera coordinate system, and the conversion matrix between the physical coordinate system and the light machine coordinate system. In this way, the coordinate point in the light machine coordinate system can be calculated by the corresponding coordinates and the homography matrix to obtain its corresponding coordinates on the projection surface, which is the coordinate in the physical coordinate system. Based on the corresponding region, the projection content can be projected based on the corresponding region.

[0112] In one embodiment, the screen-mounting process of the projection device typically follows these conventional steps: When a screen is detected in the projection area, firstly, the projection device projects a white map of the same size as the screen area. The main purpose of this step is to obtain the coordinates of the screen area, providing basic data for subsequent zoom adjustments. Next, the projection device projects a full-screen checkerboard map. The geometric pattern of the checkerboard helps the device more accurately calculate the distortion of the projection area and perform focus adjustments to ensure image clarity. Finally, combining the above information, the projection device calculates the corresponding coordinates of the screen area in the optical-mechanical coordinate system, thereby completing the screen-mounting process and achieving precise alignment between the projected image and the screen area. When no screen is detected in the projection area, firstly, the projection device projects a white map; then, the projection device projects a full-screen checkerboard map and performs focus adjustments; finally, combining the above information, the projection device calculates the corresponding coordinates of the screen area in the optical-mechanical coordinate system, thereby completing the screen-mounting process and achieving precise alignment between the projected image and the screen area. However, during the screen entry process described above, the display time of the image card is as long as 10 to 20 seconds, resulting in a long waiting time for users. Moreover, when the screen entry operation is performed in a scene that is being projected, the full-screen image card will affect the content viewed by the user, resulting in a poor user experience.

[0113] To address the aforementioned problems, this application provides a projection device 200, which can be a long-throw projection device. The projection device 200 includes a camera, an optical engine, and a controller. The camera is configured to capture an image of the projection surface. The optical engine is configured to project media resources. Figure 7 As shown, the controller is configured as follows:

[0114] S100: In response to the screen entry command, the optical engine projects the image of the positioning card into a preset area of ​​the screen area.

[0115] Understandably, the screen entry operation can be pre-adjusted by the user in the menu of the projection device 200. In this way, the projection device 200 can receive the screen entry command during operation and then determine whether to execute the screen entry operation.

[0116] In one embodiment, the projection device 200 may shift due to factors such as collisions or interact with the user by clicking an input screen command during operation. Upon detecting movement of the projection device 200 or responding to a user's input screen switch click, an input screen command is generated.

[0117] After generating the screen entry command, the projection device 200 responds to the screen entry command by projecting the image of the positioning card into a preset area of ​​the screen area through the optical engine.

[0118] In one embodiment, S100 includes the following steps S101-S104:

[0119] Step S101: In response to the entering instruction, a second image corresponding to an initial focal length is acquired by the camera.

[0120] The initial focal length is determined based on a projection ratio (Throw Ratio) of the light machine being a first numerical value. The projection ratio is an important parameter in the projection device 200, used to describe the picture width projected by the projection device 200 at different projection distances. Therefore, the projection ratio can exist in the projection device 200 in the form of a range value. For the sake of simplicity, the projection ratio is taken as an example of R in the following description. Exemplarily, the projection ratio range can be R = 0.9-1.5.

[0121] Exemplarily, the projection ratio can be calculated by using the following formula (1):

[0122]

[0123] Wherein, D is a known projection distance, and W is a known picture width.

[0124] Different projection ratios correspond to different focal lengths. For example, the focal length of the 1.5 projection ratio is F5, and the focal length of the 0.9 projection ratio is F0, wherein F5 is 1.67 times F0.

[0125] Exemplarily, the focal length can be calculated by using the following formula (2):

[0126] f = R * zoom factor formula (2)

[0127] Wherein, f is the focal length.

[0128] Exemplarily, the first numerical value is 0.9, that is, the projection ratio is 0.9. When the projection ratio is 0.9, the initial focal length F0 is calculated based on the formula (1) and the formula (2). Further, the projection device 200 responds to the entering instruction to take a picture at the focal length F0 and acquires the second image.

[0129] Step S102: In the case where it is detected that the second image includes a curtain area object, an initial coordinate corresponding to the curtain area object is acquired.

[0130] Wherein, the initial coordinate is a corner point coordinate corresponding to the curtain area object. The curtain area object refers to the curtain area in the camera coordinate system.

[0131] In an embodiment, step S102 includes steps S1021-S1023.

[0132] Step S1021: The second image is taken as the input of the pre-trained second neural network model, and the second neural network model is used to output the first probability and the output coordinate.

[0133] The first probability is used to measure the existence probability of the curtain area object in the second image. The second neural network model is used to identify the corner point coordinates of the curtain and the poster in the image. In the embodiments of the present application, the second neural network model is used to identify the corner point coordinates of the curtain area object in the first image and locate the corner point coordinates of the poster object.

[0134] The pre-trained neural network model refers to a neural network model that has been trained on a large-scale data set before use. In order to ensure the generalization ability of the neural network model, a large amount of training data needs to be used to train the neural network model.

[0135] In an embodiment, the training of the second neural network model is exemplarily illustrated.

[0136] First, a training set is obtained. The training set includes at least one training image, a target probability, and a target initial coordinate. The training image includes multiple images of a curtain and multiple images not including a curtain.

[0137] Next, the second neural network model is trained by taking the at least one training image as the input of the first neural network model and taking the target probability and the target output coordinate as the output of the second neural network model.

[0138] In one example, the at least one training image is input into the initial neural network model to output a predicted probability and a predicted output coordinate. Further, a preset loss function is used to determine the loss value of the predicted probability and the target probability and the loss value of the predicted initial coordinate and the target output coordinate. When the above loss values are respectively less than the preset loss value, it represents that the initial neural network model is trained at this time, and a trained second neural network model is obtained.

[0139] After the training of the second neural network model is completed, the first probability and the output coordinate are obtained by using the second neural network model.

[0140] Exemplarily, the second image is input into the second neural network model, and the output first probability is 0.6 and the output coordinates are (x1, y1), (x2, y2), (x3, y3), and (x4, y4).

[0141] In step S1022, when the first probability is greater than a preset probability, it is determined that the second image includes a curtain area object.

[0142] Exemplarily, the preset probability can be 0.5.

[0143] Based on the above example, it can be determined that the second image includes a curtain area object based on 0.6 being greater than 0.5.

[0144] Step S1023, in the case where the second image includes the curtain area object, the output coordinates are taken as the initial coordinates corresponding to the curtain area object.

[0145] Following the above example, the initial coordinates corresponding to the curtain area object are determined to be (x1, y1), (x2, y2), (x3, y3), and (x4, y4).

[0146] In order to ensure that the image of the positioning map card is accurately projected into the preset area, a suitable first focal length needs to be selected. The acquisition of the first focal length is specifically described below.

[0147] Step S103: based on the initial coordinates corresponding to the curtain area object, the first focal length is obtained.

[0148] The first focal length is the focal length at which the projection picture and the curtain are completely adapted in proportion, that is, at this focal length, the size proportion of the projection picture and the curtain area is completely matched, ensuring that the projection picture accurately covers the curtain area, neither appearing too large nor appearing too small.

[0149] In an embodiment, the condition for determining the first focal length is to judge whether the projection picture and the curtain area are completely adapted, which can be confirmed by checking whether the coordinates of the curtain area are located on the boundary of the maximum light-emitting area of the light machine.

[0150] The following is an example of determining whether the coordinates of the curtain area are located on the boundary of the maximum light-emitting area of the light machine in the camera coordinate system, and then obtaining the first focal length.

[0151] The steps of obtaining the first focal length include steps S1031-S1033.

[0152] Step S1031: based on the first conversion relationship, the first light machine coordinates of the light machine are converted to obtain the first camera coordinates corresponding to the camera.

[0153] The first light machine coordinates are the corner point coordinates corresponding to the maximum light-emitting area in the light machine coordinate system, that is, the first light machine coordinates are the vertex coordinates corresponding to the physical boundary area projected by the light machine at the initial focal length. The first camera coordinates are the corner point coordinates of the first light machine coordinates in the camera coordinate system.

[0154] In an example, based on the first parameter information, the first conversion relationship between the light machine and the projection surface is determined; based on the first conversion relationship and the second parameter information, the second conversion relationship between the light machine and the camera is determined; based on the second conversion relationship, the first light machine coordinates are converted to obtain the first camera coordinates. The second conversion relationship is an approximate mapping relationship between the light machine and the camera.

[0155] The first parameter information includes internal parameters of the light machine, depth information of the depth sensor, and external parameters between the depth sensor and the light machine. The internal parameters are described below. The internal parameters Mp of the light machine, also known as the internal parameter matrix of the light machine, refer to parameters describing the internal optical system of the light machine (such as the focal length of the light machine, the principal point coordinates, etc.), which are used to determine the mapping relationship from the three-dimensional space to the two-dimensional projection picture. The depth sensor can be a Time of Flight (TOF) sensor, which is a sensor used to obtain depth information, and measures distance based on the time of propagation of light signals between the object and the sensor. It calculates the distance by measuring the time it takes for laser or infrared light to travel from the sensor to the surface of the object, and then generates a depth map of the scene. The depth information of the TOF sensor is usually represented as a depth map. Each pixel in the depth map represents the distance from the sensor to a certain point. The depth map is a grayscale image, where the grayscale value of each pixel represents the distance from the sensor to the surface of the object (the curtain area). The external parameters between the TOF sensor and the light machine include a rotation matrix R P2w and a translation vector T P2w , the rotation matrix R P2w and the translation vector T P2w are the relationship between the light machine coordinate system and the TOF sensor coordinate system. The external parameters are described below.

[0156] The second parameter information includes the internal parameters M c of the camera, the first external parameters R P2c between the camera coordinate system and the light machine coordinate system, and the second external parameters T P2c . The internal parameters Mc of the camera refer to parameters describing the internal optical system of the camera, which are used to convert three-dimensional world coordinates to two-dimensional image coordinates. The internal parameters Mc of the camera usually include the focal length of the camera, the principal point coordinates, etc. The first external parameters R P2c between the camera coordinate system and the light machine coordinate system are a rotation matrix, which describes the rotation relationship between the light machine coordinate system and the camera coordinate system. The first external parameters T P2c between the camera coordinate system and the light machine coordinate system are a translation vector, which describes the translation relationship between the origin of the light machine coordinate system and the origin of the camera coordinate system, representing the relative displacement between the two coordinate systems.

[0157] The projection surface is a region or plane in the physical coordinate system, i.e., a three-dimensional space of the actual existing curtain area. That is, the first conversion relationship between the camera and the projection surface is the conversion relationship between the camera coordinate system and the physical coordinate system.

[0158] Exemplarily, the second conversion relationship can be determined by formula (3):

[0159] C uv = Mc *(R P2w *R P2c ) -1 *(T P2w (2)*R P2w *M P -1 *P uv +T P2c -T P2w ) Formula (3)

[0160] Among them, C uv M is the coordinate of the first camera. c Let R be the intrinsic parameter matrix of the camera. P2w It is the rotation matrix from the optomechanical coordinate system to the physical coordinate system, R P2c It is the rotation matrix from the optical-mechanical coordinate system to the camera coordinate system, T P2w It is the translation vector from the optomechanical coordinate system to the physical coordinate system, T P2c M is the translation vector from the optical-mechanical coordinate system to the camera coordinate system. P It is the intrinsic parameter matrix of the optomechanic, used to describe the internal parameters in the optomechanical coordinate system, P uv It is the first optomechanical coordinate in the optomechanical coordinate system.

[0161] For example, the first optomechanical coordinate P uv 1 converted to C uv 1.

[0162] Step S1032: Determine the zoom factor based on the initial coordinates of the object in the screen area and the first camera coordinates.

[0163] The zoom factor, also known as the zoom ratio, is a parameter in an optical system that indicates the degree to which a lens can magnify or reduce a scene.

[0164] There are many ways to determine the zoom factor; the following example uses an iterative approach.

[0165] In one embodiment, combined with Figure 8 As shown, step S1032 includes steps S10321-S10325.

[0166] Step S10321: If the initial coordinates are within the range corresponding to the first camera coordinates, adjust the projection ratio to the second value based on the preset step value.

[0167] The step value refers to the adjustment made according to the inherent step size, such as increasing or decreasing the projection ratio according to the inherent step size. The value of the step value can be set according to actual needs and is not specifically limited here. For example, 0.1.

[0168] In the above example, the initial projection ratio R is 0.9, and the preset step value is 0.1. The initial coordinates (x1, y1), (x2, y2), (x3, y3), and (x4, y4) are within the ranges corresponding to the first imaging coordinates (x 11 , y 11 ), (x 22 , y 22 ), (x 33 , y 33 ), and (x 44 , y 44 ), respectively. The projection ratio is adjusted to obtain a second value of the projection ratio, i.e., R = 1.

[0169] Step S10322: determining the first adjusted focal length corresponding to the second value of the projection ratio.

[0170] In the above example, the first adjusted focal length F1 corresponding to the second value of the projection ratio, i.e., R = 1, is determined.

[0171] Step S10323: determining the first imaging coordinates corresponding to the first adjusted focal length.

[0172] The first adjusted focal length changes, and the internal parameter Mp of the optical mechanism adapts to the change, which further causes the first imaging coordinates to change. Thus, the first imaging coordinates are (x 12 , y 12 ), (x 23 , y 23 ), (x 34 , y 34 ), and (x 45 , y 45 ).

[0173] Step S10324: if the initial coordinates are within the ranges corresponding to the first imaging coordinates, the projection ratio is continuously adjusted, and step S1 is performed.

[0174] In the above example, it is determined whether the initial coordinates (x1, y1), (x2, y2), (x3, y3), and (x4, y4) are within the ranges corresponding to the first imaging coordinates (x 12 , y 12 ), (x 23 , y 23 ), (x 34 , y 34 ), and (x 45 , y 45 ), respectively. If yes, the projection ratio is continuously updated. For example, it is determined that the projection ratio is a third value, i.e., R = 1.1, and steps S1-S4 are continuously performed. The process is repeated until the initial coordinates are not within the ranges corresponding to the first imaging coordinates, and step S5 is performed.

[0175] Step S10325: determining the zoom factor based on the current projection ratio and the preset step value.

[0176] For example, if the initial coordinates are not within the range corresponding to the first camera coordinates when R = 1.1, the zoom factor is determined to be Step-1 based on the projection ratio and the preset step value. For example, R = 1.1, step = 0.1, and the zoom factor is 1.

[0177] Step S1033: determining the first focal length based on the zoom factor.

[0178] Continuing with the above example, when the zoom factor is 1, the first focal length is determined to be F2.

[0179] Step S104: projecting the image of the alignment chart in the preset area of the curtain area by the optical machine based on the first focal length.

[0180] A preset area is set in the curtain area for projecting the alignment chart. The size of the preset area is not specifically limited, but must be large enough to accommodate all key points of the alignment chart and ensure that the minimum distance requirement is met between these key points. For example, the minimum distance is 1 pixel. For example, if the specified minimum distance is 1 pixel, the coordinate difference between any two corner points (whether horizontal or vertical) must be greater than 1 pixel. At the same time, the preset area must be completely within the range of the curtain area, i.e. not exceeding the actual size of the curtain area.

[0181] It should be noted that the minimum distance is not specifically limited.

[0182] Alignment charts (or calibration targets) are special images or patterns used to calibrate and adjust optical systems, projectors, cameras and other equipment. They usually contain a series of precisely designed geometric shapes, lines, points or other features that help ensure that the equipment can accurately capture or project images, and can be used to measure and correct various types of distortion or error. For example, at the first focal length, the image of the alignment chart is projected in the preset area of the curtain area by the optical machine. For example, the preset area can be the central area of the curtain area.

[0183] For example, the alignment chart is projected in the curtain area on the projection surface, and the alignment chart is set in the preset area.

[0184] S200: obtaining a first image corresponding to the projection surface by a camera.

[0185] The first image includes a curtain area object corresponding to the curtain area and a positioning map card object corresponding to the positioning map card. That is, the curtain area is the area corresponding to the real curtain in the physical coordinate system, and the positioning map card is the map card displayed in the curtain area in the physical coordinate system. The curtain area object is the curtain area in the camera coordinate system, and the positioning map card object is the positioning map card in the camera coordinate system.

[0186] Exemplarily, in combination with Figure 9 As shown, the camera is controlled to capture the curtain area to obtain a first image corresponding to the curtain area. The first image includes a curtain area object 91 and a positioning map card object 92. The positioning map card object 92 includes a first map card area 920 and a second map card area 921. A map card area is an area surrounded by image vertices of a positioning map card, and the second map card area is arranged inside the first map card area.

[0187] S300: Taking the first image as an input of a pre-trained first neural network model, using the first neural network model to obtain first coordinates of the curtain area object and second coordinates of the positioning map card object in the first image.

[0188] Optionally, the training method of the first neural network model can refer to the second neural network model in step S1021 described above, and will not be described here.

[0189] Exemplarily, the first image is input into the pre-trained first neural network model to obtain the first coordinates (x 1’ , y 1’ ), (x 2’ , y 2’ ), (x 3’ , y 3’ ), (x 4’ , y 4’ ) of the curtain area object and the second coordinates including the coordinates (x 80 , y 80 ), (x 81 , y 82 ), (x 83 , y 83 ), (x 84 , y 84 ) of the first map card area and the coordinates (x 85 , y 85 ), (x 86 , y 86 ), (x 87 , y 87 ), (x 88 , y 88 ) of the second map card area of the positioning map card object.

[0190] In order to improve the accuracy of the first coordinates of the curtain area object and the second coordinates of the positioning card object, the acquisition of the first coordinates of the curtain area object and the second coordinates of the positioning card object can be optimized.

[0191] In an embodiment, step S300 can include steps S301-S303.

[0192] Step S301: based on the first coordinates and the second coordinates, the first image is cropped to obtain at least one first cropped image corresponding to the first coordinates, and at least one second cropped image corresponding to the second coordinates.

[0193] Exemplarily, in combination with Figure 10 Exemplary description is made.

[0194] Figure 10 A first image cropping process schematic diagram is provided for some embodiments of the present application.

[0195] As Figure 10 shown, each coordinate point in the first coordinates and the second coordinates is taken as a center point to crop a 96*96 image on the first image to obtain at least one first cropped image and at least one second cropped image.

[0196] Wherein, the ratio of the cropped image can be the above-mentioned 96*96, or other ratios, which are not specifically limited here.

[0197] For example, taking the first coordinates as an example, the first coordinates are (x 1’ , y 1’ ), (x 2’ , y 2’ ), (x 3’ , y 3’ ), (x 4’ , y 4’ ), the above (x 1’ , y 1’ ) is taken as a center point, and (x 1’ -48, y 1’ -48), (x 1’ +48, y 1’ +48) are taken as vertices to crop four first cropped images corresponding to four corner points of the curtain area object, and similarly, taking the second coordinates as an example, eight second cropped images corresponding to eight corner points of the positioning card can be obtained.

[0198] Step S302: taking the at least one first cropped image and the at least one second cropped image as an input of a third neural network model pre-trained, and using the third neural network model to obtain fourth coordinates of the first cropped image and fifth coordinates of the second cropped image.

[0199] The third neural network model is used to identify the corner point coordinates in the image.

[0200] The training method of the third neural network model can refer to step S1021 described above, and will not be described here again.

[0201] According to the above example, the four first cropped images corresponding to the four corner points and the eight second cropped images corresponding to the eight corner points are input into the third neural network model to obtain the fourth coordinates (x 1” , y 1” ), (x 2” , y 2” ), (x 3” , y 3” ), (x 4” , y 4” ) and the fifth coordinates (x 80’ , y 80’ ), (x 81’ , y 82’ ), (x 83’ , y 83’ ), (x 84’ , y 84’ ), (x 85’ , y 85’ ), (x 86’ , y 86’ ), (x 87’ , y 87’ ), (x 88’ , y 88’ ).

[0202] Step S303: based on the coordinate system of the first image, converting the fourth coordinates and the fifth coordinates to obtain the calibrated first coordinates and the calibrated second coordinates of the first image.

[0203] The fourth coordinates and the fifth coordinates are the corner point coordinates of the first cropped image and the second cropped image in the camera coordinate system. The coordinates of the same point in the cropped image and the coordinates in the first image are not the same, so coordinate conversion is needed to obtain the corresponding coordinates of the first image.

[0204] According to the above example, the fourth coordinates (x 1” , y 1” ), (x 2” , y 2” ), (x 3” , y 3” ), (x 4” , y 4” ) and the fifth coordinates (x 80’ , y 80’ ), (x 81’ , y82’ ), (x 83’ , y 83’ ), (x 84’ , y 84’ ), (x 85’ , y 85’ ), (x 86’ , y 86’ ), (x 87’ , y 87’ ), (x 88’ , y 88’ ) are converted to obtain updated first coordinates (x 1”’ , y 1”’ ), (x 2”’ , y 2”’ ), (x 3”’ , y 3”’ ), (x 4”’ , y 4”’ ) in the first image, and updated second coordinates (x 80” , y 80” ), (x 81” , y 82” ), (x 83” , y 83” ), (x 84” , y 84” ), (x 85” , y 85” ), (x 86” , y 86” ), (x 87” , y 87” ), (x 88” , y 88” ) are obtained.

[0205] S400: converting the first coordinates based on the first correspondence relationship to determine third coordinates.

[0206] The first correspondence relationship is a correspondence relationship established based on the second coordinates and the preset card coordinates in the light machine, to represent the accurate mapping relationship between the camera and the light machine. The correspondence relationship can be referred to as a homography matrix H2.

[0207] Exemplarily, the third coordinates in the light machine can be determined based on formula (4):

[0208] P uv = H2 * C uv formula (4)

[0209] wherein P uv is a coordinate in a light machine coordinate system; and C uv is a coordinate in a camera coordinate system.

[0210] For example, the third coordinate in the optical-mechanical coordinate system is determined based on formula (4).

[0211] S500: Perform a projection operation based on the third coordinate.

[0212] In summary, the above method can reduce the display time of the card, reduce the user waiting time, and improve the user experience.

[0213] Corresponding to the above embodiment including the curtain area, there is also an embodiment that does not involve the curtain area.

[0214] The following will be described in conjunction with Figure 11 , as shown in Figure 11 , a second flowchart of an entering curtain method provided by some embodiments of the present application.

[0215] As Figure 11 shown, the entering curtain method includes the following steps:

[0216] S10: In response to an entering curtain instruction, acquire a third image corresponding to an initial focal length through a camera.

[0217] The third image does not include a curtain area object.

[0218] The specific content of step S10 can refer to the similar content of step S101 described above, which will not be repeated here.

[0219] S11: In the case that the third image does not include the curtain area object, determine whether the obstacle avoidance switch is on based on the third image.

[0220] S12: In the case that the obstacle avoidance switch is on, perform projection in a first target area corresponding to the optical machine based on the second optical-mechanical coordinate.

[0221] As shown in (a) of Figure 12 , the first target area is a region in the second optical-mechanical coordinate corresponding region that does not include the obstacle corresponding region, and the second optical-mechanical coordinate is the vertex coordinate corresponding to the region of the optical machine light emitting boundary.

[0222] S13: In the case that the obstacle avoidance switch is not on, perform projection in a second target area corresponding to the optical machine based on the second optical-mechanical coordinate.

[0223] As shown in (b) of Figure 12 , the second target area is a region in the second optical-mechanical coordinate corresponding region that includes the obstacle corresponding region.

[0224] In summary, the curtain is completed by the above-mentioned manner, and the user experience is improved. Based on the display device 200, the application further provides a curtain entering method in some embodiments, which includes the following steps: in response to a curtain entering instruction, projecting an image of a positioning card in a preset area of the curtain area by the light machine; the image of the positioning card includes a first card area and a second card area; the first card area is an area surrounded by the vertex of the image of the positioning card, and the second card area is arranged inside the first card area;

[0225] acquiring a first image corresponding to the curtain area by the camera;

[0226] taking the first image as an input of a pre-trained first neural network model, and obtaining a first coordinate of a curtain area object and a second coordinate of a positioning card object in the first image by the first neural network model; wherein the first coordinate is a corner point coordinate corresponding to the curtain area object, and the second coordinate is a corner point coordinate corresponding to the positioning card object;

[0227] based on a first correspondence relationship, converting the first coordinate to determine a third coordinate, the third coordinate being a corner point coordinate corresponding to the curtain area object in the light machine; the first correspondence relationship is a correspondence relationship between the camera and the light machine based on the second coordinate and a preset card coordinate in the light machine;

[0228] based on the third coordinate, performing a projection operation in the curtain area.

[0229] The above technical solution has the following beneficial effects or advantages: in response to the curtain entering instruction, the projection device 200 projects the image of the positioning card in the preset area of the curtain area, and the camera captures the image, and the pre-trained neural network model is used to quickly obtain the coordinates of the curtain area and the positioning card. Based on these coordinates, the coordinates are converted based on the correspondence relationship between the light machine and the camera, the corner point coordinates of the curtain area in the light machine can be accurately determined, and then the accurate projection operation is realized. In this way, the card display time and the automatic calibration process are shortened, which not only improves the curtain entering efficiency, but also enhances the accuracy of projection to a certain extent, and optimizes the user experience.

[0230] The following will be described in detail Figure 13 The curtain entering method will be described in detail.

[0231] Figure 13 A curtain entering operation flowchart is provided for some embodiments of the application.

[0232] As Figure 13 shown, the curtain entering method includes the following steps:

[0233] The projection device 200 determines whether to generate a curtain entering instruction.

[0234] The projection device 200 adjusts the brightness and obtains the first optical machine coordinates in the case of generating the curtain-in instruction.

[0235] The first optical machine coordinates are vertex coordinates corresponding to a physical boundary region projected by the optical machine, that is, adjusting the brightness means adjusting the brightness of the optical machine to the first brightness based on the curtain-in instruction, and the first brightness is the same as the preset brightness when the projector performs the curtain-in operation.

[0236] The projection device 200 controls the camera to capture the second image at the initial focal length.

[0237] The projection device 200 determines the first probability and the output coordinates corresponding to the second image by using the second neural network model.

[0238] The projection device 200 determines whether the second image has a curtain area.

[0239] If the second image has a curtain area, the projection device 200 calculates the zoom factor.

[0240] The projection device 200 performs the zoom factor operation to obtain the first focal length.

[0241] The projection device 200 shoots the image at the first focal length to obtain the first image.

[0242] The projection device 200 obtains the fourth coordinate and the fifth coordinate in the first image by using the first neural network model.

[0243] The projection device 200 performs 96*96 image cropping with each coordinate point in the fourth coordinate and the fifth coordinate in the first image as the center point.

[0244] The projection device 200 obtains the first coordinate and the second coordinate in the cropped first image by using the third neural network model.

[0245] The projection device 200 determines the homography matrix H2 of the camera and the optical machine based on the second coordinate and the preset card coordinate.

[0246] The projection device 200 determines whether the conversion of the first coordinate to the optical machine coordinate system is successful according to H2 and the first coordinate.

[0247] If the conversion of the first coordinate to the optical machine coordinate system is successful, the curtain-in is completed.

[0248] If the conversion of the first coordinate to the optical machine coordinate system is not successful, it is determined whether the obstacle avoidance switch is opened.

[0249] If the obstacle avoidance switch is opened, the obstacle avoidance is performed, and the projection is performed in the first target area.

[0250] If the obstacle avoidance switch is not opened, project with the second target area.

[0251] If the second image does not have a curtain area, determine whether the obstacle avoidance switch is opened.

[0252] If the obstacle avoidance switch is opened, avoid obstacles and project with the first target area.

[0253] If the obstacle avoidance switch is not opened, project with the second target area.

[0254] The same or similar parts among various embodiments in the specification can be referred to each other, and will not be described here again.

[0255] Those skilled in the art can clearly understand that the technology in the embodiments of the present application can be realized by means of software and necessary general hardware platforms. Based on such understanding, the technical solutions in the embodiments of the present application can be embodied in the form of a software product, which can be stored in a storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods of various embodiments or some parts of the embodiments of the present application.

[0256] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application. In order to facilitate explanation, the above description has been made in combination with specific embodiments. However, the above exemplary discussion is not intended to exhaust or limit the embodiments to the specific forms disclosed above. According to the above teaching, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different modified embodiments suitable for specific use considerations.

Claims

1. A projection device, characterized by The method comprises: a camera configured to capture an image of a projection surface; a light machine configured to project a media resource; a controller configured to: in response to a curtain entering instruction, project an image of a positioning card in a preset area of a curtain area by the light machine; the image of the positioning card comprises a first card area and a second card area; the first card area is an area surrounded by the vertexes of the image of the positioning card, and the second card area is arranged inside the first card area; obtain a first image corresponding to the projection surface by the camera; the first image comprises a curtain area object corresponding to the curtain area and a positioning card object corresponding to the positioning card; use the first image as an input of a pre-trained first neural network model, and obtain first coordinates of the curtain area object and second coordinates of the positioning card object in the first image by the first neural network model; the first neural network model is used to identify the corner point coordinates of the curtain and the card in the image; the first coordinates are corner point coordinates corresponding to the curtain area object; the second coordinates comprise corner point coordinates corresponding to the first card area and corner point coordinates corresponding to the second card area; convert the first coordinates based on a first correspondence relationship to determine third coordinates, which are corner point coordinates corresponding to the curtain area object in the light machine; the first correspondence relationship is a correspondence relationship established based on the second coordinates and preset card coordinates in the light machine to represent the mapping relationship between the camera and the light machine; perform a projection operation based on the third coordinates.

2. The projection device according to claim 1, characterized in that, The controller performing the operation of projecting the image of the positioning card in the preset area of the curtain area by the light machine in response to the curtain entering instruction is specifically configured to: obtain a second image corresponding to an initial focal length by the camera in response to the curtain entering instruction; the initial focal length is determined when the projection ratio of the light machine is a first value; obtain initial coordinates corresponding to the curtain area object when it is detected that the second image comprises the curtain area object; the initial coordinates are corner point coordinates corresponding to the curtain area object; obtain a first focal length based on the initial coordinates corresponding to the curtain area object; project the image of the positioning card in the preset area of the curtain area by the light machine based on the first focal length.

3. The projection device according to claim 2, characterized in that, The controller performing the operation of obtaining the initial coordinates corresponding to the curtain area object in the second image when it is detected that the second image comprises the curtain area object is specifically configured to: use the second image as an input of a pre-trained second neural network model, and output a first probability and output coordinates by the second neural network model; the first probability is used to measure the probability that the second image comprises the curtain area object; determine that the second image comprises the curtain area object when the first probability is greater than a preset probability; use the output coordinates as the initial coordinates corresponding to the curtain area object when the second image comprises the curtain area object.

4. The projection device according to claim 2 or 3, characterized in that, The controller is configured to obtain a first focal length based on initial coordinates of the curtain area object, and configured to: convert the first light machine coordinates of the light machine based on a second conversion relationship to obtain first camera coordinates corresponding to the camera; wherein the first light machine coordinates are corner point coordinates corresponding to a boundary region projected by the light machine at the initial focal length; the second conversion relationship is an approximate mapping relationship between the light machine and the camera; determine a zoom factor based on the initial coordinates of the curtain area object and the first camera coordinates; determine the first focal length based on the zoom factor.

5. The projection apparatus according to claim 4, wherein, Further comprising: a depth sensor configured to obtain depth information; the controller is configured to convert the first light machine coordinates of the light machine based on a second conversion relationship to obtain first camera coordinates corresponding to the camera, and is specifically configured to: determine a first conversion relationship between the light machine and the projection surface based on first parameter information, wherein the first parameter information includes internal parameters of the light machine, depth information of the depth sensor, and external parameters between the depth sensor and the light machine; determine the second conversion relationship between the light machine and the camera based on the first conversion relationship and second parameter information, wherein the second parameter information includes first and second external parameters between the camera coordinate system and the light machine coordinate system, and internal parameters of the camera; convert the first light machine coordinates based on the second conversion relationship to obtain the first camera coordinates.

6. The projection apparatus according to claim 4, wherein, The controller is configured to determine a zoom factor based on the initial coordinates of the curtain area object and the first camera coordinates, and is specifically configured to: if the initial coordinates are within the range corresponding to the first camera coordinates, adjust the projection ratio to a second value based on a preset step value; determine a first adjusted focal length corresponding to the second value of the projection ratio; determine the first camera coordinates corresponding to the first adjusted focal length; if the initial coordinates are within the range corresponding to the first camera coordinates, continue to adjust the projection ratio; if the initial coordinates are not within the range corresponding to the first camera coordinates, determine the zoom factor based on the current projection ratio and the preset step value.

7. The projection apparatus according to claim 1, 2 or 3, wherein, The controller is configured to use the first image as input of a pre-trained second neural network model, and use the first neural network model to obtain first coordinates of the curtain area object in the first image and second coordinates of the image of the positioning map card, and is specifically configured to: use the first image as input of a pre-trained first neural network model, and use the first neural network model to obtain first coordinates of the curtain area object in the first image and second coordinates of the positioning map object; crop the first image based on the first coordinates and the second coordinates to obtain at least one first cropped image corresponding to the first coordinates, and at least one second cropped image corresponding to the second coordinates; inputting the at least one first cropped image and the at least one second cropped image into a third neural network model pre-trained, and obtaining fourth coordinates of the first cropped image and fifth coordinates of the second cropped image by using the third neural network model; converting the fourth coordinates and the fifth coordinates based on a coordinate system of the first image to obtain calibrated first coordinates of the curtain area object and calibrated second coordinates of the positioning card object.

8. The projection apparatus according to any one of claims 1-3, wherein, Before the controller performs the step of responding to the curtain-in instruction, the controller is further configured to: generate the curtain-in instruction in response to detection of movement of the projection device or in response to a click operation of a user on a curtain-in switch.

9. The projection apparatus according to claim 3, wherein, After the controller performs the step of inputting the second image into a second neural network model pre-trained, and outputting first probabilities and output coordinates by using the second neural network model, the controller is further configured to: determine that the second image does not include the curtain area object in a case where the first probabilities are less than a preset probability; and determine whether an obstacle avoidance switch is turned on based on the second image; in a case where the obstacle avoidance switch is turned on, project a first target area corresponding to the light machine based on second light machine coordinates, the first target area being a region corresponding to the second light machine coordinates and not including a region corresponding to an obstacle, and the second light machine coordinates being vertex coordinates corresponding to a physical boundary region projected by the light machine; in a case where the obstacle avoidance switch is not turned on, project a second target area corresponding to the light machine based on the second light machine coordinates, the second target area being a region corresponding to the second light machine coordinates and including a region corresponding to an obstacle.

10. A method of projecting an image into a scene, characterized by comprising: in response to the curtain-in instruction, projecting an image of a positioning card in a preset region of a curtain area by a light machine; the image of the positioning card including a first card region and a second card region; the first card region being a region surrounded by vertexes of the image of the positioning card, and the second card region being arranged inside the first card region; obtaining a first image corresponding to a projection surface by a camera; the first image including a curtain area object corresponding to the curtain area and a positioning card object corresponding to the positioning card; inputting the first image into a first neural network model pre-trained, and obtaining first coordinates of the curtain area object and second coordinates of the positioning card object in the first image by using the first neural network model; the first neural network model being used for identifying corner point coordinates of a curtain and a card in an image; the first coordinates being corner point coordinates corresponding to the curtain area object; and the second coordinates including corner point coordinates corresponding to the first card region and corner point coordinates corresponding to the second card region. The first coordinates are converted based on a first correspondence relationship to determine third coordinates, the third coordinates being corner point coordinates corresponding to the curtain area object in the light machine; the first correspondence relationship is a correspondence relationship established based on the second coordinates and preset card coordinates in the light machine to represent a mapping relationship between the camera and the light machine; A projection operation is performed based on the third coordinates.

Images