Projection equipment and screen entering method

Abstract

The embodiment of the invention discloses projection equipment and a screen entering method, and the method comprises a camera which is configured to be used for shooting an image of a projection surface; the optical machine is configured to be used for projecting the media asset resources; the controller is configured to: in response to the screen entering instruction, project an image of the positioning graph card in a preset area of the screen area through the ray machine; acquiring a first image corresponding to the projection surface through the camera; taking the first image as the input of a pre-trained first neural network model, and utilizing the first neural network model to obtain a first coordinate of a curtain area object in the first image and a second coordinate of a positioning graph card object; based on the first corresponding relation, converting the first coordinate, and determining a third coordinate; and performing projection operation based on the third coordinate. According to the method, the display time of the positioning graph card can be shortened when the projection equipment enters the screen, the positioning graph card is only printed in the preset area, and the watching experience of a user is not affected in a projection scene.

Description

Technical Field

[0001] The present application relates to the technical field of projection equipment, and in particular to a projection equipment and a projection screen method. Background Art

[0002] As a visual display device, projection equipment plays an important role in many fields. With the development of technology, the functions of projection equipment have become more intelligent and the operation has become simpler, making projection equipment gradually enter thousands of households. In order to improve the viewing experience, it is particularly important to perform the screen entry operation. Screen entry refers to adjusting the projection picture so that the light output area of ​​the optical machine in the projection device can be accurately projected onto the screen area. In this process, the position coordinates of the screen area in the physical coordinate system need to be converted to the optical machine coordinate system to ensure that the projection area of ​​the optical machine is accurately connected to the screen area.

[0003] At present, the entry process of projection equipment is usually through displaying white image cards and full-screen checkerboard image cards, adjusting the focus, and calculating the corresponding coordinates of the screen area in the optical machine coordinate system through traditional image processing algorithms. However, during the entire entry process, the image card exists for a long time, such as 10-20 seconds, resulting in a long waiting time for users and a poor user experience. Summary of the invention

[0004] Some embodiments of the present application provide a projection device and a method for entering a screen. A positioning map card can be set in a preset area in the screen area, the screen area can be acquired using a neural network model, and the coordinates of the screen area in the optical-mechanical coordinate system can be further acquired, thereby shortening the display time of the positioning map card and only placing the positioning map card in the preset area. If the projection scene is in progress, the user's viewing experience will not be affected.

[0005] In a first aspect, some embodiments of the present application provide a projection device, including:

[0006] A camera configured to capture an image of the projection surface;

[0007] An optical machine is configured to project media resources;

[0008] The controller is configured as:

[0009] In response to the screen entry instruction, the image of the positioning card is projected in a preset area of ​​the screen area by the optical machine; the image of the positioning card includes a first card area and a second card area; the first card area is an area surrounded by the vertices of the image of the positioning card, and the second card area is set inside the first card area;

[0010] Acquire a first image corresponding to the projection surface through the camera; the first image includes a curtain area object corresponding to the curtain area and a positioning card object corresponding to the positioning card;

[0011] The first image is used as an input of a pre-trained first neural network model, and the first neural network model is used to obtain the first coordinates of the curtain area object and the second coordinates of the positioning card object in the first image; wherein the first neural network model is used to identify the corner coordinates of the curtain and the card in the image; the first coordinates are the corner coordinates corresponding to the curtain area object; the second coordinates include the corner coordinates corresponding to the first card area and the corner coordinates corresponding to the second card area;

[0012] Based on the first corresponding relationship, the first coordinate is transformed to determine a third coordinate, where the third coordinate is the corner point coordinate corresponding to the curtain area object in the optical machine; the first corresponding relationship is a corresponding relationship established based on the second coordinate and the preset image card coordinate in the optical machine to characterize the mapping relationship between the camera and the optical machine;

[0013] A projection operation is performed based on the third coordinate.

[0014] The above technical solution has the following beneficial effects or advantages: in response to the screen entry instruction, the projection device projects the positioning card image in the preset area of ​​the screen area, and captures the image through the camera, and uses the pre-trained neural network model to quickly obtain the coordinates of the screen area and the positioning card. Based on these coordinates, combined with the corresponding relationship between the optical machine and the camera, coordinate conversion is performed, and the coordinates of the corner points of the screen area in the optical machine can be accurately determined, thereby achieving accurate projection operations. In this way, in the process of using the first neural network model to obtain the coordinates of the corner points of the screen area in the optical machine, the image card display time can be shortened, the user's waiting time can be reduced, and the positioning card is only placed in the preset area. If the projection scene is in progress, it will not affect the user's viewing experience.

[0015] In some embodiments, the controller executes, in response to the screen entry instruction, projecting the image of the positioning chart card in a preset area of ​​the screen area through the optical machine, and is specifically configured as follows:

[0016] In response to the screen entry instruction, a second image corresponding to an initial focal length is acquired through the camera; the initial focal length is determined when the projection ratio of the optical machine is a first value;

[0017] In the case where it is detected that the second image includes the curtain area object, initial coordinates corresponding to the curtain area object are acquired; wherein the initial coordinates are the coordinates of the corner points corresponding to the curtain area object;

[0018] Based on the initial coordinates corresponding to the curtain area object, obtaining a first focal length;

[0019] Based on the first focal length, the image of the positioning chart is projected into a preset area of ​​the screen area by the optical machine.

[0020] The above technical solution has the following beneficial effects or advantages: by focusing before setting the positioning chart, the projection device can be in the best imaging state when performing coordinate conversion, thereby avoiding blurry or unclear images when projecting the chart, which helps to improve the accuracy of subsequent coordinate conversions and optimize the user experience.

[0021] In some embodiments, the controller executes, when detecting that the second image includes the curtain area object, obtaining the initial coordinates corresponding to the curtain area object in the second image, and is specifically configured as follows:

[0022] The second image is used as an input of a pre-trained second neural network model, and the second neural network model is used to output a first probability and an output coordinate; the first generalization is used to measure the probability that the second image includes the curtain area object;

[0023] When the first probability is greater than a preset probability, determining that the second image includes the curtain area object;

[0024] In the case where the second image includes the curtain area object, the output coordinates are used as the initial coordinates corresponding to the curtain area object.

[0025] The above technical solution has the following beneficial effects or advantages: the pre-trained second neural network model can accurately identify the curtain area object and the initial coordinates corresponding to the curtain area, thereby optimizing the efficiency of the curtain entry process and the user experience.

[0026] In some embodiments, the controller executes acquiring the first focal length based on the initial coordinates corresponding to the curtain area object, and is specifically configured as follows:

[0027] Based on the second conversion relationship, the first optical-mechanical coordinates of the optical machine are converted to obtain the first camera coordinates corresponding to the camera; wherein the first optical-mechanical coordinates are the corner point coordinates corresponding to the physical boundary area projected by the optical machine at the initial focal length; and the second conversion relationship is an approximate mapping relationship between the optical machine and the camera;

[0028] Determine a zoom factor based on the initial coordinates of the curtain area object and the first camera coordinates;

[0029] Based on the zoom factor, the first focal length is determined.

[0030] The above technical solution has the following beneficial effects or advantages: by obtaining the most suitable first focal length, the projection ratio can be effectively adjusted to avoid exceeding the screen area due to the projection ratio being too large, or affecting the user experience due to the ratio being too small. This method ensures that the projection device can automatically adapt according to the actual size of the screen area, thereby providing the best projection effect.

[0031] In some embodiments,

[0032] It also includes: a depth sensor configured to obtain depth information; the controller performs a conversion based on a second conversion relationship on the first optical machine coordinate of the optical machine to obtain a first camera coordinate corresponding to the camera, which is specifically configured as follows:

[0033] Determine a first conversion relationship between the optical machine and the projection surface based on first parameter information, where the first parameter information includes internal parameters of the optical machine, depth information of the depth sensor, and external parameters between the depth sensor and the optical machine;

[0034] Determine the second conversion relationship between the optical machine and the camera based on the first conversion relationship and the second parameter information, wherein the second parameter information includes the first external parameter and the second external parameter between the camera coordinate system and the optical machine coordinate system, and the internal parameters of the camera;

[0035] Based on the second conversion relationship, the first optical-mechanical coordinates are converted to obtain the first camera coordinates.

[0036] The above technical solution has the following beneficial effects or advantages: by obtaining accurate depth information through the depth sensor, and combining the internal and external parameters of the optical machine and the camera, the conversion relationship between the optical machine coordinate system and the camera coordinate system can be accurately calculated, and the corresponding position of the maximum coordinate in the optical machine in the camera coordinate system can be determined, thereby providing an accurate basis for subsequent focal length adjustment, ensuring that the projection equipment can adapt the focal length within the maximum range, and improving the accuracy and clarity of the projection effect.

[0037] In some embodiments, the controller determines the zoom factor based on the initial coordinates of the curtain area object and the first camera coordinates, and is specifically configured as follows:

[0038] If the initial coordinates are within the range corresponding to the first camera coordinates, adjusting the projection ratio to a second value;

[0039] Determine a first adjustment focal length corresponding to when the projection ratio is the second value;

[0040] Determining the first camera coordinates corresponding to the first adjusted focal length;

[0041] If the initial coordinates are within the range corresponding to the first camera coordinates, continue adjusting the projection ratio;

[0042] When the initial coordinates are not within the range corresponding to the first camera coordinates, the zoom factor is determined based on the current projection ratio and the preset step value.

[0043] The above technical solution has the following beneficial effects or advantages: by dynamically adjusting the projection ratio according to the relationship between the initial coordinates and the first camera coordinates, the projected image is ensured to be always within the field of view of the camera, avoiding image offset or insufficient coverage, thereby ensuring that the projection device achieves the best projection effect within the maximum range.

[0044] In some embodiments,

[0045] The controller executes the step of using the first image as an input of a pre-trained second neural network model and obtaining the first coordinates of the curtain area object in the first image and the second coordinates of the image of the positioning map card by using the first neural network model, which is specifically configured as follows:

[0046] Using the first image as an input of a pre-trained first neural network model, and using the first neural network model, obtaining first coordinates of a curtain area object and second coordinates of a positioning card object in the first image;

[0047] Based on the first coordinates and the second coordinates, the first image is cropped to obtain at least one first cropped image corresponding to the first coordinates, and at least one second cropped image corresponding to the second coordinates;

[0048] Using the at least one first cropped image and the at least one second cropped image as inputs of a pre-trained third neural network model, and using the third neural network model to obtain a fourth coordinate of the first cropped image and a fifth coordinate of the second cropped image;

[0049] Based on the coordinate system of the first image, the fourth coordinate and the fifth coordinate are converted to obtain the calibrated first coordinate of the curtain area object and the calibrated second coordinate of the positioning card object.

[0050] The above technical solution has the following beneficial effects or advantages: by cropping the image, the key area can be extracted for optimization processing to improve the coordinate accuracy. By analyzing the cropped image using the third neural network model, the first coordinate and the second coordinate of the first image can be effectively obtained, thereby ensuring that the projection device can perform more accurate image projection and improve the overall projection effect and user experience.

[0051] In some embodiments, before executing the step of responding to the entry-screen instruction, the controller is further configured to: generate an entry-screen instruction upon detecting movement of the projection device or in response to a user clicking an entry-screen switch.

[0052] The above technical solution has the following beneficial effects or advantages: whether the projection device is moving or the entry-screen switch is turned on, the entry-screen instruction can be generated in real time before or during projection, which can effectively capture and adapt to user behavior changes, thereby improving the user experience.

[0053] In some embodiments, the controller executes the step of taking the second image as the input of a pre-trained second neural network model, and outputting a first probability and an output coordinate using the second neural network model; after the step of the first generalization being used to measure the probability that the second image includes the curtain area object, the controller is further configured to:

[0054] When the first probability is less than a preset probability, determining that the second image does not include the curtain area object; and determining whether an obstacle avoidance switch is turned on based on the second image;

[0055] When the obstacle avoidance switch is turned on, projection is performed on a first target area corresponding to the optical machine based on the second optical machine coordinates, where the first target area is an area not including obstacles within the area corresponding to the second optical machine coordinates, and the second optical machine coordinates are vertex coordinates corresponding to the physical boundary area projected by the optical machine;

[0056] When the obstacle avoidance switch is not turned on, based on the second optical machine coordinates, in the second target area corresponding to the optical machine, the second target area is an area corresponding to the obstacle within the area corresponding to the second optical machine coordinates.

[0057] The above technical solution has the following beneficial effects or advantages: by using the camera to capture images and obtaining the operation of the obstacle avoidance switch, the projection area can be intelligently adjusted to improve the accuracy and flexibility of subsequent projections.

[0058] In a second aspect, the embodiment of the present application further provides a method for entering a screen, comprising:

[0059] In response to the screen entry instruction, the image of the positioning card is projected in a preset area of ​​the screen area by the optical machine; the image of the positioning card includes a first card area and a second card area; the first card area is an area surrounded by the vertices of the image of the positioning card, and the second card area is set inside the first card area;

[0060] Acquire a first image corresponding to the projection surface through the camera; the first image includes a curtain area object corresponding to the curtain area and a positioning card object corresponding to the positioning card;

[0061] The first image is used as an input of a pre-trained first neural network model, and the first neural network model is used to obtain the first coordinates of the curtain area object and the second coordinates of the positioning card object in the first image; wherein the first neural network model is used to identify the corner coordinates of the curtain and the card in the image; the first coordinates are the corner coordinates corresponding to the curtain area object; the second coordinates include the corner coordinates corresponding to the first card area and the corner coordinates corresponding to the second card area;

[0062] Based on the first corresponding relationship, the first coordinate is transformed to determine a third coordinate, where the third coordinate is the corner point coordinate corresponding to the curtain area object in the optical machine; the first corresponding relationship is a corresponding relationship established based on the second coordinate and the preset image card coordinate in the optical machine to characterize the mapping relationship between the camera and the optical machine;

[0063] A projection operation is performed based on the third coordinate.

[0064] The above technical solution has the following beneficial effects or advantages: in response to the screen entry instruction, the projection device projects the positioning card image in the preset area of ​​the screen area, and captures the image through the camera, and uses the pre-trained neural network model to quickly obtain the coordinates of the screen area and the positioning card. Based on these coordinates, combined with the corresponding relationship between the optical machine and the camera, coordinate conversion can be performed to accurately determine the coordinates of the corner points of the screen area in the optical machine, thereby achieving accurate projection operation. In this way, the display time of the card and the automated calibration process are shortened, which not only improves the efficiency of screen entry, but also enhances the accuracy of projection to a certain extent and optimizes the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 A schematic diagram of a projection scene of a projection device provided in some embodiments of the present application;

[0066] Figure 2 A schematic diagram of the optical path of a projection device provided in some embodiments of the present application;

[0067] Figure 3 A schematic diagram of a circuit architecture provided for some embodiments of the present application;

[0068] Figure 4 A schematic diagram of the structure of a projection device provided in some embodiments of the present application;

[0069] Figure 5 A schematic diagram of a system framework for implementing display control for a projection device provided in some embodiments of the present application;

[0070] Figure 6 A schematic diagram of a coordinate system conversion relationship provided in some embodiments of the present application;

[0071] Figure 7 A first flow chart of a curtain entry method provided for some embodiments of the present application;

[0072] Figure 8 A schematic flow chart of a method for obtaining a zoom factor provided in some embodiments of the present application;

[0073] Fig. 9 A schematic diagram of a first image provided in some embodiments of the present application;

[0074] Fig.10 A schematic diagram of a first image cropping process provided in some embodiments of the present application;

[0075] Fig.11 A second flow chart of a curtain entry method provided in some embodiments of the present application;

[0076] Fig.12 A projection schematic diagram provided for some embodiments of the present application.

[0077] Fig.13 A schematic diagram of the entry operation process provided for some embodiments of the present application. DETAILED DESCRIPTION

[0078] In order to make the purpose and implementation method of the present application clearer, the exemplary implementation method of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0079] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their ordinary and common meanings.

[0080] The terms "first", "second", and "third" in the specification, claims, and drawings of this application are used interchangeably.

[0081] The term "etc." is used to distinguish between similar or similar objects or entities and does not necessarily imply a specific order or sequence.

[0082] Unless otherwise indicated, it is to be understood that the terms so used are interchangeable under appropriate circumstances.

[0083] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0084] Projection equipment is a device that can project media data onto a projection medium. Projection equipment can be connected to computers, broadcasting networks, the Internet, VCD (Video Compact Disc), DVD (Digital Versatile Disc Recordable), game consoles, DV and other devices through different interfaces to receive media data to be projected. Among them, media data includes but is not limited to images, videos, texts and other types, and projection media includes but is not limited to physical forms such as walls, curtains, and screens.

[0085] Figure 1 A schematic diagram of a projection scene of a projection device provided in some embodiments of the present application.

[0086] In some embodiments, the projection device 100 may be a device with a projection function such as a projector or a laser TV. The present application does not limit the type of the projection device. The projection device is used to project a projection image onto a projection medium. For example, the projection device is a projector. Figure 1 The projector may include a projection host 2. The projection medium 1 is fixed at a first position, and the projection host 2 is placed at a second position. By adjusting the relationship between the first position and the second position, the projection image of the projection host 2 is made to match the projection medium 1, that is, the second position is the best placement position of the projection host 2. The projection medium may be a screen, a white wall, or the like.

[0087] Figure 2 A schematic diagram of the optical path of a projection device provided in some embodiments of the present application.

[0088] The projection host 2 includes a projection component, which includes a light source 210, an optical machine 220 and a lens 230. The light source 210 provides illumination for the optical machine 220, the optical machine 220 modulates the light beam of the light source and outputs it to the lens 230, the lens 230 forms an image and projects it to the projection medium 1, and the projection medium 1 presents a projection picture.

[0089] In some embodiments, the light source 210 may include a bulb assembly or an LED (Light Emitting Diode) light source, and the light beam emitted by the light source may be modulated and adjusted by the optical engine 220 to provide the required light source for the projection image. In this case, the optical engine 220 is usually composed of different optical components, including a color separation device and an adjustment device, to process the light beam emitted by the light source and convert it into a projection image.

[0090] In some embodiments, the optical engine 220 may also include blue, green and red optical engine modules, through which lasers or light sources suitable for displaying different colors are generated to achieve clear and accurate projection images. In addition, the optical engine 220 is usually equipped with a heat dissipation system and a circuit control system to ensure stable operation of the device.

[0091] In some embodiments, the light-emitting component of the projector can be realized in a variety of ways such as a bulb, LED or laser light source. Different light source types determine the brightness, color accuracy and service life of the projector.

[0092] Figure 3 A schematic diagram of a circuit architecture provided for some embodiments of the present application.

[0093] In some embodiments, see Figure 3 The projection host 2 may include a display control circuit 240, a laser light source 210, at least one laser driving component 250, and at least one brightness sensor 260. The laser light source 210 may include at least one laser corresponding to the at least one laser driving component. The at least one refers to one or more, and the multiple refers to two or more.

[0094] In some embodiments, the display control circuit 240 is used to output light control signals corresponding to different primary colors to the laser driving component 250 to drive the corresponding laser to emit light. For example, the light control signal includes a blue light control signal, a red light control signal, and a green light control signal. Figure 3 The display control circuit 240 is connected to the laser driving component 250, and is used to output at least one light control signal corresponding to the three primary colors of each frame of the multi-frame display image, and transmit the at least one light control signal to the corresponding laser driving component 250. For example, the display control circuit 240 can be a microcontroller unit (MCU), also known as a single chip microcomputer.

[0095] In some embodiments, the projector can achieve adaptive adjustment. For example, by setting a brightness sensor 260 in the light output path of the laser light source 210, the brightness sensor 260 can detect the first brightness value of the laser light source 210 and send the first brightness value to the display control circuit 240. The display control circuit 240 can obtain the second brightness value corresponding to the driving current of each laser, and when the difference between the second brightness value of the laser and the first brightness value of the laser is greater than the difference threshold, it is determined that the laser has a COD (Catastrophic optical damage) failure, and the display control circuit 240 can adjust the current control signal of the laser driving component corresponding to the laser until the aforementioned difference is less than or equal to the difference threshold, thereby eliminating the COD failure of the laser, reducing the damage rate of the laser, and improving the image display effect of the projection device.

[0096] Figure 4 A schematic diagram of the structure of a projection device provided in some embodiments of the present application.

[0097] In some embodiments, see Figure 4 The optical path structure includes a laser light source 210 and an optical component 214. The laser light source 210 may include an independently arranged blue laser 211, a red laser 212, and a green laser 213. The projection device may also be referred to as a three-color projection device. The blue laser 211, the red laser 212, and the green laser 213 are all module lightweight (Mirai Console Loader, MCL) packaged lasers with a small size, which is conducive to the compact arrangement of the optical path.

[0098] In some embodiments, the projection host 2 may include a controller, which includes at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), a RAM Random Access Memory (RAM), a ROM (Read-Only Memory, ROM), a first interface to an nth interface for input / output, and a communication bus (Bus), etc. The controller is connected to the relevant hardware of the projection device, such as the display control circuit, the brightness sensor, the distance sensor, the image collector, etc., and is used to control the projection, focusing, correction, calibration, switch screen state adjustment, etc. of the projection device.

[0099] In some embodiments, several types of interfaces may be provided on the body of the projection device (e.g., a laser TV), such as a power interface, a USB interface, an HDMI (High Definition Multimedia Interface) interface, a network cable interface, a VGA (Video Graphics Array) interface, a DVI (Digital Visual Interface), etc., to connect a signal source for transmitting media.

[0100] In some embodiments, after the projection device is started, it can directly enter the display interface of the signal source selected last time, or the signal source selection interface, where the signal source is, for example, a preset video-on-demand program, or can be one of the signal sources such as an HDMI interface, a USB interface, a live TV interface, etc. After the user selects the target signal source, the projection host 2 can obtain media data from the target signal source and project the media data onto the projection medium 1 for display.

[0101] In some embodiments, the projection host 2 may be configured with a camera for cooperating with the projection host 2 to achieve relevant adjustment and control of the projection process. For example, the projection device may be configured with a 3D camera, a monocular camera, or a binocular camera. When the camera is implemented as a binocular camera, it specifically includes a left camera and a right camera; the binocular camera can obtain the screen corresponding to the projection device, that is, the image and playback content presented by the projection surface, and the image or playback content is projected by the built-in optical machine of the projection device.

[0102] When the projection device moves, its projection angle and distance to the projection surface change, which will cause the projected image to deform. The projected image will appear as a trapezoidal image or other deformed image. The projection device controller can automatically enter the screen based on the image taken by the camera by coupling the angle between the optical and mechanical projection surfaces and the correct display of the projection image.

[0103] Figure 5 A schematic diagram of a system framework for implementing display control for a projection device provided in some embodiments of the present application.

[0104] In some embodiments, see Figure 5 The system framework includes application service layer, process communication framework, operation layer, framework layer, correction service, camera service, flight time service, hardware and its driver, etc. The controller of the projection host 2 controls the overall system architecture and implements projection control of the projection device based on the underlying program logic, including but not limited to automatic screen entry, automatic obstacle avoidance, automatic focus, anti-eye shooting, screen switch control, automatic correction of the projection picture and fine-tuning correction, etc.

[0105] In some embodiments, the projection host 2 is also configured with a distance sensor for detecting distance. The distance sensor can be a time of flight (TOF) sensor. The time of flight sensor uses the flight time of a signal between a transmitting end and a reflecting end to measure the distance between nodes. After the time of flight sensor collects the distance data, it sends the distance data to the time of flight service; after the time of flight service obtains the distance data, it sends the collected distance data to the application service layer through the process communication framework. The distance data will be used for interactive use such as controller data calls, user interfaces, and program applications.

[0106] In some embodiments, the projection host 2 may also be configured with an image collector, which may be a monocular camera, a binocular camera, a depth camera or a 3D camera, etc.; the image collector sends the collected image data to the camera service, and then the camera service sends the image data to the process communication framework and / or the correction service; the process communication framework sends the image data to the application service layer, and the image data will be used for interactive use such as the controller's data call, user interface, and program application.

[0107] In some embodiments, the projection device can be refocused after automatic calibration is completed, and the controller detects whether the automatic focusing function is turned on; if the automatic focusing function is not turned on, the controller will end the automatic focusing service; if the automatic focusing function is turned on, the controller performs focusing calculations based on the distance detection value of the time of flight sensor.

[0108] In some embodiments, the controller queries a preset mapping table based on the distance detection value of the time-of-flight sensor, and the preset mapping table records the mapping relationship between distance and focal length, so as to obtain the focal length of the projection device corresponding to the distance detection value; then the middleware sends the acquired focal length to the optical machine of the projection device; after the optical machine performs laser emission according to the above focal length, at least one image collector captures the projection content image, and the controller performs clarity detection on the projection content image to determine whether the current lens focal length is appropriate. If the focal length is not appropriate, focusing processing is required. The projection device locates the focus position with the highest clarity by adjusting the lens position and shooting, and comparing the clarity changes of the projection content image before and after the adjustment.

[0109] If the judgment result meets the preset completion conditions, the control of the automatic focusing process ends; if the judgment result does not meet the preset completion conditions, the middleware will fine-tune the focal length parameters of the projection device optical machine, such as gradually fine-tuning the focal length according to the preset step length, and setting the adjusted focal length parameters to the optical machine again, through multiple photos, clarity evaluation and other steps, and finally through the clarity comparison of the projection picture, the optimal focal length is locked, thereby completing the automatic focusing.

[0110] In some embodiments, in order to improve the viewing experience, the projection device needs to calibrate the light-emitting component when the projection content is first projected onto the projection surface or when displacement occurs during the projection process. This ensures that the projection content is displayed completely and smoothly on the projection surface. This process is called "entering the screen", that is, adjusting the projection picture so that the light-emitting area of ​​the optical machine can be accurately projected onto the screen area. In this process, the position coordinates of the screen area in the physical coordinate system need to be converted to the optical machine coordinate system to ensure that the projection area of ​​the optical machine is accurately docked with the screen area. When converting the position coordinates of the screen area in the physical coordinate system to the optical machine coordinate system, it is generally necessary to perform zooming and "picture card" operations. The purpose of zooming is to adjust the image ratio so that the projected image is better adapted to the screen area, to avoid the image being too small to affect the viewing experience, or too large to cause waste of light source. The "picture card" operation is used to determine the correspondence between the physical coordinate system and the optical machine coordinate system, so as to facilitate subsequent coordinate conversion.

[0111] In one embodiment, in combination Figure 6 As shown, the coordinate system of the camera can be used as an intermediate coordinate system between the physical coordinate system and the optomechanical coordinate system to help complete the coordinate conversion process between the physical coordinate system and the optomechanical coordinate system. Among them, the physical coordinate system usually refers to the coordinate system corresponding to the actual physical environment. In the scene of the projection device, the specific position of the curtain area in the three-dimensional space can be clarified. The camera coordinate system is a coordinate system related to the internal imaging system of the camera, especially the position and direction on the image sensor (such as CMOS or CCD sensor). The optomechanical coordinate system is a coordinate system related to the internal optical system of the projection device (such as a laser projector, a projector, etc.), which is used to describe the position and direction of the optical elements (such as light sources, optomechanical, etc.) in the projection device. Exemplarily, the corresponding relationship between different coordinate systems is determined, such as the conversion matrix between the optomechanical coordinate system and the camera coordinate system (for example, the homography matrix H2), and the conversion matrix between the physical coordinate system and the optomechanical coordinate system, so that the coordinate points in the optomechanical coordinate system can be calculated by the corresponding coordinates and the homography matrix to obtain the corresponding coordinates on the projection surface, which are the coordinates in the physical coordinate system. The corresponding area is determined based on the coordinates, and the projection content is projected based on the corresponding area.

[0112] In one embodiment, during the process of the projection device entering the screen, the following conventional steps are usually performed: when it is recognized that there is a screen in the projection area, first, the projection device prints a white picture card with the same size and specifications as the screen area. The main purpose of this step is to obtain the coordinates of the screen area and provide basic data for subsequent zoom adjustment; then, the projection device projects a full-screen checkerboard picture card, and through the geometric pattern of the checkerboard, it helps the device to more accurately calculate the distortion of the projection area and perform focusing operations to ensure the clarity of the image; finally, based on the above information, the projection device calculates the corresponding coordinates of the screen area in the optical-mechanical coordinate system, and then completes the process of entering the screen, and realizes the precise docking of the projection picture and the screen area. When it is recognized that there is no screen in the projection area, first, the projection device prints a white picture card, then, the projection device projects a full-screen checkerboard picture card, and performs focusing operations, finally, based on the above information, the projection device calculates the corresponding coordinates of the screen area in the optical-mechanical coordinate system, and then completes the process of entering the screen, and realizes the precise docking of the projection picture and the screen area. However, during the process of entering the screen according to the above-mentioned entry operation, the display time of the picture card is as long as 10 to 20 seconds, causing the user to wait for a long time. Moreover, when the entry operation is performed in the scene being projected, the full-screen picture card will affect the user's viewing content, and the user experience is poor.

[0113] In order to solve the above problems, the present application provides a projection device 200, wherein the projection device can be a telephoto projection device. The projection device 200 includes a camera, an optical machine, and a controller. The camera is configured to capture an image of a projection surface. The optical machine is configured to project media resources. Figure 7 As shown, the controller is configured as:

[0114] S100: In response to the screen entry instruction, an image of a positioning card is projected in a preset area of ​​the screen area by an optical machine.

[0115] It is understandable that the entry operation can be adjusted by the user in advance in the menu of the projection device 200. In this way, the projection device 200 can obtain the entry instruction during operation, and then determine whether to perform the entry operation.

[0116] In one embodiment, the projection device 200 is displaced during operation due to collision or other factors, or interacts with the user by clicking the entry screen operation instruction. When the movement of the projection device 200 is detected, or in response to the user clicking the entry screen switch, the entry screen instruction is generated.

[0117] After generating the entry instruction, the projection device 200 responds to the entry instruction and projects the image of the positioning chart card in a preset area of ​​the screen area through an optical machine.

[0118] In one embodiment, the above S100 includes the following steps S101 to S104:

[0119] Step S101: in response to a screen entry instruction, a second image corresponding to an initial focal length is acquired through a camera.

[0120] The initial focal length is determined when the throw ratio (Throw Ratio) of the optical machine is a first value. Among them, the throw ratio is an important parameter in the projection device 200, which is used to describe the width of the picture projected by the projection device 200 at different projection distances. Therefore, the throw ratio can exist in the projection device 200 in the form of a range value. For the sake of simplicity, the throw ratio is described below using R as an example. Exemplarily, the throw ratio range can be R = 0.9-1.5.

[0121] For example, the projection ratio may be calculated using the following formula (1):

[0122]

[0123] Wherein, D is the known projection distance, and W is the known screen width.

[0124] Different projection ratios correspond to different focal lengths. For example, the focal length of a projection ratio of 1.5 is F5, and the focal length of a projection ratio of 0.9 is F0, where F5 is 1.67 times that of F0.

[0125] For example, the focal length may be calculated using the following formula (2):

[0126] f=R*zoom factor formula (2)

[0127] Where f is the focal length.

[0128] Exemplarily, the first value is 0.9, that is, the projection ratio is 0.9. When the projection ratio is 0.9, based on formula (1) and formula (2), the initial focal length is calculated to be F0. Further, the projection device 200 responds to the screen entry instruction and shoots at the focal length F0 to obtain the second image.

[0129] Step S102: when it is detected that the second image includes a curtain area object, initial coordinates corresponding to the curtain area object are acquired.

[0130] The initial coordinates are the coordinates of the corner points corresponding to the curtain area object. The curtain area object refers to the curtain area in the camera coordinate system.

[0131] In one embodiment, step S102 includes steps S1021 to S1023.

[0132] Step S1021, using the second image as input of a pre-trained second neural network model, and using the second neural network model to output a first probability and output coordinates.

[0133] The first generalization is used to measure the probability of the existence of the curtain area object in the second image. The second neural network model is used to identify the corner coordinates of the curtain and the card in the image. In an embodiment of the present application, the second neural network model is used to identify the corner coordinates of the curtain area object in the first image and the corner coordinates of the positioning card object.

[0134] Among them, the pre-trained neural network model refers to a neural network model that has been trained on a large-scale data set before use. In order to ensure the generalization ability of the neural network model, the neural network model needs to be trained based on a large amount of training data.

[0135] In one implementation, the training method of the second neural network model is used for exemplary description.

[0136] First, a training set is obtained. The training set includes at least one training image, target probability, and target initial coordinates. The training images include multiple images of the curtain and multiple images without the curtain.

[0137] Next, at least one training image is used as input of the first neural network model, and the target probability and target output coordinates are used as outputs of the second neural network model to train the second neural network model.

[0138] In one example, at least one training image is input into the initial neural network model, and the predicted probability and predicted output coordinates are output. Further, a preset loss function is used to determine the loss values ​​of the predicted probability and the target probability, and to predict the loss values ​​of the initial coordinates and the target output coordinates. When the above loss values ​​are respectively less than the preset loss values, it means that the training of the initial neural network model is completed, and a trained second neural network model is obtained.

[0139] After the second neural network model is trained, the second neural network model is used to obtain the first probability and output coordinates.

[0140] Exemplarily, the second image is input into the second neural network model, the output first probability is 0.6, and the output coordinates are (x1, y1), (x2, y2), (x3, y3), (x4, y4).

[0141] Step S1022: when the first probability is greater than a preset probability, determining that the second image includes a curtain area object.

[0142] Exemplarily, the preset probability may be 0.5.

[0143] Continuing with the above example, based on the fact that 0.6 is greater than 0.5, it can be determined that the second image includes a curtain area object.

[0144] Step S1023: when the second image includes a curtain area object, the output coordinates are used as initial coordinates corresponding to the curtain area object.

[0145] Continuing with the above example, the initial coordinates corresponding to the curtain area object are determined to be (x1, y1), (x2, y2), (x3, y3), and (x4, y4).

[0146] In order to ensure that the image of the positioning card is accurately projected into the preset area, it is necessary to select a suitable first focal length. The following is a schematic description of how to obtain the first focal length.

[0147] Step S103: acquiring a first focal length based on the initial coordinates corresponding to the curtain area object.

[0148] Among them, the first focal length is the focal length at which the ratio of the projection image to the screen is fully adapted, that is, at this focal length, the size ratio of the projection image and the screen area is fully matched, ensuring that the projection image accurately covers the screen area, and the image does not appear too large or too small.

[0149] In one embodiment, the condition for determining the first focal length is to determine whether the projection image is completely matched with the screen area, which can be confirmed by checking whether the coordinates of the screen area are located on the boundary of the maximum light output area of ​​the optical machine.

[0150] The following is an example of determining whether the coordinates of the screen area are located on the boundary of the maximum light output area of ​​the optical machine in the camera coordinate system, and then obtaining the first focal length.

[0151] Exemplarily, the step of acquiring the first focal length includes steps S1031 to S1033.

[0152] Step S1031: based on the first conversion relationship, convert the first optical-mechanical coordinates of the optical machine to obtain the first camera coordinates corresponding to the camera.

[0153] The first optical coordinates are the coordinates of the corner points corresponding to the maximum light-emitting area in the optical coordinate system, that is, the first optical coordinates are the vertex coordinates corresponding to the physical boundary area projected by the optical machine at the initial focal length. The first camera coordinates are the coordinates of the corner points of the first optical coordinates in the camera coordinate system.

[0154] In one example, based on the first parameter information, a first conversion relationship between the optical machine and the projection surface is determined; based on the first conversion relationship and the second parameter information, a second conversion relationship between the optical machine and the camera is determined; based on the second conversion relationship, the first optical machine coordinates are converted to obtain the first camera coordinates. The second conversion relationship is an approximate mapping relationship between the optical machine and the camera.

[0155] The first parameter information includes the internal parameters of the optical machine, the depth information of the depth sensor, and the external parameters between the depth sensor and the optical machine. The following is an explanation with internal parameters. Among them, the internal parameters Mp of the optical machine, also known as the internal parameter matrix of the optical machine, refer to the parameters that describe the internal optical system of the optical machine (such as the focal length of the optical machine, the coordinates of the principal point, etc.), which are used to determine the mapping relationship from the three-dimensional space to the two-dimensional projection screen. Among them, the depth sensor can be a time of flight (Time of Flight, TOF) sensor. The TOF sensor is a sensor for obtaining depth information. It measures the distance based on the propagation time of the light signal between the object and the sensor. It calculates the distance by measuring the time it takes for the laser or infrared light to be emitted from the sensor and reach the surface of the object, and then generates a depth map of the scene. The depth information of the TOF sensor is usually expressed as a depth map (Depth Map). Each pixel in the depth map represents the distance from the sensor to a certain point. The depth map is a grayscale image, in which the grayscale value of each pixel represents the distance from the sensor to the surface of the object (curtain area). The external parameters between the TOF sensor and the optical machine include the rotation matrix R P2w and the translation vector T P2w , the rotation matrix R P2w and the translation vector T P2w It is the relationship between the optical machine coordinate system and the TOF sensor coordinate system. The external parameters are explained below.

[0156] The second parameter information includes the camera's internal parameter M c , the first external reference R between the camera coordinate system and the optical machine coordinate system P2c and the second external parameter T P2c The internal parameters Mc of the camera refer to the parameters describing the internal optical system of the camera, which are used to convert the three-dimensional world coordinates into two-dimensional image coordinates. The internal parameters Mc of the camera usually include the focal length of the camera, the coordinates of the principal point, etc. The first external parameter R between the camera coordinate system and the optomechanical coordinate system is P2c It is a rotation matrix used to describe the rotation relationship between the optical-mechanical coordinate system and the camera coordinate system. The first external parameter T between the camera coordinate system and the optical-mechanical coordinate system P2c It is a translation vector, which is used to describe the translation relationship from the origin of the optical-mechanical coordinate system to the origin of the camera coordinate system, and represents the relative displacement between the two coordinate systems.

[0157] The projection surface is an area or plane in the physical coordinate system, that is, the three-dimensional space of the actual screen area. In other words, the first conversion relationship between the camera and the projection surface is the conversion relationship between the camera coordinate system and the physical coordinate system.

[0158] Exemplarily, the second conversion relationship may be determined using formula (3):

[0159] C uv =Mc *(R P2w *R P2c ) -1 *(T P2w (2)*R P2w *M P -1 *P uv +T P2c -T P2w ) Formula (3)

[0160] Among them, C uv is the first camera coordinate, M c is the camera’s intrinsic parameter matrix, R P2w is the rotation matrix from the optomechanical coordinate system to the physical coordinate system, R P2c is the rotation matrix from the optical machine coordinate system to the camera coordinate system, T P2w is the translation vector from the optomechanical coordinate system to the physical coordinate system, T P2c is the translation vector from the optical-mechanical coordinate system to the camera coordinate system, M P is the internal parameter matrix of the optomechanical system, which is used to describe the internal parameters in the optomechanical coordinate system. uv is the first optomechanical coordinate in the optomechanical coordinate system.

[0161] Exemplarily, the first optical coordinate P uv 1 is converted to c uv 1.

[0162] Step S1032: Determine the zoom factor based on the initial coordinates of the curtain area object and the first camera coordinates.

[0163] The zoom factor, also known as the zoom ratio, is a parameter in the optical system that indicates the degree to which the lens can magnify or reduce the scene.

[0164] There are various ways to determine the zoom factor, and an iterative method is used for exemplary description below.

[0165] In one embodiment, in combination Figure 8 As shown, step S1032 includes steps S10321-S10325

[0166] Step S10321: If the initial coordinates are within the range corresponding to the first camera coordinates, the projection ratio is adjusted to a second value based on a preset step value.

[0167] The step value refers to the adjustment according to its inherent step length, such as increasing the projection ratio according to its inherent step length, or reducing the projection ratio according to its inherent step length. The value of the step value can be set according to actual needs and is not specifically limited here. For example, 0.1.

[0168] Continuing with the above example, the initial projection ratio R is 0.9, and the preset step value is 0.1. The initial coordinates (x1, y1), (x2, y2), (x3, y3), (x4, y4) are at the first camera coordinates (x 11 ,y 11 ), (x 22 ,y 22 ), (x 33 ,y 33 ), (x 44 ,y 44 ) within a range corresponding to the value of R, the projection ratio is adjusted to obtain a second value of the projection ratio, that is, R=1.

[0169] Step S10322: Determine a first adjustment focal length corresponding to a second value of the projection ratio.

[0170] Following the above example, the projection ratio is determined to be a second value, that is, the first adjustment focal length F1 where R=1.

[0171] Step S10323: Determine the first camera coordinates corresponding to the first adjusted focal length.

[0172] When the focal length changes, the internal parameter Mp of the optical machine will adapt to the change, which will cause the first camera coordinates to change. At this time, the first camera coordinates are (x 12 ,y 12 ), (x 23 ,y 23 ), (x 34 ,y 34 ), (x 45 ,y 45 ).

[0173] Step S10324: If the initial coordinates are within the range corresponding to the first camera coordinates, continue to adjust the projection ratio and execute step S1.

[0174] Continuing with the above example, determine whether the initial coordinates (x1, y1), (x2, y2), (x3, y3), (x4, y4) are within the first camera coordinates (x 12 ,y 12 ), (x 23 ,y 23 ), (x 34 ,y 34 ), (x 45 ,y 45 ), if it is within the range corresponding to the first camera coordinates, continue to update the projection ratio. If the projection ratio is determined to be the third value, that is, R=1.1, continue to execute steps S1-S4, and repeat this cycle until the initial coordinates are no longer within the range corresponding to the first camera coordinates, and then execute step S5.

[0175] Step S10325: Determine the zoom factor based on the current projection ratio and the preset step value.

[0176] For example, if R=1.1, the initial coordinate is not within the range corresponding to the first camera coordinate, based on the projection ratio and the preset step value, the zoom factor is determined to be Step-1. For example, R=1.1, step=0.1, and the zoom factor is 1.

[0177] Step S1033: Determine a first focal length based on the zoom factor.

[0178] Continuing with the above example, when the zoom factor is 1, the first focal length is determined to be F2.

[0179] Step S104: Based on the first focal length, an image of the positioning chart is projected into a preset area of ​​the screen area by an optical machine.

[0180] A preset area is set in the screen area for projecting the positioning card. The size of the preset area is not specifically limited, but it must be large enough to accommodate all the key points of the positioning card and ensure that the minimum spacing requirements are met between these key points. For example, the minimum spacing is 1 pixel. For example, if the specified minimum spacing is 1 pixel, then the coordinate difference between any two corner points (either horizontally or vertically) must be greater than 1 pixel. At the same time, the preset area must be completely within the scope of the screen area, that is, it must not exceed the actual size of the screen area.

[0181] It should be noted that there is no specific limitation on the setting of the minimum spacing.

[0182] Alignment Chart or Calibration Target is a special image or pattern used to calibrate and adjust optical systems, projectors, cameras and other equipment. They usually contain a series of precisely designed geometric shapes, lines, points or other features that help ensure that the equipment can accurately capture or project images, and can be used to measure and correct various types of distortion or errors. Exemplarily, at a first focal length, the image of the alignment chart is projected into a preset area in the screen area by an optical machine. For example, the preset area can be the central area of ​​the screen area.

[0183] Exemplarily, a positioning map card is projected in a curtain area on the projection surface, and the positioning map card is arranged in a preset area.

[0184] S200: Acquire a first image corresponding to the projection surface through a camera.

[0185] The first image includes a curtain area object corresponding to the curtain area and a positioning card object corresponding to the positioning card. That is, the curtain area is the area corresponding to the real curtain in the physical coordinate system, and the positioning card is the card displayed in the curtain area in the physical coordinate system. The curtain area object is the curtain area in the camera coordinate system, and the positioning card object is the positioning card in the camera coordinate system.

[0186] For example, in combination Fig. 9 As shown, the camera is controlled to shoot the curtain area to obtain a first image corresponding to the curtain area. The first image includes a curtain area object 91 and a positioning card object 92. The positioning card object 92 includes a first card area 920 and a second card area 921. The card area is an area surrounded by the image vertices of the positioning card, and the second card area is set inside the first card area.

[0187] S300: Using the first image as an input of a pre-trained first neural network model, and using the first neural network model, obtaining first coordinates of a curtain area object and second coordinates of a positioning card object in the first image.

[0188] Optionally, the training method of the first neural network model may refer to the second neural network model in step S1021 above, which will not be described in detail here.

[0189] Exemplarily, the first image is input into a pre-trained first neural network model to obtain the first coordinates (x 1’ ,y 1’ ), (x 2’ ,y 2’ ), (x 3’ ,y 3’ ), (x 4’ ,y 4’ ), and the second coordinates of the positioning card object include the coordinates of the first card area (x 80 ,y 80 ), (x 81 ,y 82 ), (x 83 ,y 83 ), (x 84 ,y 84 ), and the coordinates of the second chart area (x 85 ,y 85 ), (x 86 ,y 86 ), (x 87 ,y 87 ), (x 88 ,y 88 ).

[0190] In order to improve the accuracy of the first coordinates of the curtain area object and the second coordinates of the positioning card object, the acquisition of the first coordinates of the curtain area object and the second coordinates of the positioning card object may be optimized.

[0191] In one embodiment, step S300 may include steps S301 to S303.

[0192] Step S301: based on the first coordinates and the second coordinates, a first image is cropped to obtain at least one first cropped image corresponding to the first coordinates, and at least one second cropped image corresponding to the second coordinates.

[0193] For example, in combination Fig.10 An exemplary description is given.

[0194] Fig.10 A schematic diagram of a first image cropping process provided for some embodiments of the present application.

[0195] like Fig.10 As shown, each coordinate point in the first coordinate and the second coordinate is used as a center point, and a 96*96 image is cropped on the first image to obtain at least one first cropped image and a second cropped image.

[0196] The ratio of the cropped image may be the above-mentioned 96*96, or may be other ratios, which are not specifically limited here.

[0197] For example, taking the first coordinate as an example, the first coordinate is (x 1’ ,y 1’ ), (x 2’ ,y 2’ ), (x 3’ ,y 3’ ), (x 4’ ,y 4’ ), the above (x 1’ ,y 1’ ) as the center point, with (x 1’ -48,y 1’ -48)、(x 1’ +48,y 1’ +48) as the vertex to obtain four first cropped images corresponding to the four corner points corresponding to the curtain area object. Similarly, taking the second coordinate as an example, eight second cropped images corresponding to the eight corner points corresponding to the positioning map card can be obtained.

[0198] Step S302: using at least one first cropped image and at least one second cropped image as inputs of a pre-trained third neural network model, and using the third neural network model to obtain a fourth coordinate of the first cropped image and a fifth coordinate of the second cropped image.

[0199] Among them, the third neural network model is used to identify the coordinates of corner points in the image.

[0200] The training method of the third neural network model can refer to the above step S1021 and will not be repeated here.

[0201] Following the above example, the four first cropped images corresponding to the four corner points and the eight second cropped images corresponding to the eight corner points are input into the third neural network model to obtain the fourth coordinate (x 1” ,y 1” ), (x 2” ,y 2” ), (x 3” ,y 3” ), (x 4” ,y 4” ) and the fifth coordinate (x 80’ ,y 80’ ), (x 81’ ,y 82’ ), (x 83’ ,y 83’ ), (x 84’ ,y 84’ ), (x 85’ ,y 85’ ), (x 86’ ,y 86’ ), (x 87’ ,y 87’ ), (x 88’ ,y 88’ ).

[0202] Step S303: based on the coordinate system of the first image, transform the fourth coordinate and the fifth coordinate to obtain the calibrated first coordinate and the calibrated second coordinate of the first image.

[0203] Among them, the fourth coordinate and the fifth coordinate are the corner point coordinates of the first cropped image and the second cropped image in the camera coordinate system. The coordinates of the same point in the cropped image are not the same as those in the first image, so coordinate conversion is required to obtain the coordinates corresponding to the first image.

[0204] Continuing with the above example, the fourth coordinate (x 1” ,y 1” ), (x 2” ,y 2” ), (x 3” ,y 3” ), (x 4” ,y 4” ) and the fifth coordinate (x 80’ ,y 80’ ), (x 81’ ,y82’ ), (x 83’ ,y 83’ ), (x 84’ ,y 84’ ), (x 85’ ,y 85’ ), (x 86’ ,y 86’ ), (x 87’ ,y 87’ ), (x 88’ ,y 88’ ) is transformed to obtain the updated first coordinate (x 1”’ ,y 1”’ ), (x 2”’ ,y 2”’ ), (x 3”’ ,y 3”’ ), (x 4”’ ,y 4”’ ), and obtain the updated second coordinate (x 80” ,y 80” ), (x 81” ,y 82” ), (x 83” ,y 83” ), (x 84” ,y 84” ), (x 85” ,y 85” ), (x 86” ,y 86” ), (x 87” ,y 87” ), (x 88” ,y 88” ).

[0205] S400: Based on the first corresponding relationship, transform the first coordinate to determine a third coordinate.

[0206] Among them, the first corresponding relationship is a corresponding relationship established based on the second coordinate and the preset image card coordinates in the optical machine to characterize the precise mapping relationship between the camera and the optical machine. The corresponding relationship can be called the homography matrix H2.

[0207] Exemplarily, the third coordinate in the optical machine can be determined based on formula (4):

[0208] P uv =H2*C uv Formula (4)

[0209] Among them, P uv is the coordinate in the optical-mechanical coordinate system; C uv is the coordinate in the camera coordinate system.

[0210] For example, the third coordinate in the optical-mechanical coordinate system is determined based on formula (4).

[0211] S500: Perform a projection operation based on the third coordinate.

[0212] In summary, the above method can reduce the display time of the image card, reduce the user waiting time, and improve the user experience.

[0213] Corresponding to the above-mentioned embodiment including the curtain area, there is also an embodiment not involving the curtain area.

[0214] Combine the following Fig.11 As shown, Fig.11 A second flow chart of a method for entering a screen provided in some embodiments of the present application.

[0215] like Fig.11 As shown, the curtain entry method includes the following steps:

[0216] S10: In response to the screen entry instruction, a third image corresponding to the initial focal length is acquired through the camera.

[0217] The third image does not include the curtain area object.

[0218] The specific content of step S10 can refer to the similar content of the above-mentioned step S101, which will not be repeated here.

[0219] S11: When the third image does not include the curtain area object, determine whether the obstacle avoidance switch is turned on based on the third image.

[0220] S12: When the obstacle avoidance switch is turned on, projection is performed on the first target area corresponding to the optical machine based on the second optical machine coordinates.

[0221] Among them, combined Fig.12 As shown in (a), the first target area is the area corresponding to the second optical-mechanical coordinates that does not include the area corresponding to the obstacle, and the second optical-mechanical coordinates are the vertex coordinates corresponding to the area of ​​the optical-mechanical light-emitting boundary.

[0222] S13: When the obstacle avoidance switch is not turned on, projection is performed on a second target area corresponding to the optical machine based on the second optical machine coordinates.

[0223] Among them, combined Fig.12 As shown in (b), the second target area is the area corresponding to the obstacle within the area corresponding to the second optical-mechanical coordinates.

[0224] In summary, the above method is used to complete the screen entry, which improves the user experience. Based on the above display device 200, some embodiments of the present application also provide a screen entry method, including the following steps: in response to the screen entry instruction, the optical machine projects the image of the positioning card in the preset area of ​​the screen area; the image of the positioning card includes a first card area and a second card area; the first card area is the area surrounded by the vertices of the image of the positioning card, and the second card area is set inside the first card area;

[0225] Acquire a first image corresponding to the curtain area through the camera;

[0226] The first image is used as an input of a pre-trained first neural network model, and the first neural network model is used to obtain a first coordinate of a curtain area object and a second coordinate of a positioning card object in the first image; wherein the first coordinate is a corner point coordinate corresponding to the curtain area object; and the second coordinate is a corner point coordinate corresponding to the positioning card object;

[0227] Based on the first corresponding relationship, the first coordinate is converted to determine a third coordinate, where the third coordinate is the corner point coordinate corresponding to the curtain area object in the optical machine; the first corresponding relationship is a corresponding relationship between the camera and the optical machine established based on the second coordinate and the preset image card coordinate in the optical machine;

[0228] Based on the third coordinate, a projection operation is performed in the screen area.

[0229] The above technical solution has the following beneficial effects or advantages: in response to the screen entry instruction, the projection device 200 projects the positioning card image in the preset area of ​​the screen area, and shoots the image through the camera, and uses the pre-trained neural network model to quickly obtain the coordinates of the screen area and the positioning card. Based on these coordinates, combined with the corresponding relationship between the optical machine and the camera, coordinate conversion can be performed to accurately determine the coordinates of the corner points of the screen area in the optical machine, thereby achieving accurate projection operation. In this way, the display time of the card and the automated calibration process are shortened, which not only improves the efficiency of screen entry, but also enhances the accuracy of projection to a certain extent and optimizes the user experience.

[0230] Combine the following Fig.13 The method of entering the curtain is explained in detail.

[0231] Fig.13 A schematic diagram of the entry operation process provided for some embodiments of the present application.

[0232] like Fig.13 As shown, the curtain entry method includes the following steps:

[0233] The projection device 200 determines whether a screen entry instruction is generated.

[0234] When generating a screen entry instruction, the projection device 200 adjusts the brightness and obtains the first optical-mechanical coordinates.

[0235] Among them, the first optical machine coordinate is the vertex coordinate corresponding to the physical boundary area projected by the optical machine, that is, adjusting the brightness means adjusting the brightness of the optical machine to the first brightness based on the entry-screen instruction, and the first brightness is the same as the preset brightness when the projector performs the entry-screen operation.

[0236] The projection device 200 controls the camera to capture the second image at the initial focal length.

[0237] The projection device 200 uses the second neural network model to determine the first probability and output coordinates corresponding to the second image.

[0238] The projection device 200 determines whether there is a curtain area in the second image.

[0239] If the second image includes the screen area, the projection device 200 calculates a zoom factor.

[0240] The projection device 200 performs a zoom factor operation to obtain a first focal length.

[0241] The projection device 200 projects an intermediate positioning chart, captures an image at a first focal length, and obtains a first image.

[0242] The projection device 200 uses the first neural network model to obtain the fourth coordinate and the fifth coordinate in the first image.

[0243] The projection device 200 uses each coordinate point of the fourth coordinate and the fifth coordinate in the first image as a center point to perform 96*96 image cropping.

[0244] The projection device 200 uses the third neural network model to obtain the first coordinates and the second coordinates in the cropped first image.

[0245] Based on the second coordinates and the preset image card coordinates, the homography matrix H2 of the camera and the optical machine is determined.

[0246] The projection device 200 determines whether the conversion of the first coordinate into the optical-mechanical coordinate system is successful according to H2 and the first coordinate.

[0247] If the first coordinate is successfully converted to the optical machine coordinate system, the screen entry is completed.

[0248] If the conversion of the first coordinate to the optical-mechanical coordinate system fails, determine whether the obstacle avoidance switch is turned on.

[0249] If the obstacle avoidance switch is on, obstacles are avoided and the projection is performed in the first target area.

[0250] If the obstacle avoidance switch is not turned on, the second target area is projected.

[0251] If the second image does not have a curtain area, it is determined whether the obstacle avoidance switch is turned on.

[0252] If the obstacle avoidance switch is on, obstacles are avoided and the projection is performed in the first target area.

[0253] If the obstacle avoidance switch is not turned on, the second target area is projected.

[0254] The same and similar parts between the various embodiments in this specification can be referenced to each other and will not be described again here.

[0255] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of various embodiments of the present invention or certain parts of the embodiments.

[0256] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application is described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application. For the convenience of explanation, the above description has been made in conjunction with specific implementation methods. However, the above exemplary discussion is not intended to be exhaustive or to limit the implementation methods to the specific forms disclosed above. According to the above teachings, a variety of modifications and variations can be obtained. The selection and description of the above implementation methods are intended to better explain the principles and practical applications, so that those skilled in the art can better use the implementation methods and various different variations of implementation methods suitable for specific use considerations.

Claims

1. A projection device, characterized in that: include: A camera configured to capture an image of the projection surface; An optical machine is configured to project media resources; The controller is configured as: In response to the screen entry instruction, the image of the positioning card is projected in a preset area of ​​the screen area by the optical machine; the image of the positioning card includes a first card area and a second card area; the first card area is an area surrounded by the vertices of the image of the positioning card, and the second card area is set inside the first card area; Acquire a first image corresponding to the projection surface through the camera; the first image includes a curtain area object corresponding to the curtain area and a positioning card object corresponding to the positioning card; The first image is used as an input of a pre-trained first neural network model, and the first neural network model is used to obtain the first coordinates of the curtain area object and the second coordinates of the positioning card object in the first image; wherein the first neural network model is used to identify the corner coordinates of the curtain and the card in the image; the first coordinates are the corner coordinates corresponding to the curtain area object; the second coordinates include the corner coordinates corresponding to the first card area and the corner coordinates corresponding to the second card area; Based on the first corresponding relationship, the first coordinate is transformed to determine a third coordinate, where the third coordinate is the corner point coordinate corresponding to the curtain area object in the optical machine; the first corresponding relationship is a corresponding relationship established based on the second coordinate and the preset image card coordinate in the optical machine to characterize the mapping relationship between the camera and the optical machine; A projection operation is performed based on the third coordinate.

2. The projection device according to claim 1, characterized in that: The controller executes, in response to the screen entry instruction, projecting the image of the positioning chart card in a preset area of ​​the screen area through the optical machine, and is specifically configured as follows: In response to the screen entry instruction, a second image corresponding to an initial focal length is acquired through the camera; the initial focal length is determined when the projection ratio of the optical machine is a first value; In the case where it is detected that the second image includes the curtain area object, initial coordinates corresponding to the curtain area object are acquired; wherein the initial coordinates are the coordinates of the corner points corresponding to the curtain area object; Based on the initial coordinates corresponding to the curtain area object, obtaining a first focal length; Based on the first focal length, the image of the positioning chart is projected into a preset area of ​​the screen area by the optical machine.

3. The projection device according to claim 2, characterized in that: The controller executes, when detecting that the second image includes the curtain area object, acquiring the initial coordinates corresponding to the curtain area object in the second image, and is specifically configured as follows: The second image is used as an input of a pre-trained second neural network model, and the second neural network model is used to output a first probability and an output coordinate; the first generalization is used to measure the probability that the second image includes the curtain area object; When the first probability is greater than a preset probability, determining that the second image includes the curtain area object; In the case where the second image includes the curtain area object, the output coordinates are used as the initial coordinates corresponding to the curtain area object.

4. The projection device according to claim 2 or 3, characterized in that: The controller executes acquiring a first focal length based on the initial coordinates corresponding to the curtain area object, and is specifically configured as follows: Based on the second conversion relationship, the first optical-mechanical coordinates of the optical machine are converted to obtain the first camera coordinates corresponding to the camera; wherein the first optical-mechanical coordinates are the corner point coordinates corresponding to the boundary area projected by the optical machine at the initial focal length; and the second conversion relationship is an approximate mapping relationship between the optical machine and the camera; Determine a zoom factor based on the initial coordinates of the curtain area object and the first camera coordinates; Based on the zoom factor, the first focal length is determined.

5. The projection device according to claim 4, characterized in that: Also includes: A depth sensor configured to obtain depth information; The controller performs a conversion based on the second conversion relationship on the first optical-mechanical coordinate of the optical machine to obtain the first camera coordinate corresponding to the camera, and is specifically configured as follows: Determine a first conversion relationship between the optical machine and the projection surface based on first parameter information, where the first parameter information includes internal parameters of the optical machine, depth information of the depth sensor, and external parameters between the depth sensor and the optical machine; Determine the second conversion relationship between the optical machine and the camera based on the first conversion relationship and the second parameter information, wherein the second parameter information includes the first external parameter and the second external parameter between the camera coordinate system and the optical machine coordinate system, and the internal parameters of the camera; Based on the second conversion relationship, the first optical-mechanical coordinates are converted to obtain the first camera coordinates.

6. The projection device according to claim 4, characterized in that: The controller determines the zoom factor based on the initial coordinates of the curtain area object and the first camera coordinates, and is specifically configured as follows: If the initial coordinates are within the range corresponding to the first camera coordinates, adjusting the projection ratio to a second value based on a preset step value; Determine a first adjustment focal length corresponding to when the projection ratio is the second value; Determining the first camera coordinates corresponding to the first adjusted focal length; If the initial coordinates are within the range corresponding to the first camera coordinates, continue adjusting the projection ratio; When the initial coordinates are not within the range corresponding to the first camera coordinates, the zoom factor is determined based on the current projection ratio and the preset step value.

7. The projection device according to claim 1, 2 or 3, characterized in that: The controller executes the step of using the first image as an input of a pre-trained second neural network model and obtaining the first coordinates of the curtain area object in the first image and the second coordinates of the image of the positioning map card by using the first neural network model, which is specifically configured as follows: Using the first image as an input of a pre-trained first neural network model, and using the first neural network model, obtaining first coordinates of a curtain area object and second coordinates of a positioning card object in the first image; Based on the first coordinates and the second coordinates, the first image is cropped to obtain at least one first cropped image corresponding to the first coordinates, and at least one second cropped image corresponding to the second coordinates; Using the at least one first cropped image and the at least one second cropped image as inputs of a pre-trained third neural network model, and using the third neural network model to obtain a fourth coordinate of the first cropped image and a fifth coordinate of the second cropped image; Based on the coordinate system of the first image, the fourth coordinate and the fifth coordinate are converted to obtain the calibrated first coordinate of the curtain area object and the calibrated second coordinate of the positioning card object.

8. The projection device according to any one of claims 1 to 3, characterized in that: Before the controller executes the step of responding to the curtain entry instruction, the controller is further configured to: When the movement of the projection device is detected, or in response to a user clicking operation on the screen entry switch, an entry instruction is generated.

9. The projection device according to claim 3, characterized in that: The controller executes the step of taking the second image as an input of a pre-trained second neural network model, and outputting a first probability and an output coordinate using the second neural network model; after the step of the first generalization being used to measure the probability that the second image includes the curtain area object, the controller is further configured to: When the first probability is less than a preset probability, determining that the second image does not include the curtain area object; and determining whether an obstacle avoidance switch is turned on based on the second image; When the obstacle avoidance switch is turned on, projection is performed on a first target area corresponding to the optical machine based on the second optical machine coordinates, where the first target area is an area not including obstacles within the area corresponding to the second optical machine coordinates, and the second optical machine coordinates are vertex coordinates corresponding to the physical boundary area projected by the optical machine; When the obstacle avoidance switch is not turned on, based on the second optical machine coordinates, in the second target area corresponding to the optical machine, the second target area is an area corresponding to the obstacle within the area corresponding to the second optical machine coordinates.

10. A method for projecting onto a screen, characterized in that: include: In response to the screen entry instruction, the image of the positioning card is projected in a preset area of ​​the screen area by the optical machine; the image of the positioning card includes a first card area and a second card area; the first card area is an area surrounded by the vertices of the image of the positioning card, and the second card area is set inside the first card area; Acquire a first image corresponding to the projection surface through the camera; the first image includes a curtain area object corresponding to the curtain area and a positioning card object corresponding to the positioning card; The first image is used as an input of a pre-trained first neural network model, and the first neural network model is used to obtain the first coordinates of the curtain area object and the second coordinates of the positioning card object in the first image; wherein the first neural network model is used to identify the corner coordinates of the curtain and the card in the image; the first coordinates are the corner coordinates corresponding to the curtain area object; the second coordinates include the corner coordinates corresponding to the first card area and the corner coordinates corresponding to the second card area; Based on the first corresponding relationship, the first coordinate is transformed to determine a third coordinate, where the third coordinate is the corner point coordinate corresponding to the curtain area object in the optical machine; the first corresponding relationship is a corresponding relationship established based on the second coordinate and the preset image card coordinate in the optical machine to characterize the mapping relationship between the camera and the optical machine; A projection operation is performed based on the third coordinate.

Images