Projection picture adjustment method, device and projector

Abstract

The present application discloses a method, device and projector for adjusting a projection screen. The method comprises: obtaining continuous frame images, wherein each frame image includes a projection screen and an input gesture; determining the gesture category of the input gesture according to the continuous frame images, and determining the control information corresponding to the gesture category; and adjusting the position or size of the projection screen according to the gesture category and the control information. The present application solves the technical problem of poor user interaction experience when adjusting the projection screen using a remote control in the related art.

Description

Technical Field

[0001] The present application relates to the field of projection technology, and in particular to a projection image adjustment method, device and projector. Background Art

[0002] Although home projectors are becoming more and more intelligent in controlling the projection screen, they still cannot achieve the desired effect in some scenarios. At this time, users are often required to use remote controls to fine-tune the projection screen, adjust the position of the projection screen, or scale the size of the projection screen to achieve an effect that better meets user expectations. However, the use of remote controls for adjustment will make users lack a sense of personal participation and fun, and the interactive experience is poor. At the same time, when the adjustment function increases, it will also bring greater learning and memory costs. How to adjust the projection screen through a more convenient and comfortable interactive method has become a more important topic for current home projectors.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a method, device and projector for adjusting a projection image, so as to at least solve the technical problem in the related art that the user interaction experience is poor when adjusting the projection image using a remote control.

[0005] According to one aspect of an embodiment of the present application, a method for adjusting a projection screen is provided, comprising: acquiring continuous frame images, wherein each frame image includes a projection screen and an input gesture; determining a gesture category of the input gesture based on the continuous frame images, and determining control information corresponding to the gesture category; and adjusting a position or size of the projection screen based on the gesture category and the control information.

[0006] Optionally, a first preset number of images whose shooting time is closest to the current system time are selected from the continuous frame images, and a second preset number of images are selected at equal intervals from the first preset number of images as a target image set; the target image set is input into a pre-trained dynamic gesture detection model, and the dynamic gesture category of the input gesture is detected by the dynamic gesture detection model.

[0007] Optionally, when it is detected that the dynamic gesture category is the first category, the control information is determined to translate the entire projection screen in the target direction by a target distance; when it is detected that the dynamic gesture category is the second category, the control information is determined to scale the entire projection screen to a target ratio.

[0008] Optionally, the target distance is a preset fixed distance; and the target ratio is a preset fixed ratio.

[0009] Optionally, the process of determining the target distance or the target ratio includes: determining the first frame image and the last frame image in the continuous frame images; determining the hand position in the first frame image and the hand position in the last frame image through a target detection model, wherein the hand position includes at least one of the following: hand detection frame position, hand key point position; determining the target distance or target ratio based on the hand position in the first frame image and the hand position in the last frame image.

[0010] Optionally, determine a first distance between the center position of the hand detection frame in the first frame image and the center position of the hand detection frame in the last frame image, and determine the target distance based on the first distance; or, determine a first speed of hand movement based on the number of image frames of the consecutive frame images and the first distance, and determine the target distance based on the first speed; or, determine a second distance between the palm position of the hand key points in the first frame image and the palm position of the hand key points in the last frame image, and determine the target distance based on the second distance; or, determine a second speed of hand movement based on the number of image frames of the consecutive frame images and the second distance, and determine the target distance based on the second speed. Optionally,

[0011] Optionally, a first proportional relationship between the hand detection frame in the first frame image and the hand detection frame in the last frame image is determined, and the target ratio is determined based on the first proportional relationship; or, a third speed of hand scaling is determined based on the number of image frames of the consecutive frame images and the first proportional relationship, and the target ratio is determined based on the third speed; or, a second proportional relationship between the circumscribed rectangle of the hand key points in the first frame image and the circumscribed rectangle of the hand key points in the last frame image is determined, and the target ratio is determined based on the second proportional relationship; or, a fourth speed of hand scaling is determined based on the number of image frames of the consecutive frame images and the second proportional relationship, and the target ratio is determined based on the fourth speed.

[0012] Optionally, any one image in the continuous frame images is determined to be a target image; the target image is input into a pre-trained static gesture detection model, and the static gesture category of the gesture in the target image is detected by the static gesture detection model.

[0013] Optionally, when it is detected that the static gesture category is the third category, the control information is determined to be translating the entire projection screen in a target direction by a target distance, wherein the target distance is a pre-set fixed distance; or, when it is detected that the static gesture category is the fourth category, the control information is determined to be scaling the entire projection screen by a target ratio, wherein the target ratio is a pre-set fixed ratio; when it is detected that the static gesture category is the fifth category, the control information is determined to stop adjusting the projection screen.

[0014] Optionally, when it is detected that the projection picture is adjusted to a critical position, the adjustment of the projection picture is stopped and a first information is generated, wherein the first information is used to prompt that the projection picture has been adjusted to the critical position; when it is detected that the projection picture is adjusted to coincide with a boundary of an obstacle in the projection surface, the adjustment of the projection picture is stopped and a second information is generated, wherein the second information is used to prompt that the projection picture encounters an obstacle.

[0015] Optionally, the target direction and target distance of the projection image to be translated are determined according to the gesture category and the control information, or the target ratio of the projection image to be scaled is determined; the position coordinates of the image to be projected in the projector are adjusted according to the target direction and the target distance, or according to the target ratio; and the image to be projected is projected according to the adjusted position coordinates of the image to be projected in the projector to obtain an adjusted target projection image.

[0016] Optionally, a target direction and a target distance for the projection image to be translated are determined according to the gesture category and the control information; and the projection image is translated in the target direction by the target distance through optical-mechanical axis shifting.

[0017] Optionally, in response to a control instruction, the projection picture in the adjustment mode is displayed based on a preset background content; or, in response to a control instruction, the projection picture in the adjustment mode is displayed based on a current projection picture content.

[0018] Optionally, a set of sample video data is obtained, wherein each sample video data corresponds to a dynamic gesture category; for any of the sample video data, a second preset number of images are selected at equal intervals from the sample video as a sample image subset, and the dynamic gesture categories corresponding to the sample image subset are labeled to obtain first labeling information; the sample image subset and the first labeling information are input into the dynamic gesture detection model for training.

[0019] According to another aspect of an embodiment of the present application, a projection screen adjustment device is also provided, including: an acquisition module, used to acquire continuous frame images, wherein each frame image includes a projection screen and an input gesture; a determination module, used to determine the gesture category of the input gesture based on the continuous frame images, and determine the control information corresponding to the gesture category; an adjustment module, used to adjust the position or size of the projection screen according to the gesture category and the control information.

[0020] According to another aspect of an embodiment of the present application, a projector is also provided, which includes a processor and a memory, wherein the memory is connected to the processor and is used to provide the processor with instructions for processing the following processing steps: acquiring continuous frame images, wherein each frame image includes a projection screen and an input gesture; determining a gesture category of the input gesture based on the continuous frame images, and determining control information corresponding to the gesture category; and adjusting the position or size of the projection screen based on the gesture category and the control information.

[0021] According to another aspect of an embodiment of the present application, a non-volatile storage medium is further provided, wherein the non-volatile storage medium includes a stored program, wherein when the program is executed, the device where the non-volatile storage medium is located is controlled to execute the above-mentioned projection picture adjustment method.

[0022] In the embodiment of the present application, firstly, continuous frame images are obtained, wherein each frame image includes a projection screen and an input gesture; then, the gesture category of the input gesture is determined based on the continuous frame images, and the control information corresponding to the gesture category is determined; finally, the position or size of the projection screen is adjusted based on the gesture category and the control information. The embodiment of the present application analyzes the user's gestures to understand the meaning that the user wants to express, allowing the user to adjust the position and size of the projection screen through their own gestures, thereby enhancing the experience and fun of personal participation, and thus solving the technical problem of poor user interaction experience when using a remote control to adjust the projection screen in the related art. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0024] Figure 1 is a flow chart of a projection picture adjustment method according to an embodiment of the present application;

[0025] Figure 2 is a schematic diagram of a projection screen translation according to an embodiment of the present application;

[0026] Figure 3is a schematic diagram of scaling a projection image according to an embodiment of the present application;

[0027] Figure 4 is a schematic diagram of a projection picture adjustment process according to an embodiment of the present application;

[0028] Figure 5 is a structural schematic diagram of a projection image adjustment device according to an embodiment of the present application;

[0029] Figure 6 It is a structural schematic diagram of a projector according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] Example 1

[0033] According to an embodiment of the present application, an embodiment of a projection picture adjustment method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0034] Figure 1 According to the projection image adjustment method of the embodiment of the present application, Figure 1 As shown, the method includes steps S102-S106, wherein:

[0035] Step S102, acquiring continuous frame images, wherein each frame image includes a projection picture and an input gesture.

[0036] In some optional embodiments of the present application, when the user adjusts the projection screen, he can enter different adjustment modes through different control instructions. The corresponding projection screen will be displayed in each adjustment mode, and the user can adjust the position or size of the projection screen in the adjustment mode.

[0037] Specifically, the projection screen in the adjustment mode can be displayed based on the preset background content in response to the control command. For example, the user can enter the gesture control interface adjustment mode through a preset trigger method such as a setting menu or a voice command. In this mode, the display content in the current projection screen will be replaced with the preset background content to obtain the projection screen in the adjustment mode, and then the projection screen will be adjusted according to the gesture action input by the user. Among them, since a solid color background can effectively improve the recognition accuracy of gestures, the background content is preferably a solid color background. Of course, the user can also define the background content according to his own preferences.

[0038] Optionally, the projection screen in the adjustment mode can also be displayed based on the current projection screen content in response to the control instruction. For example, the user can enter the non-sensing gesture control adjustment mode through a preset trigger method such as a setting menu or a voice command. In this mode, there is no need to adjust the current projection screen content, that is, there is no need to replace it with a preset solid color background or enter other perceptible operation interfaces. Instead, the current projection screen is directly used as the projection screen in the adjustment mode, and the projection screen is adjusted according to the gesture action input by the user. For example, when the user is watching a projected video, by entering the non-sensing gesture control adjustment mode, the current viewing experience can be continued while adjusting the projection screen.

[0039] It should be noted that the user can also set a specific gesture to enter a corresponding adjustment mode. In this case, it is necessary to detect the gesture action input by the user when the projector starts projecting.

[0040] In some optional embodiments of the present application, when acquiring continuous frame images, the front RGB (red-green-blue) camera of the projector can be used to shoot continuous frame images including the projection image. When shooting, different exposure parameters need to be set for bright environments and dark environments respectively to ensure that clear images of the projection surface can be captured in both bright and dark environments. Among them, the camera can also be used as an accessory and placed at any position in the user's projection environment, as long as the hand gestures can be clearly captured, but in this case, the positional relationship between the camera and the projector needs to be calibrated by yourself. Optionally, continuous frames of the projection surface can also be shot by an infrared camera. Since it can form images well in bright or dark environments, the influence of the projection image on the hand image can be effectively filtered out.

[0041] Step S104 , determining the gesture category of the input gesture according to the continuous frame images, and determining control information corresponding to the gesture category.

[0042] In some optional embodiments of the present application, the user can use dynamic gestures to adjust the projection screen, or use static gestures to adjust the projection screen. Therefore, when determining the gesture category of the input gesture and the control information corresponding to the gesture category, the continuous frame images can be detected using a dynamic gesture detection model or a static gesture detection model according to the gesture control method set by the user.

[0043] Specifically, when detecting dynamic gestures, a first preset number of images whose shooting time is closest to the current system time can be selected from the continuous frame images, and a second preset number of images can be selected at equal intervals from the first preset number of images as the target image set; the target image set is input into a pre-trained dynamic gesture detection model, and the dynamic gesture category of the input gesture is detected by the dynamic gesture detection model. The first preset number and the second preset number can both be pre-set based on experimental data, and can be adjusted later according to actual conditions. For example, the first preset number can be 30, and the second preset number can be 8.

[0044] Among them, the dynamic gesture detection model is obtained through the following training process: obtaining a set of sample video data, each sample video data corresponds to a dynamic gesture category; for any sample video data, selecting a second preset number of images at equal intervals from the sample video as a sample image subset, and marking the dynamic gesture category corresponding to the sample image subset to obtain first marking information; inputting the sample image subset and the first marking information into the dynamic gesture detection model for training.

[0045] For example, first collect 6000 groups of 1s videos as sample video data sets, which include 1000 groups of action videos of the palm from closed to open, the palm from open to closed, the palm pushing to the left, the palm pushing to the right, the palm lifting up, and the palm pressing down; extract 8 frames of images at equal intervals from each group of 1s videos as a sample image subset, and label them with corresponding categories, such as numbers 1-6 corresponding to 6 gestures, to obtain the first labeling information. At this time, the data set obtained is 6000 groups of data, each group of data is 8 frames of images and corresponding labeling information, and the data set is input into the dynamic gesture detection model for continuous training, so that the final dynamic gesture detection model can detect any of the above gestures contained in the image. It should be noted that the above 6 gestures are only for illustration, and can also be replaced by other gestures. Of course, the number of gestures can also be adjusted according to needs.

[0046] In actual detection, 8 frames of images can be selected at equal intervals from the most recently taken 30 frames as the target image set, and input into the trained dynamic gesture detection model to detect the dynamic gesture category. The dynamic gesture detection model can use a video classification network model such as C3D and TSN based on a spatiotemporal convolutional network structure, or a network model based on hand and limb bone points for dynamic recognition.

[0047] Optionally, in addition to using camera image recognition technology, the above dynamic gesture detection process can also be detected by other wireless sensor gesture recognition devices.

[0048] In some optional embodiments of the present application, when the dynamic gesture category is detected to be the first category, the control information is determined to translate the entire projection screen in the target direction by a target distance; when the dynamic gesture category is detected to be the second category, the control information is determined to scale the entire projection screen to a target ratio.

[0049] Specifically, the first category of dynamic gestures mainly includes gestures such as pushing the palm to the left, pushing the palm to the right, lifting the palm up, and pressing the palm down, which are used to control the overall translation of the projected image to the target direction by a target distance; the second category of dynamic gestures mainly includes gestures such as the palm changing from closed to open, and the palm changing from open to closed, which are used to control the overall zooming of the projected image to the target ratio. For example, when a gesture of pushing the palm to the left is detected, the projected image is translated to the left by a target distance as a whole; when a gesture of pushing the palm to the right is detected, the projected image is translated to the right by a target distance as a whole; when a gesture of lifting the palm up is detected, the projected image is translated upward by a target distance as a whole; when a gesture of pressing the palm down is detected, the projected image is translated downward by a target distance as a whole; when a gesture of closing the palm to opening is detected, the projected image is enlarged by a target ratio as a whole; when a gesture of opening the palm to closing is detected, the projected image is reduced by a target ratio as a whole. Figure 2 This is a schematic diagram of the overall translation of the projection image up, down, left, and right to the target distance. Figure 3 This is a schematic diagram showing how the entire projection image can be enlarged or reduced.

[0050] There are multiple ways to determine the target distance and target ratio, which can be selected by the user according to their needs. The following are several optional solutions provided in the embodiments of the present application:

[0051] In the first solution, the target distance is directly determined as a preset fixed distance, and the target ratio is determined as a preset fixed ratio. In order to ensure that the user has a sense of continuity when the projection screen is panned or zoomed, the fixed distance or fixed ratio can be set slightly smaller. For example, the user pre-sets the fixed distance to D = 5cm and the fixed ratio to P = 5%. When the gesture of the palm pushing to the left is detected, the entire projection screen is panned to the left by 5cm; when the gesture of the palm from closing to opening is detected, the entire projection screen is enlarged to 105% of the original.

[0052] In the second scheme, it is necessary to first determine the first frame image and the last frame image in the continuous frame image; determine the hand position in the first frame image and the hand position in the last frame image through the target detection model, wherein the hand position includes at least one of the following: the hand detection frame position, the hand key point position; determine the target distance or target ratio based on the hand position in the first frame image and the hand position in the last frame image.

[0053] It can be understood that in order to determine the target distance or target ratio based on the change information of the hand position, it is necessary to first determine the starting position and the ending position of the hand movement. Therefore, the first frame image and the last frame image in the continuous frame image corresponding to the dynamic gesture category are first determined, and then input into the target detection model to determine the hand position information therein.

[0054] The target detection model can directly adopt the existing hand detection model, or can be obtained through the following training process: obtain a sample image set, and annotate the hand position in each image in the sample image set to obtain annotation information; input the sample image set and the annotation information into the target detection model for training. For example, 2000 hand images are obtained as the sample image set, and the hand position in each image is annotated, such as annotating the hand detection frame position coordinates (x, y, w, h), where x, y, w, h correspond to the horizontal coordinates, vertical coordinates, width and height of the upper left point of the hand detection frame, respectively, to obtain annotation information, and the key point position coordinates of the hand can also be annotated, such as annotating the palm position coordinates (x1, y1), the five finger position coordinates (x2, y2), (x3, y3), (x4, y4), (x5, y5), (x6, y6), etc., to obtain corresponding annotation information; input the sample image set and the corresponding annotation information into the target detection model for continuous training, so that the target detection model finally obtained can detect the hand position information in any image.

[0055] Afterwards, when determining the target distance, the first distance between the center point position of the hand detection frame in the first frame image and the center point position of the hand detection frame in the last frame image is determined, and the target distance is determined based on the first distance. For example, the coordinates of the center point of the hand detection frame in the first frame image are determined to be (x1, y1), and the coordinates of the center point of the hand detection frame in the last frame image are determined to be (x2, y2). Based on the two coordinates, the Euclidean distance d between the two center points at the start of the gesture action is calculated as the first distance, and based on the mapping relationship between the hand translation distance d and the projection screen translation distance D preset by the user, the target distance D to be translated by the projection screen is determined.

[0056] Optionally, a second distance between the palm position in the hand key point in the first frame image and the palm position in the hand key point in the last frame image is determined, and the target distance is determined based on the second distance. For example, the palm coordinates in the first frame image are determined to be (x1', y1'), and the palm coordinates in the last frame image are determined to be (x2', y2'), and the Euclidean distance d between the palms at the start of the gesture action is calculated based on the two coordinates as the second distance, and then the target distance D to be translated by the projection screen is determined based on a preset mapping relationship.

[0057] When determining the target ratio, the first ratio relationship between the hand detection frame in the first frame image and the hand detection frame in the last frame image is determined, and the target ratio is determined according to the first ratio relationship. For example, the coordinates of the hand detection frame in the first frame image are determined to be (x1, y1, w1, h1), and the coordinates of the hand detection frame in the last frame image are determined to be (x2, y2, w2, h2). The scaling ratio p of the hand detection frame at the start of the gesture action is calculated as the first ratio relationship according to the width and height information therein, and the target ratio P to be scaled for the projection screen is determined according to the mapping relationship between the hand scaling ratio p and the projection screen scaling ratio P preset by the user. If the user's hand is initially in a closed state, the detection frame is relatively small at this time. As the user's hand gradually opens, the detection frame will gradually increase. In this process, the projection screen will be enlarged according to the expansion of the user's hand detection frame; if the user's hand is initially in an open state, the detection frame is relatively large at this time. As the user's hand gradually closes, the detection frame will gradually decrease. The projection screen will also decrease according to the decrease of the user's hand detection frame.

[0058] Optionally, determine the second proportional relationship between the circumscribed rectangle of the hand key points in the first frame image and the circumscribed rectangle of the hand key points in the last frame image, and determine the target ratio based on the second proportional relationship. For example, determine that the coordinates of the circumscribed rectangle of the hand key points in the first frame image are (x1', y1', w1', h1'), and the coordinates of the circumscribed rectangle of the hand key points in the last frame image are (x2', y2', w'2', h2'), calculate the scaling ratio p of the circumscribed rectangle of the hand key points at the start of the gesture action based on the width and height information as the second proportional relationship, and then determine the target ratio P of the projection screen to be scaled based on the preset mapping relationship. When the user's hands are closed and clenched, the key points are closed together, and the circumscribed rectangle of the key points becomes smaller, and the projection screen is reduced accordingly; when the user's hands are opened, the key points are stretched together, and the circumscribed rectangle of the key points becomes larger, and the projection screen is enlarged accordingly.

[0059] In some optional embodiments of the present application, when determining the target distance or target ratio, the following third solution can be obtained based on the second solution.

[0060] When determining the target distance, the first speed of the hand movement is determined according to the number of image frames of the continuous frame images and the first distance, and the target distance is determined according to the first speed; or the second speed of the hand movement is determined according to the number of image frames of the continuous frame images and the second distance, and the target distance is determined according to the second speed. For example, the number of image frames from the first frame image to the last frame image is determined to be t, and the first distance or the second distance is d, then the first speed or the second speed of the hand movement is obtained as v=d / t, and the target distance D to be translated by the projection screen is determined according to the mapping relationship between the hand movement speed v and the projection screen translation distance D preset by the user.

[0061] When determining the target ratio, the third speed of hand zooming is determined based on the image frame number of consecutive frame images and the first proportional relationship, and the target ratio is determined based on the third speed; or the fourth speed of hand zooming is determined based on the image frame number of consecutive frame images and the second proportional relationship, and the target ratio is determined based on the fourth speed. For example, the image frame number from the first frame image to the last frame image is determined to be t, and the first proportional relationship or the second proportional relationship is p, then the third speed or the fourth speed of hand zooming is obtained as v=p / t, and the target ratio P of the projection screen to be zoomed is determined based on the mapping relationship between the hand zooming speed v and the projection screen zoom ratio P preset by the user.

[0062] When detecting static gestures, any one of the continuous frame images can be determined as a target image; the target image is input into a pre-trained static gesture detection model, and the static gesture category of the gesture in the target image is detected by the static gesture detection model.

[0063] Among them, the static gesture detection model is obtained through the following training process: obtaining a sample image set, and annotating the static gesture category in each image in the sample image set to obtain second annotation information; inputting the sample image set and the second annotation information into the static gesture detection model for training.

[0064] For example, 7,000 images are collected as a sample image set, including 1,000 images of fingertips to the left, fingertips to the right, fingertips up, fingertips down, open palm, clenched palm, and OK gestures. In order to better simulate the user's usage environment, the collected images may include images in bright environments and images in dark environments; then, the static gesture categories in each image are labeled using a labeling tool, such as numbers 1-7 correspond to 7 static gesture categories, and the second labeling information is obtained; the sample image set and the corresponding second labeling information are input into the static gesture detection model for continuous training, so that the final static gesture detection model can detect the static gesture category in any image. It should be noted that the above 7 static gestures are only for illustration and can be replaced by other static gestures. Of course, the number of static gestures can also be adjusted according to needs.

[0065] In some optional embodiments of the present application, when the static gesture category is detected as the third category, the control information is determined to translate the entire projection screen in the target direction by a target distance, wherein the target distance is a pre-set fixed distance; when the static gesture category is detected as the fourth category, the control information is determined to scale the entire projection screen to a target ratio, wherein the target ratio is a pre-set fixed ratio; when the static gesture category is detected as the fifth category, the control information is determined to stop adjusting the projection screen.

[0066] Specifically, the third category of static gestures mainly includes static gestures such as fingertips to the left, fingertips to the right, fingertips up, and fingertips down, which are used to control the overall translation of the projection picture to the target direction and the target distance; the fourth category of static gestures mainly includes static gestures such as open palm and clenched palm, which are used to control the overall zoom ratio of the projection picture; the fifth category of static gestures mainly includes OK gestures, etc., which are used to indicate that the adjustment of the projection picture is completed.

[0067] For example, when a gesture with the fingertips pointing to the left is detected in the target image, it is determined that the overall projection screen will be translated 5 cm to the left; when a gesture with the fingertips pointing to the right is detected in the target image, it is determined that the overall projection screen will be translated 5 cm to the right; when a gesture with the fingertips pointing upward is detected in the target image, it is determined that the overall projection screen will be translated 5 cm upward; when a gesture with the fingertips pointing downward is detected in the target image, it is determined that the overall projection screen will be translated 5 cm downward; when a gesture with an open palm is detected in the target image, it is determined that the overall projection screen will be enlarged by 5%; when a gesture with a clenched palm is detected in the target image, it is determined that the overall projection screen will be reduced by 5%; when an OK gesture is detected, it is determined that the adjustment of the projection screen will be ended and the projection screen will no longer change.

[0068] Step S106, adjusting the position or size of the projection image according to the gesture type and the control information.

[0069] In some optional embodiments of the present application, the target direction and target distance of the projection image to be translated, or the target ratio of the projection image to be scaled can be determined based on the gesture category and control information; the position coordinates of the image to be projected in the projector can be adjusted based on the target direction and target distance, or based on the target ratio; and the image to be projected is projected based on the adjusted position coordinates of the image to be projected in the projector to obtain the adjusted target projection image.

[0070] Specifically, to adjust the position or size of the projection screen, it can be achieved by adjusting the position coordinates of the screen to be projected in the projector. Since the above process has determined the target direction and target distance of the projection screen to be translated, or the target ratio of the projection screen to be scaled according to the gesture category and control information, the position coordinates of the screen to be projected in the projector can be adjusted through this information. For example, when adjusting the position of the projection screen, the screen to be projected in the projector can be translated as a whole in the target direction by the corresponding pixels based on the mapping relationship between the translation distance of the projection screen and the translation pixels of the screen to be projected in the projector, so as to achieve the translation of the projection screen in the target direction by the target distance. When scaling the size of the projection screen, the screen to be projected in the projector can be directly scaled as a whole by the target ratio to achieve the scaling of the projection screen to the target ratio.

[0071] Optionally, the target direction and target distance of the projected image to be translated can be determined based on the gesture category and control information, and the projected image can be translated in the target direction and target distance by optical mechanical axis shifting. The user can pre-set the mapping relationship between the translation distance of the projected image and the step length of the projector's axis shift motor. After determining the target direction and target distance of the projected image to be translated, the projector's axis shift motor can be controlled to rotate in the corresponding direction by the corresponding step length.

[0072] In some optional embodiments of the present application, a static gesture detection model can also be trained to detect the hand position in the image, and the static gesture detection model is used to detect the static gesture category and hand position in each frame of the captured image in real time. When the third category gesture is continuously detected and the hand position changes, the difference (dx, dy, 1) between the hand position in the current frame image and the hand position in the first frame image where the third category gesture appears is calculated, and at the same time, a homography matrix H (i.e., the conversion matrix of the optical machine in the camera and the projector) between the projection picture in the captured image and the actual projection picture is established. The position to which the actual projection picture is to be translated is calculated by the homography matrix H (dx', dy', dz') = (dx, dy, 1) * H, where dx' = dx' / dz', dy' = dy' / dz'. When the overall projection picture is translated by dx', dy' in the horizontal and vertical directions respectively on the basis of the initial position, the projection picture can be translated in real time following the hand movement.

[0073] In some optional embodiments of the present application, when it is detected that the projection picture is adjusted to a critical position, the adjustment of the projection picture is stopped and a first message is generated, the first message being used to prompt that the projection picture has been adjusted to the critical position. It can be understood that whether it is optical mechanical shifting or adjusting the picture to be projected in the projector, there is an adjustment range, and when it is adjusted to the critical position of the adjustment range, the projection picture can no longer be moved or scaled, and a toast message can be popped up to prompt the user that the projection picture can no longer be adjusted.

[0074] Optionally, when it is detected that the projection screen is adjusted to coincide with the boundary of an obstacle in the projection area, the adjustment of the projection screen is stopped, and a second message is generated, where the second message is used to prompt that the projection screen encounters an obstacle. For example, when the user controls the movement or scaling of the projection screen through gestures, when the projection screen overlaps with the boundary of obstacles such as switches or stickers in the projection area, or moves to the boundary of the projection area, in order to ensure the user's viewing experience, it is necessary to stop adjusting the projection screen and pop up a toast message to prompt the user that the projection screen encounters an obstacle.

[0075] Figure 4 A schematic diagram of an optional projection picture adjustment process provided in an embodiment of the present application, the process includes:

[0076] 1. Collect 6000 sets of 1s videos as sample video data sets;

[0077] 2. Label the gesture actions corresponding to each sample video data to obtain labeling information;

[0078] 3. Input the sample video data set and annotation information into the dynamic gesture detection model for training;

[0079] 4. The camera captures a set of 30 consecutive target images of the projection area;

[0080] 5. Input the target image set into the trained dynamic gesture detection model;

[0081] 6. The dynamic gesture detection model detects the gesture actions corresponding to the target image set;

[0082] 7. Adjust the projection image according to gestures.

[0083] In the embodiment of the present application, firstly, continuous frame images are obtained, wherein each frame image includes a projection screen and an input gesture; then, the gesture category of the input gesture is determined based on the continuous frame images, and the control information corresponding to the gesture category is determined; finally, the position or size of the projection screen is adjusted based on the gesture category and the control information. The embodiment of the present application analyzes the user's gestures to understand the meaning that the user wants to express, allowing the user to adjust the position and size of the projection screen through their own gestures, thereby enhancing the experience and fun of personal participation, and thus solving the technical problem of poor user interaction experience when using a remote control to adjust the projection screen in the related art.

[0084] Example 2

[0085] According to an embodiment of the present application, a projection picture adjustment device for implementing the above projection picture adjustment method is also provided. Figure 5 As shown, the device includes an acquisition module 50, a determination module 52 and an adjustment module 54, wherein:

[0086] The acquisition module 50 is used to acquire continuous frame images, wherein each frame image includes a projection picture and an input gesture.

[0087] In some optional embodiments of the present application, when the user adjusts the projection screen, he can enter different adjustment modes through different control instructions. In each adjustment mode, the corresponding projection screen will be displayed, and the user can adjust the position or size of the projection screen in the adjustment mode. Specifically, the projection screen in the adjustment mode can be displayed based on the preset background content in response to the control instruction. For example, the user can enter the gesture control interface adjustment mode through a preset trigger method such as a setting menu or a voice command. In this mode, the display content in the current projection screen will be replaced with the preset background content to obtain the projection screen in the adjustment mode, and then the projection screen will be adjusted according to the gesture action input by the user. Optionally, the projection screen in the adjustment mode can also be displayed based on the current projection screen content in response to the control instruction. For example, the user can enter the non-sensing gesture control adjustment mode through a preset trigger method such as a setting menu or a voice command. In this mode, there is no need to adjust the current projection screen content, that is, there is no need to replace it with a preset solid color background or enter other perceptible operation interfaces, but directly use the current projection screen as the projection screen in the adjustment mode, and adjust the projection screen according to the gesture action input by the user.

[0088] When acquiring continuous frame images, the front RGB camera of the projector can be used to shoot continuous frames including the projection screen. When shooting, different exposure parameters can be set for bright environments and dark environments respectively to ensure that clear images of the projection surface can be captured in both bright and dark environments. Among them, the camera can also be used as an accessory and placed at any position in the user's projection environment, as long as the hand gestures can be clearly captured, but in this case, the positional relationship between the camera and the projector needs to be calibrated by yourself. Optionally, continuous frames of the projection surface can also be shot by an infrared camera. Since it can form good images in bright or dark environments, it can effectively filter out the impact of the projection image on the hand image.

[0089] The determination module 52 is used to determine the gesture category of the input gesture according to the continuous frame images, and determine the control information corresponding to the gesture category.

[0090] In some optional embodiments of the present application, the user can use dynamic gestures to adjust the projection screen, or use static gestures to adjust the projection screen. Therefore, when determining the gesture category of the input gesture and the control information corresponding to the gesture category, the continuous frame images can be detected using a dynamic gesture detection model or a static gesture detection model according to the gesture control method set by the user.

[0091] Specifically, when detecting dynamic gestures, a first preset number of images whose shooting time is closest to the current system time can be selected from the continuous frame images, and a second preset number of images can be selected at equal intervals from the first preset number of images as the target image set; the target image set is input into a pre-trained dynamic gesture detection model, and the dynamic gesture category of the input gesture is detected by the dynamic gesture detection model. When it is detected that the dynamic gesture category is the first category, the control information is determined to translate the entire projection screen in the target direction by a target distance; when it is detected that the dynamic gesture category is the second category, the control information is determined to scale the entire projection screen to a target ratio.

[0092] The target distance may be a preset fixed distance, and the target ratio may be a preset fixed ratio. Optionally, the first frame image and the last frame image in the continuous frame image may be determined first; the hand position in the first frame image and the hand position in the last frame image may be determined by the target detection model, wherein the hand position includes at least one of the following: the hand detection frame position, the hand key point position; the target distance or target ratio may be determined based on the hand position in the first frame image and the hand position in the last frame image.

[0093] When detecting static gestures, any image in the continuous frame images can be determined as the target image; the target image is input into a pre-trained static gesture detection model, and the static gesture category of the gesture in the target image is detected by the static gesture detection model. When the static gesture category is detected as the third category, the control information is determined to translate the entire projection screen in the target direction by a target distance, wherein the target distance is a preset fixed distance; when the static gesture category is detected as the fourth category, the control information is determined to scale the entire projection screen by a target ratio, wherein the target ratio is a preset fixed ratio; when the static gesture category is detected as the fifth category, the control information is determined to stop adjusting the projection screen.

[0094] The adjustment module 54 is used to adjust the position or size of the projection image according to the gesture type and the control information.

[0095] In some optional embodiments of the present application, the target coordinate information after adjusting the position or size of the projection screen can be determined based on the gesture category and control information; the homography matrix H between the screen to be projected in the projector and the projection screen is determined; the position coordinates of the screen to be projected in the projector are adjusted based on the target coordinate information and the homography matrix; the projected screen is projected based on the position coordinates of the adjusted screen to be projected in the projector to obtain the adjusted target projection screen. Optionally, the target position after adjusting the position of the projection screen can also be determined based on the gesture category and control information; the projection screen is adjusted to the target position by optical mechanical shifting.

[0096] In some optional embodiments of the present application, when it is detected that the projection picture is adjusted to a critical position, the projection picture is stopped from being adjusted, and a first message is generated, the first message is used to prompt that the projection picture has been adjusted to the critical position. When it is detected that the projection picture is adjusted to coincide with the boundary of an obstacle in the projection area, the projection picture is stopped from being adjusted, and a second message is generated, the second message is used to prompt that the projection picture encounters an obstacle.

[0097] It should be noted that each module in the projection image adjustment device in the embodiment of the present application corresponds one by one to the implementation steps of the projection image adjustment method in Example 1. Since a detailed description has been given in Example 1, some details not reflected in this embodiment can be referred to Example 1 and will not be repeated here.

[0098] Example 3

[0099] According to an embodiment of the present application, a projector is also provided, such as Figure 6 As shown, the projector at least includes a processor 60 and a memory 62, wherein the memory 62 is connected to the processor 60 and is used to provide the processor 60 with instructions for processing the following processing steps:

[0100] Acquire continuous frame images, wherein each frame image includes a projection screen and an input gesture; determine a gesture category of the input gesture based on the continuous frame images, and determine control information corresponding to the gesture category; and adjust a position or size of the projection screen based on the gesture category and the control information.

[0101] Example 4

[0102] According to an embodiment of the present application, a non-volatile storage medium is also provided, which includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the above method.

[0103] Optionally, when the program is running, the device where the non-volatile storage medium is located is controlled to execute the following steps: obtaining continuous frame images, wherein each frame image includes a projection screen and an input gesture; determining a gesture category of the input gesture based on the continuous frame images, and determining control information corresponding to the gesture category; and adjusting the position or size of the projection screen based on the gesture category and the control information.

[0104] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0105] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0106] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0107] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0108] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0109] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0110] The above are only preferred implementations of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A projection picture adjustment method, characterized in that: include: Acquire continuous frame images, wherein each frame image includes a projection image and an input gesture; Determining a gesture category of the input gesture according to the continuous frame images, and determining control information corresponding to the gesture category; Adjusting the position or size of the projection image according to the gesture category and the control information; Wherein, the gesture category includes: a dynamic gesture category or a static gesture category; Determining control information corresponding to the gesture category includes: when it is detected that the dynamic gesture category is the first category, determining that the control information is to translate the entire projection screen in a target direction by a target distance; when it is detected that the dynamic gesture category is the second category, determining that the control information is to scale the entire projection screen to a target ratio; when it is detected that the static gesture category is the third category, determining that the control information is to translate the entire projection screen in a target direction by a preset fixed distance; when it is detected that the static gesture category is the fourth category, determining that the control information is to scale the entire projection screen to a preset fixed ratio; when it is detected that the static gesture category is the fifth category, determining that the control information is to stop adjusting the projection screen; The process of determining the target distance or the target ratio includes: determining the first frame image and the last frame image in the continuous frame images; determining the hand position in the first frame image and the hand position in the last frame image through a target detection model, wherein the hand position includes at least one of the following: hand detection frame position, hand key point position; determining the target distance or target ratio based on the hand position in the first frame image and the hand position in the last frame image.

2. The method according to claim 1, characterized in that The step of determining the gesture category of the input gesture according to the continuous frame images includes: Selecting a first preset number of images whose shooting time is closest to the current system time from the continuous frame images, and selecting a second preset number of images at equal intervals from the first preset number of images as a target image set; The target image set is input into a pre-trained dynamic gesture detection model, and the dynamic gesture detection model is used to detect the dynamic gesture category of the input gesture.

3. The method according to claim 1, characterized in that The target distance is a preset fixed distance; The target ratio is a preset fixed ratio.

4. The method according to claim 1, characterized in that: The determining the target distance according to the hand position in the first frame image and the hand position in the last frame image includes: Determine a first distance between a center point position of a hand detection frame in the first frame image and a center point position of a hand detection frame in the last frame image, and determine the target distance based on the first distance; or, Determine a first speed of hand movement according to the number of image frames of the continuous frame images and the first distance, and determine the target distance according to the first speed; or, Determine a second distance between the palm position of the hand key points in the first frame image and the palm position of the hand key points in the last frame image, and determine the target distance based on the second distance; or, A second speed of hand movement is determined according to the number of image frames of the continuous frame images and the second distance, and the target distance is determined according to the second speed.

5. The method according to claim 1, characterized in that The determining the target ratio according to the hand position in the first frame image and the hand position in the last frame image includes: Determine a first proportional relationship between the hand detection frame in the first frame image and the hand detection frame in the last frame image, and determine the target ratio according to the first proportional relationship; or, Determine a third speed of hand zooming according to the relationship between the number of image frames of the continuous frame images and the first ratio, and determine the target ratio according to the third speed; or, Determine a second proportional relationship between a circumscribed rectangle of the hand key points in the first frame image and a circumscribed rectangle of the hand key points in the last frame image, and determine the target ratio according to the second proportional relationship; or, A fourth speed of hand zooming is determined according to the image frame number of the continuous frame images and the second proportional relationship, and the target proportion is determined according to the fourth speed.

6. The method according to claim 1, characterized in that The step of determining the gesture category of the input gesture according to the continuous frame images includes: Determine any one of the continuous frame images as a target image; The target image is input into a pre-trained static gesture detection model, and the static gesture category of the gesture in the target image is detected by the static gesture detection model.

7. The method according to claim 1, characterized in that The method further comprises: When it is detected that the projection picture is adjusted to the critical position, the adjustment of the projection picture is stopped, and first information is generated, where the first information is used to prompt that the projection picture has been adjusted to the critical position; or, When it is detected that the projection picture is adjusted to coincide with the boundary of the obstacle in the projection surface, the adjustment of the projection picture is stopped, and second information is generated, where the second information is used to prompt that the projection picture encounters an obstacle.

8. The method according to claim 1, characterized in that The adjusting the position or size of the projection image according to the gesture category and the control information includes: Determine a target direction and a target distance for the projection image to be translated, or determine a target scale for the projection image to be scaled, according to the gesture category and the control information; According to the target direction and the target distance, or according to the target ratio, adjusting the position coordinates of the image to be projected in the projector; Projection is performed according to the position coordinates of the image to be projected after adjustment in the projector to obtain an adjusted target projection image.

9. The method according to claim 1, characterized in that: The adjusting the position of the projection image according to the gesture category and the control information includes: Determining a target direction and a target distance of the projection image to be translated according to the gesture category and the control information; The projection image is translated in the target direction by the target distance through optical mechanical axis shifting.

10. The method according to claim 1, characterized in that Before acquiring the continuous frame images, the method further includes: In response to a control instruction, the projection picture in the adjustment mode is displayed based on a preset background content; or, in response to a control instruction, the projection picture in the adjustment mode is displayed based on the current projection picture content.

11. The method according to claim 2, characterized in that The training process of the dynamic gesture detection model includes: Obtain a set of sample video data, wherein each sample video data corresponds to a dynamic gesture category; For any of the sample video data, selecting a second preset number of images at equal intervals from the sample video as a sample image subset, and marking the dynamic gesture categories corresponding to the sample image subset to obtain first marking information; The sample image subset and the first annotation information are input into the dynamic gesture detection model for training.

12. A projection picture adjustment device, characterized in that: include: An acquisition module, used to acquire continuous frame images, wherein each frame image includes a projection picture and an input gesture; A determination module, used to determine the gesture category of the input gesture based on the continuous frame images, and determine the control information corresponding to the gesture category, wherein the gesture category includes: a dynamic gesture category or a static gesture category; determining the control information corresponding to the gesture category includes: when it is detected that the dynamic gesture category is the first category, determining that the control information is to translate the entire projection image in a target direction by a target distance; when it is detected that the dynamic gesture category is the second category, determining that the control information is to scale the entire projection image to a target ratio; when it is detected that the static gesture category is the third category, determining that the control information is to translate the entire projection image in a target direction by a preset fixed distance; when it is detected that the When the static gesture category is the fourth category, the control information is determined to be to scale the entire projection screen to a preset fixed ratio; when it is detected that the static gesture category is the fifth category, the control information is determined to be to stop adjusting the projection screen; the process of determining the target distance or the target ratio includes: determining the first frame image and the last frame image in the continuous frame images; determining the hand position in the first frame image and the hand position in the last frame image through a target detection model, wherein the hand position includes at least one of the following: the hand detection frame position, the hand key point position; determining the target distance or target ratio based on the hand position in the first frame image and the hand position in the last frame image; The adjustment module is used to adjust the position or size of the projection image according to the gesture category and the control information.

13. A projector, characterized in that: include: processor; as well as A memory connected to the processor, for providing the processor with instructions for processing the following processing steps: acquiring continuous frame images, wherein each frame image includes a projection screen and an input gesture; determining a gesture category of the input gesture based on the continuous frame images, and determining control information corresponding to the gesture category; adjusting the position or size of the projection screen based on the gesture category and the control information; wherein the gesture category includes: a dynamic gesture category or a static gesture category; determining the control information corresponding to the gesture category, including: when it is detected that the dynamic gesture category is a first category, determining that the control information is to translate the entire projection screen in a target direction by a target distance; when it is detected that the dynamic gesture category is a second category, determining that the control information is to scale the entire projection screen to a target ratio; when it is detected that the static gesture category is a third category, determining that the control information is to scale the entire projection screen to a target ratio. When it is detected that the static gesture category is the fourth category, the control information is determined to be to translate the entire projection image in the target direction by a preset fixed distance; when it is detected that the static gesture category is the fifth category, the control information is determined to be to stop adjusting the projection image; the process of determining the target distance or the target ratio includes: determining the first frame image and the last frame image in the continuous frame images; determining the hand position in the first frame image and the hand position in the last frame image through a target detection model, wherein the hand position includes at least one of the following: a hand detection frame position, a hand key point position; determining the target distance or the target ratio according to the hand position in the first frame image and the hand position in the last frame image.

14. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein when the program is executed, the device where the non-volatile storage medium is located is controlled to execute the projection picture adjustment method according to any one of claims 1 to 11.

Images