Projection control method and device, projector and storage medium

Abstract

The invention discloses a projection control method and device, a projector and a storage medium, and the method comprises the steps: obtaining shot images of a camera for shooting an activity region at a plurality of continuous time points in response to determining that a human body carrying object exists in the activity region; determining target object position information corresponding to the human body bearing object in the shot image at the second time point according to the object position information of the human body bearing object in the shot image at the plurality of first time points; and determining a target content position in a projection space of the projector according to the target object position information, and controlling the projector to project projection content according to the target content position in response to arrival of a projection time point corresponding to the second time point. In the application, the acquired position information of the target object refers to the position information of the human body bearing object at the plurality of first time points, so that the acquired position of the human body bearing object is more stable, and the content display on the human body bearing object has good stability.

Description

Technical Field

[0001] The present application relates to the field of projection control technology, and more specifically, to a projection control method, device, electronic device and computer-readable storage medium. Background Art

[0002] Currently, a projection image can be projected onto a designated projection surface to display the projection content on the projection surface. However, when a moving object is used as the projection surface, the projection content may be unstable, which affects the user experience. Summary of the invention

[0003] The present application proposes a projection control method, device, projector and storage medium to improve the above-mentioned defects.

[0004] In a first aspect, an embodiment of the present application provides a projection control method, the method comprising:

[0005] In response to determining that a human-carrying object exists in the activity area, acquiring images of the activity area taken by a camera at a plurality of consecutive time points;

[0006] Determine the target object position information corresponding to the human body-carrying object in the captured images at a second time point according to the object position information of the human body-carrying object in the captured images at a plurality of consecutive first time points; the second time point is any one of the plurality of time points, and the plurality of first time points are the time points before the second time point in the plurality of time points;

[0007] Determine the target content position in the projection space of the projector according to the target object position information;

[0008] In response to the projection time point corresponding to the second time point arriving, the projector is controlled to project the projection content according to the target content position.

[0009] In a second aspect, an embodiment of the present application further provides a projection control device, the device comprising:

[0010] An acquisition module, configured to acquire images of the activity area taken by a camera at a plurality of consecutive time points in response to determining that a human-bearing object exists in the activity area;

[0011] A first determination module is used to determine the target object position information corresponding to the human body-carrying object in the captured images at a second time point according to the object position information of the human body-carrying object in the captured images at a plurality of consecutive first time points; the second time point is any one of the plurality of time points, and the plurality of first time points are the time points before the second time point in the plurality of time points;

[0012] A second determination module, used to determine the target content position in the projection space of the projector according to the target object position information;

[0013] The control module is used to control the projector to project the projection content according to the target content position in response to the projection time point corresponding to the second time point arriving.

[0014] In a third aspect, an embodiment of the present application further provides a projector, characterized in that it includes: one or more processors; a memory; one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by one or more processors, and the one or more programs are configured to execute the above method.

[0015] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores a program code executable by a processor, and when the program code is executed by the processor, the processor executes the above method.

[0016] The present application provides a projection control method, device, projector and storage medium. In the present application, in response to determining that there is a human-bearing object in an activity area, images of the activity area taken by a camera at multiple consecutive time points are obtained; according to the object position information of the human-bearing object in the images taken at multiple consecutive first time points, the target object position information corresponding to the human-bearing object in the images taken at a second time point is determined; the second time point is any time point among the multiple time points, and the multiple first time points are time points before the second time point among the multiple time points, so that the target object position information of the human-bearing object at the second time point obtained refers to the object position information thereof. The position information of the human body-carrying object at the previous multiple first time points effectively avoids the situation where the position of the human body-carrying object is inaccurate due to unstable data at one time point when the position of the human body-carrying object is determined according to the captured image at that time point, and effectively improves the stability of the position of the human body-carrying object at the determined second time point. Afterwards, according to the target object position information, the target content position in the projection space of the projector is determined, and in response to the arrival of the projection time point corresponding to the second time point, the projector is controlled to project the projection content according to the target content position, so that the content display on the human body-carrying object has good stability, thereby ensuring the user experience.

[0017] Other features and advantages of the embodiments of the present application will be described in the subsequent description, and partly become apparent from the description, or can be understood by practicing the embodiments of the present application. The purposes and other advantages of the embodiments of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A flowchart of a projection control method proposed according to an embodiment of the present application is shown.

[0020] Figure 2 A schematic diagram of a key positioning point in an embodiment of the present application is shown.

[0021] Figure 3 A flow chart of a method for calibrating the transformation relationship between the camera space of a camera and the projection space of a projector in an embodiment of the present application is shown.

[0022] Figure 4 A schematic diagram of a second calibration image and a second captured image in an embodiment of the present application is shown.

[0023] Figure 5 A schematic diagram of a third calibration image and a third captured image in an embodiment of the present application is shown.

[0024] Figure 6 Shown according to Figure 4 Schematic diagram of another second captured image obtained from the second calibration image in .

[0025] Figure 7 Shows Figure 5 The second projection image in the figure includes a schematic diagram of a second projection feature point.

[0026] Figure 8 A schematic diagram of a calibration image, a projection area of ​​a projector, and a shooting area of ​​a camera in an embodiment of the present application is shown.

[0027] Fig. 9 A schematic diagram showing the positions of each shooting feature point in the second shot image before and after flipping the positions of each shooting feature point according to the relative direction in an embodiment of the present application is shown.

[0028] Fig.10 A schematic diagram showing the positional relationship between a projector, a camera, an electronic device and a palm in an embodiment of the present application is shown.

[0029] Fig.11 A flowchart of a projection control method proposed according to another embodiment of the present application is shown.

[0030] Fig.12A schematic diagram of a posture key point in an embodiment of the present application is shown.

[0031] Fig.13 A schematic diagram of determining the state of a location state machine at a second time point and information that needs to be updated in the process of determining the state of the location state machine in an embodiment of the present application is shown.

[0032] Fig.14 A schematic diagram of a state process of determining a posture state machine at a second time point in an embodiment of the present application is shown.

[0033] Fig.15 A schematic diagram of a gesture updating process in an embodiment of the present application is shown.

[0034] Fig.16 A flowchart of a projection control method proposed according to another embodiment of the present application is shown.

[0035] Fig.17 A schematic diagram of a third transformation relationship in an embodiment of the present application is shown.

[0036] Fig.18 A schematic diagram of a projection control system in an embodiment of the present application is shown.

[0037] Fig.19 A schematic diagram of a content control module in an embodiment of the present application is shown.

[0038] Fig. 20 A structural block diagram of a projection control device proposed according to an embodiment of the present application is shown.

[0039] Fig.21 A schematic diagram of a projector provided in an embodiment of the present application is shown.

[0040] Fig. 22 A schematic diagram of a computer-readable storage medium provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0041] In order to make those skilled in the art better understand the present application scheme, the technical scheme in the present application embodiment will be clearly and completely described below in conjunction with the drawings in the present application embodiment. Obviously, the described embodiment is only a part of the present application embodiment, rather than all the embodiments. The components of the present application embodiment usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the present application for protection, but merely represents the selected embodiment of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0042] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0043] See also Figure 1 , Figure 1 A flowchart of a projection control method proposed in one embodiment of the present application is shown, which is used in an electronic device. The method includes:

[0044] S101. In response to determining that a human-bearing object exists in an activity area, obtain images of the activity area taken by a camera at a plurality of consecutive time points.

[0045] Among them, the human body carrying object refers to the object used to receive and present the projection content in the projection technology. The object is the entire human body or a part of the human body, such as the face, palm, knee, etc. The active area is the active area of ​​the human body carrying object. The area of ​​the active area is not larger than the shooting area of ​​the camera. There is no restriction on the type of camera here, which can be an RGB camera, infrared camera, Tof camera, etc.

[0046] A time point may be the moment of any frame in the projection process. There may be dozens of frames in one second, and there is no specific limitation.

[0047] In some embodiments, the starting time point is the time point when the presence of a human-bearing object is determined in the activity area. The human-bearing object may be detected by a camera in an image taken of the activity area at a certain time point, and the time point is determined as the starting time point. The starting time point may be the time point when the human-bearing object is detected for the first time after the start of the method of the present application, or it may be the time point when the human-bearing object is re-detected at one or more time points after the human-bearing object is detected.

[0048] It is worth mentioning that, in the present application, the aforementioned multiple continuous time points may include a starting time point. In this case, for example, the starting time point may be the first time point of the multiple continuous time points.

[0049] S102: Determine target object position information corresponding to the human body-carrying object in an image captured at a second time point based on object position information of the human body-carrying object in images captured at a plurality of consecutive first time points.

[0050] Among them, the second time point is any time point among the multiple time points, the multiple first time points are time points before the second time point among the multiple time points, and the time interval between two adjacent time points is the same, which can be set specifically according to needs.

[0051] Among them, the last first time point among the multiple first time points can be the time point before the second time point, or can be two time points before the second time point, which can be set according to needs.

[0052] Among them, the object position information refers to the position information of the human body carrying object in the captured image at a point in time. The object position information can be the actual detected position information of the human body carrying object in the captured image at a point in time, or it can be the assigned position information of the human body carrying object in the captured image at a point in time.

[0053] In some embodiments, S102 may include: if the difference between the object position information of the human body-carrying object in the captured images at multiple first time points and the object position information of the human body-carrying object in the captured images at the second time point is less than a difference threshold, obtaining the object position information of the human body-carrying object in the captured images at the last first time point as the target object position information.

[0054] Among them, the difference between the object position information of the human body carrying object in the captured images at multiple first time points and the object position information of the human body carrying object in the captured images at the second time point can be regarded as the relative distance between the human body carrying object in the captured images at the multiple first time points and the human body carrying object in the captured images at the second time point. Correspondingly, the difference threshold can also be a relative distance threshold. The relative distance threshold can be set according to needs and is not limited here.

[0055] In some embodiments, the coordinates of the target key points on the human body-carrying object can be obtained as the object position information of the human body-carrying object, and the relative distance between the human body-carrying object at a first time point and a second time point can be calculated respectively based on the coordinates of the target key points at multiple first time points and second time points.

[0056] Among them, the target key point can be any point on the human body carrying object. Taking the palm as the human body carrying object as an example, it can be a point on the palm or a point on the finger. The position of the target key point on the human body carrying object remains unchanged at multiple time points.

[0057] In some embodiments, the last first time point among multiple first time points is the time point closest to the second time point. Therefore, when calculating the difference between the object position information of the human body-bearing object in the captured image at the multiple first time points and the object position information of the human body-bearing object in the captured image at the second time point, the difference between the object position information of the human body-bearing object in the captured image at the last first time point among the multiple first time points and the object position information of the human body-bearing object in the captured image at the second time point can be used. As long as the difference between the object position information of the human body-bearing object in the captured image at the last first time point among the multiple first time points and the object position information of the human body-bearing object in the captured image at the second time point is less than the difference threshold, it is considered that the difference between the object position information of the human body-bearing object in the captured image at the multiple first time points and the object position information of the human body-bearing object in the captured image at the second time point is different.

[0058] That is, the relative distance between the target key point and the second key point at the last first time point can be calculated. If the relative distance is less than the relative distance threshold, the object position information of the human body carrying object in the captured image at the last first time point is obtained as the target object position information.

[0059] It is impossible for the human body carrying object to remain absolutely still. Through the above process, it is achieved that after the human body carrying object undergoes a slight position change, the object position information of the human body carrying object before the position change is obtained as the object position information of the human body carrying object after the slight position change occurs. This avoids the problem of jitter in the projection position calculated by the subsequent projection control module due to slight position changes of the human body carrying object, which seriously affects the user experience.

[0060] Since it takes a certain amount of time to detect the human body-carrying object in the captured image and determine the object position information of the human body-carrying object during the processing of the captured image, there is a data processing delay between the time point when the captured image is obtained and the time point when the projection content is projected. Assuming that the data processing delay is x milliseconds, it means that the object position information of the human body-carrying object obtained at the second time point is the object position information x milliseconds ago. In the real world, the human body-carrying object has moved to other positions, causing the projection of the subsequent projector to lag behind the movement of the human body-carrying object, affecting the user experience.

[0061] Therefore, in some embodiments, S102 may include: if the difference between the object position information of the human body-carrying object in the captured images at multiple first time points and the object position information of the human body-carrying object in the captured images at the second time point is not less than a difference threshold, determining the movement speed corresponding to the human body-carrying object according to the difference between the object position information of the human body-carrying object in the captured images at the multiple first time points; determining the predicted displacement based on the movement speed and the data processing delay; the data processing delay is the time difference between the second time point and the corresponding projection time point; determining the target object position information based on the object position information of the human body-carrying object in the captured images at the second time point and the predicted displacement.

[0062] Similarly, the difference and the difference threshold can be understood as the relative distance and the relative distance threshold, and whether the difference between the object position information of the human body-bearing object in the captured image at the last first time point and the object position information of the human body-bearing object in the captured image at the second time point is less than the difference threshold can be used to determine whether the difference between the object position information of the human body-bearing object in the captured image at multiple first time points and the object position information of the human body-bearing object in the captured image at the second time point is less than the difference threshold.

[0063] In some embodiments, the relative distances between objects carried by the human body in images captured at two adjacent time points at multiple first time points can be calculated respectively, and the movement speeds corresponding to the objects carried by the human body can be determined based on the calculated multiple relative distances and the time intervals corresponding to the multiple relative distances.

[0064] The movement speed of the object carried by the human body may be the sum of multiple relative distances divided by the sum of multiple time intervals, or may be a reference speed obtained by dividing a relative distance by a time interval, or an average speed of multiple reference speeds.

[0065] In some implementations, the target object position information may be obtained through a Kalman filter algorithm based on the object position information of the object carried by the human body in the captured image at the second time point and the predicted displacement.

[0066] Through the above process, the predicted displacement is determined to correct the position of the human-carrying object detected at the second time point, that is, the movement trajectory of the human-carrying object is predicted, and the accurate position of the human-carrying object at the projection time point corresponding to the second time point is obtained, which solves the problem that the projection of the projector lags behind the movement of the human-carrying object and ensures the user experience.

[0067] In some embodiments, before S102, the method further includes: for any one of the multiple time points, if no human-bearing object is detected in the captured image at the time point, and the time difference between the first reference time point corresponding to the captured image in which the human-bearing object was detected last time and the time point is less than a time difference threshold, obtaining the object position information of the human-bearing object in the captured image at the first reference time point as the object position information of the human-bearing object in the captured image at the time point.

[0068] Among them, any time point under the multiple time points can be the first time point or the second time point, and the time difference threshold can be set according to the needs, and is not specifically limited. For example, the time difference threshold can be a time interval greater than the time interval and less than twice, that is, the first reference time point is limited to the time point before the time point.

[0069] Taking the example where the time difference threshold is greater than the time interval and less than twice the time interval, the time point is the second time point, and the last first time point is the time point before the second time point, if the human body-bearing object is not detected in the captured image at the second time point, and the human body-bearing object is detected in the captured image at the last first time point among multiple first time points, the object position information of the human body-bearing object in the captured image at the last first time point is obtained as the object position information of the human body-bearing object in the captured image at the second time point, that is, the target object position information.

[0070] In some embodiments, the captured image can be sent to a detection device, and the detection device can detect whether there is a positioning key point in the captured image to determine whether there is a human body-bearing object in the captured image. The positioning key point can be a landmark point or any point on the human body-bearing object; Figure 2 As shown, taking the palm as the human body bearing object as an example, the positioning key points can be Figure 2 Shown as black circle.

[0071] In some implementations, the target key points mentioned above may be positioning key points.

[0072] If no human-carrying object is detected in the captured image at a time point, and the time difference between the first reference time point corresponding to the captured image in which a human-carrying object was detected last time and the time point is not less than the time difference threshold, it is determined that no human-carrying object exists in the captured image at the time point.

[0073] Still taking the example where the time difference threshold is greater than the time interval and less than twice the time interval, the time point is the second time point, and the last first time point is the time point before the second time point, if no human-bearing object is detected in the captured image at the second time point, the first time point of the captured image in which the human-bearing object was previously detected is not the last first time point, that is, the human-bearing object is not detected in the captured image at the last first time point, it is determined that no human-bearing object exists in the captured image at the second time point.

[0074] Through the above process, the problem of missed detection when detecting objects carried by the human body in the captured image is solved, and flickering of the projected content due to missed detection is avoided.

[0075] In some embodiments, before S102, the method further includes: for any one of the multiple time points, if there are multiple position information of the object to be detected of the human body-carrying object in the captured image at the time point, determining, from the multiple position information of the object to be detected, the position information of the object to be detected that has the smallest difference with the object position information of the human body-carrying object in the captured image at a second reference time point before the time point, as the object position information of the human body-carrying object in the captured image at the time point.

[0076] Among them, there are multiple position information of objects to be detected of the human body carrying object in the captured image at a certain time point. It can be that there are multiple human body carrying objects in the captured image, and the position of each human body detection object is used as a position information of an object to be detected; or there is one human body carrying object in the captured image, and multiple position information of objects to be detected is detected for the human body detection object.

[0077] Similarly, the difference can be understood as a relative distance, and the smallest corresponding difference means the smallest relative distance. That is, among multiple pieces of position information of objects to be detected, the position information of the detected object with the smallest relative distance to the object position information of the human body-carrying object in the captured image at the second reference time point before the time point is retained as the object position information of the human body-carrying object in the captured image at the time point, and other position information of objects to be detected is discarded.

[0078] Through the above process, it is achieved that a unique human-bearing object can be tracked at each time point when the human-bearing object is detected.

[0079] S103: Determine the target content position in the projection space of the projector according to the target object position information.

[0080] The target content position is the content position of the projection content in the projection space when the projection content is projected onto the human body bearing object.

[0081] In some embodiments, the target object indication area can be determined according to the target object position information and the projection requirement, wherein the projection requirement includes the target content display area on the human body carrying object where the projection content is projected, such as the palm of the hand, and the target object indication area is the area of ​​the target content display area on the human body carrying object in the camera's captured image.

[0082] In some implementations, according to a pre-calibrated transformation relationship between the camera space of the camera and the projection space of the projector, the target object indication area in the camera space can be transformed into the projection space to obtain a corresponding target content position.

[0083] In some embodiments, Figure 3 As shown, the method for calibrating the transformation relationship between the camera space of the camera and the projection space of the projector specifically includes S201-S203, which are specifically as follows:

[0084] S201: Obtain the relative direction between the camera and the projector.

[0085] The relative direction indicates the flipping situation between the captured image acquired by the camera and the image projected by the projector. The flipping situation can be no flipping, only vertical flipping, only horizontal flipping, and both vertical flipping and horizontal flipping. Usually, the relative direction between the camera and the projector remains unchanged.

[0086] Among them, horizontal flipping means that the image is rotated 180 degrees with the horizontal axis (X axis) as the axis. Specifically, it is to flip the image 180 degrees from left to right or from right to left, so that the content originally on the left appears on the right, and the content originally on the right appears on the left. Vertical flipping means that the image is rotated 180 degrees with the vertical axis (Y axis) as the axis. Specifically, it is to flip the image 180 degrees from top to bottom or from bottom to top.

[0087] In some embodiments, S201 includes A1-A3, which are as follows:

[0088] A1. When the projector projects the second calibration image, a second captured image corresponding to the second calibration image is acquired through a camera.

[0089] The second projection feature points in the second calibration image are symmetrically distributed about the first symmetry axis; the first symmetry axis is the vertical center axis of the second calibration image; the pixel values ​​of the second projection feature points on both sides of the first symmetry axis are different. There are at least two second projection feature points. If there are two second projection feature points, the pixel values ​​of the two second projection feature points are different. If there are four second projection feature points, the pixel values ​​of the second projection feature points on both sides of the first symmetry axis are different. It is not limited whether the pixel values ​​of the two second feature points on the same side are the same.

[0090] Take the second calibration image as a black and white chessboard image as an example, Figure 4 As shown, Figure 4 310 in is the first symmetry axis, Figure 4 (a) in the figure is the second calibration image. The gray points A and B in (a) are the second projection feature points. Figure 4 (b) in the figure is the second captured image obtained by taking the second calibration image by the camera. For the convenience of distinction, the captured feature points corresponding to the second projection feature points in the second captured image are called second captured feature points, that is, the gray points A' and B' in (b) are called first captured feature points, point A' is the second captured feature point corresponding to point A, and point B' is the second captured feature point corresponding to point B.

[0091] A2. When the projector projects the third calibration image, a third captured image corresponding to the third calibration image is acquired by the camera.

[0092] The third projection feature points in the third calibration image are symmetrically distributed about the second symmetry axis; the second symmetry axis is the horizontal center axis of the second calibration image; the pixel values ​​of the third projection feature points on both sides of the second symmetry axis are different. There are at least two third projection feature points. If there are two third projection feature points, the pixel values ​​of the two third projection feature points are different. If there are four third projection feature points, the pixel values ​​of the third projection feature points on both sides of the second symmetry axis are different. It is not limited whether the pixel values ​​of the two third feature points on the same side are the same.

[0093] Take the third calibration image as a black and white chessboard image as an example, Figure 5 As shown, Figure 5 410 in is the first straight line, Figure 5 420 in is the second straight line, Figure 5 (a) in (a) is the third calibration image, and the gray points C and D in (a) are the third projection feature points. Figure 5 (b) in the figure is the third captured image obtained by taking the third calibration image by the camera. For the convenience of distinction, the capturing feature points corresponding to the third projection feature points in the third captured image are called third capturing feature points, that is, the gray points C' and D' in (b) are called third capturing feature points, point C' is the third capturing feature point corresponding to point C, and point D' is the third capturing feature point corresponding to point D.

[0094] A3. Determine the relative direction between the camera and the projector according to the pixel values ​​of each second projection feature point and the pixel values ​​of each second shooting feature point corresponding to each second projection feature point in the second captured image, the pixel values ​​of each third projection feature point and the pixel values ​​of each third shooting feature point corresponding to each third projection feature point in the third captured image.

[0095] The relative direction indicates the flipping condition between the captured image acquired by the camera and the image projected by the projector. The relative direction includes a first relative direction and a second relative direction, the first relative direction indicates the horizontal flipping condition between the captured image acquired by the camera and the image projected by the projector, and the second relative direction indicates the vertical flipping condition between the captured image acquired by the camera and the image projected by the projector.

[0096] Wherein, whether there is a flip between the captured image acquired by the camera and the image projected by the projector and whether the flip is horizontal flip and / or vertical flip, it does not affect the positions of the second captured feature points corresponding to the second projection feature points in the second captured image and the positions of the third captured feature points corresponding to the third projection feature points in the third captured image. That is, Figure 4 (b) is the second captured image obtained by the camera shooting (a) without flipping, point A' is the first captured feature point corresponding to the second projection feature point A, point B' is the first captured feature point corresponding to the second projection feature point B, Figure 6 It is the second captured image obtained by the camera shooting (a) with horizontal flipping, point A" is the first captured feature point corresponding to the second projection feature point A, and point B" is the first captured feature point corresponding to the second projection feature point B.

[0097] In some embodiments, S203 may include: determining the relative direction between the camera and the projector based on the comparison results of the pixel values ​​of each second projection feature point and the relationship between the comparison results of the pixel values ​​of each second shooting feature point corresponding to each second projection feature point in the second captured image, the comparison results of the pixel values ​​of each third projection feature point and the relationship between the comparison results of the pixel values ​​of each third shooting feature point in the third captured image corresponding to each third projection feature point in the third captured image.

[0098] Specifically, the first relative direction between the camera and the projector is determined according to the comparison result of the pixel values ​​of each second projection feature point and the relationship between the comparison results of the pixel values ​​of each second shooting feature point corresponding to each second projection feature point in the second shot image.

[0099] If the comparison result of the pixel values ​​of each second projection feature point is the same as the comparison result of the pixel values ​​of each second shooting feature point corresponding to each second projection feature point in the second captured image, the first relative direction indicates that there is no horizontal flip between the captured image acquired by the camera and the image projected by the projector; if the comparison result of the pixel values ​​of each second projection feature point is different from the comparison result of the pixel values ​​of each second shooting feature point corresponding to each second projection feature point in the second captured image, the first relative direction indicates that there is a horizontal flip between the captured image acquired by the camera and the image projected by the projector.

[0100] In some embodiments, a method for obtaining the pixel value of the second projection feature point and the pixel value of the first shooting feature point may include: calculating the coordinates of the second projection feature point and the coordinates of the first shooting feature point; obtaining the pixel value of the second projection feature point from the second calibration image according to the coordinates of the second projection feature point; and obtaining the pixel value of the first shooting feature point from the second shooting image according to the coordinates of the first shooting feature point.

[0101] For example, Figure 4 The second projected feature point A in (a), the area of ​​point A is enlarged to obtain the local image of (a) as shown in Figure 7 As shown, the checkerboard image is decoded into corner points to obtain Figure 7 When the coordinates of the white point are (a, b) and the chessboard step length is s, the coordinates of point A are (x–s / 2, y–s / 2). Similarly, the coordinates of points A', A", B, B', B", C, C', D and D' can be obtained.

[0102] The white point may be any point with known coordinates in the chessboard image, and may be selected according to the requirements.

[0103] like Figure 4 As shown, it is known Figure 4 The coordinates of midpoint A, point A', point B, and point B', and Figure 6 The coordinates of the midpoint A" and point B" can be used to obtain the pixel values ​​of the corresponding points. Figure 4 In (a), the pixel value of point A is 0, and the pixel value of point B is 255. The comparison result of pixel values ​​is v(A)>v(B). Figure 4 In (b), the pixel value of point A' is 0, the pixel value of point B' is 255, and the comparison result of the pixel values ​​is v(A')>v(B'). Obviously, the comparison result of the pixel values ​​of each second projection feature point is the same as the comparison result of the pixel values ​​of each second shooting feature point. At this time, the first relative direction indicates that there is no horizontal flip between the captured image obtained by the camera and the image projected by the projector. Figure 6 The pixel value of point A" is 255, and the pixel value of point B" is 0. The comparison result of the pixel values ​​is v(A")<v(B"). Obviously, the comparison result of the pixel values ​​of each second projection feature point is different from the comparison result of the pixel values ​​of each second shooting feature point. At this time, the first relative direction indicates that there is a horizontal flip between the captured image obtained by the camera and the image projected by the projector.

[0104] Specifically, the second relative direction between the camera and the projector is determined based on the comparison results of the pixel values ​​of each third projection feature point and the relationship between the comparison results of the pixel values ​​of each third shooting feature point in the third captured image and the corresponding third projection feature points in the third captured image.

[0105] If the comparison results of the pixel values ​​of each third projection feature point and the comparison results of the pixel values ​​of each third shooting feature point in the third captured image corresponding to each third projection feature point in the third captured image are the same, the second relative direction indicates that there is no vertical flip between the captured image acquired by the camera and the image projected by the projector; if the comparison results of the pixel values ​​of each third projection feature point and the comparison results of the pixel values ​​of each third shooting feature point in the third captured image corresponding to each third projection feature point in the third captured image are different, the second relative direction indicates that there is a vertical flip between the captured image acquired by the camera and the image projected by the projector.

[0106] Similarly, the method for obtaining the pixel value of the third projection feature point and the pixel value of the third shooting feature point can be the same as before, specifically including: calculating the coordinates of the third projection feature point and the coordinates of the third shooting feature point; obtaining the pixel value of the third projection feature point from the third calibration image according to the coordinates of the third projection feature point; obtaining the pixel value of the third shooting feature point from the fourth shooting image according to the coordinates of the third shooting feature point.

[0107] like Figure 5 As shown, it is known Figure 5 The coordinates of midpoint C, point C', point D, and point D', and Figure 6 The coordinates of the midpoint C" and point D" can be used to obtain the pixel values ​​of the corresponding points. Figure 4 The pixel value of point C in (C) is 0, and the pixel value of point D is 255. The comparison result of the pixel values ​​is v(C)<v(D). Figure 4 The pixel value of point C' in (D) is 0, and the pixel value of point D' is 255. The pixel value comparison result is that v(C') is less than v(D'). Obviously, the pixel value comparison result of each third projection feature point is the same as the pixel value comparison result of each third shooting feature point. At this time, the second relative direction indicates that there is no vertical flip between the captured image obtained by the camera and the image projected by the projector.

[0108] After the first relative direction and the second relative direction are acquired, the relative direction between the camera and the projector is further determined according to the first relative direction and the second relative direction.

[0109] In some embodiments, the second calibration image and the third calibration image may be the same calibration image as a fourth calibration image, which includes at least three fourth projection feature points, among which there is one fourth projection feature point that is symmetrical with the other two fourth projection feature points along the first symmetry axis and the second symmetry axis, and the pixel values ​​of each fourth projection feature point are different.

[0110] The relative direction between the camera and the projector is automatically determined through A1-A3, avoiding the problem of inaccurate calibration of the transformation relationship from the camera space to the projection space of the projector caused by the flip between the camera and the projector.

[0111] S202: When the projector projects the first calibration image, a first captured image corresponding to the first calibration image is acquired by a camera.

[0112] The first calibration image includes the first projection feature point, and the first calibration image may be any one of the second calibration image and the third calibration image, and the first calibration image may also be the fourth calibration image.

[0113] There is no limitation on the calibration features in the first calibration image, and it can be, for example Figure 8 The chessboard shown, Figure 8 The area 510 in FIG. 5 is the shooting area of ​​the camera, and the area 520 is the projection area of ​​the projector. Both the projection area and the shooting area cover the display area of ​​the first calibration image, and both the projection area and the shooting area cover the position before and after the position of the human body carrying object changes, that is, the activity area of ​​the human body carrying object (such as Figure 8 middle area 530).

[0114] S203 , determining a second transformation relationship according to the relative direction, the coordinates of each first projection feature point in the first calibration image, and the coordinates of each first shooting feature point corresponding to each first projection feature point in the first shooting image.

[0115] In some embodiments, S203 may include: flipping the positions of each first captured feature point corresponding to the positions of each first projection feature point in the first captured image according to the relative relationship, and determining the second transformation relationship according to the coordinates of the first projection feature point after the position flipping and the coordinates of the first projection feature point.

[0116] In some implementations, corner point decoding may be performed on the first calibration image and the first captured image to obtain the coordinates of the first projection feature point after position flipping and the coordinates of the first projection feature point.

[0117] Take the example that the relative direction indicates that there is only a horizontal flip between the image captured by the camera and the image projected by the projector, such as Figure 4 (a) and Figure 6 As shown, after the horizontal flipping occurs, point A is actually flipped to the position of point B'', and the coordinates of point A in the first captured image are the coordinates of point B''. That is to say, it is necessary to horizontally flip the positions of all first captured feature points, and obtain the coordinates of the first captured feature points after the position flipping as the coordinates corresponding to the actual first projection feature points in the third projection image, for example Fig. 9 , Fig. 9 Each letter in represents a first shot feature point, and there are 16 first shot feature points in total. Fig. 9 The table on the left shows the positions of the 16 first projection feature points. Fig. 9 The table on the right in the middle shows the positions of the 16 first shooting feature points after horizontal flipping. The coordinates of the first shooting feature points after horizontal flipping are the coordinates corresponding to the first projection feature points in the first shot image.

[0118] After obtaining the coordinates of the first projection feature point and the coordinates of the first shooting feature point, the transformation relationship between the camera space and the projection space can be calculated according to the coordinates of the first projection feature point and the coordinates of the first shooting feature point. The transformation relationship can be represented by a homography matrix (Homography Matrix, H matrix).

[0119] Among them, the homography matrix is ​​used to describe the projective transformation from one plane to another plane on a two-dimensional plane. Calculating the homography matrix usually requires at least four corresponding points (because four independent conditions are required to determine a transformation in two-dimensional space). Given a set of corresponding points in a pair of images (usually four or more), the homography matrix can be obtained by solving a system of linear equations or linear optimization methods.

[0120] Let the homogeneous coordinates of corresponding points in a pair of images be (x, y, 1) and (x1', y1', 1), the homography matrix According to the definition of the homography matrix, formula (1) is constructed as follows:

[0121]

[0122] For each set of corresponding points in a pair of images, an equation as shown in equation (1) is constructed to obtain a set of equations corresponding to the pair of images. Finally, the homography matrix H can be obtained by solving the set of equations.

[0123] Here, each first projection feature point and its corresponding first shooting feature point in the first shot image constitute a group of corresponding points. For each group of corresponding points, an equation as shown in the above formula (1) is constructed to obtain an equation group consisting of 16 equations. The transformation relationship between the camera space and the projection space can be obtained by solving the equation group.

[0124] S104 . In response to the projection time point corresponding to the second time point arriving, control the projector to project the projection content according to the target content position.

[0125] Among them, the second time point refers to the time point when the camera takes an image of the activity area. After the image is acquired, it is necessary to process the image, such as detecting the human body-carrying object in the image, obtaining the object position information of the human body-carrying object, etc. There is a certain data processing delay in the whole processing process. The projection time point corresponding to the second time point is the time point after adding the data processing delay to the second time point.

[0126] In response to the projection time point corresponding to the second time point arriving, the projection content is projected according to the target content position, and the projection content can be projected onto the human body carrying object.

[0127] Among them, the projection area of ​​the projector is not smaller than the active area, and the degree of the angle between the main optical axis of the projector and the active plane is not greater than a preset degree, and the preset degree can be set according to demand, such as 45°.

[0128] Taking the human body object as the palm as an example, the positional relationship between the projector, camera, electronic device and palm is as follows: Fig.10 As shown, Fig.10 1000 in the middle is the palm, Fig.10 Here, 1010 is a camera, 1020 is a projector, and 1030 is an electronic device.

[0129] In this embodiment, in response to determining that there is a human-bearing object in the activity area, captured images of the activity area captured by the camera at multiple consecutive time points are obtained; according to the object position information of the human-bearing object in the captured images at multiple consecutive first time points, the target object position information corresponding to the human-bearing object in the captured images at the second time point is determined; the second time point is any one of the multiple time points, and the multiple first time points are the time points before the second time point among the multiple time points, so that the target object position information of the human-bearing object at the second time point obtained refers to the position information of the human-bearing object at the multiple first time points before it, effectively avoiding the situation that when the position of the human-bearing object is determined according to the captured image at a time point, the position of the human-bearing object is inaccurate due to the instability of the data at the time point, and effectively improving the stability of the position of the human-bearing object at the second time point. Afterwards, according to the target object position information, the target content position in the projection space of the projector is determined, and in response to the arrival of the projection time point corresponding to the second time point, the projector is controlled to project the projection content according to the target content position, so that the content display on the human-bearing object has good stability, ensuring user experience.

[0130] In some embodiments, Fig.11 As shown, before S104, the method further includes:

[0131] S301: Acquire the object posture of a human body carrying an object in a captured image at a second time point.

[0132] Among them, the object posture refers to the shape and direction of the object carried by the human body. Taking the palm of the hand as an example, the object posture refers to the gesture, which can be gestures such as OK and like.

[0133] In some implementations, the target object position information includes position information of posture key points, and the object posture of the human body carrying the object in the captured image at the second time point is determined based on the position information of the posture key points.

[0134] Specifically, the position information of the posture key points may be input into a posture recognition algorithm, and the posture of the object carried by the human body may be determined through the posture recognition algorithm.

[0135] like Fig.12 As shown, taking the palm as the human body bearing object as an example, the posture key points can be Fig.12 Shown as black circle.

[0136] S302: If the object posture of the object carried by the human body at the second time point is the target object posture, determine the duration of the target object posture according to the object posture of the object carried by the human body in the captured images at multiple consecutive third time points.

[0137] Among them, multiple third time points are time points among the multiple time points that are located before the second time point, and the last third time point among the multiple third time points is the time point before the second time point, or it can be two time points before the second time point. It can be set specifically according to needs, and the multiple third time points can be partially the same as the multiple first time points, or can be completely the same as the multiple first time points.

[0138] In some embodiments, determining the duration of the target object posture based on the object posture of the human body in the captured images at multiple consecutive third time points can include: traversing forward from the object posture of the human body corresponding to the second time point, if the object posture corresponding to the traversed third time point is the target object posture, adding a time interval to the duration until the object posture corresponding to the traversed third time point is not the target object posture, and obtaining the final duration as the duration of the target object posture.

[0139] In other embodiments, determining the duration of the target object posture based on the object posture of the human body in the captured images at multiple consecutive third time points may include: when determining that the object posture of the human body in the captured image at a certain third time point is the target object posture, if the object posture of the human body in the captured images at all third time points between the third time point and the second time point is the target object posture, accumulating the time from the third time point to the second time point, and obtaining the accumulated time as the duration; if there is a third time point between the third time point and the second time point at which the object posture of the human body in the captured image is not the target object posture, starting from a third time point after the third time point at which the object posture of the human body in the captured image closest to the second time point is not the target object posture, re-accumulating the time to obtain the accumulated time as the duration.

[0140] Among them, the target object posture is a preset object posture. Only when the object posture is the target object posture, will the subsequent operation of determining the duration of the target object posture be triggered. There is at least one target object posture. If there are multiple target object postures, when determining the duration of the target object posture, only the duration of the same target object posture is considered, that is, when the target object posture at the last third time point is the first target object posture, if the target object posture at the second time point is a second target object posture different from the first target object posture, then what needs to be determined is the duration of the second target posture, and the duration of the second target posture is 0.

[0141] After obtaining that the object posture of the object carried by the human body at the second time point is the target object posture, it is necessary to determine whether the object posture of the object carried by the human body at the second time point is one of the at least one target object posture mentioned above; if it is determined that the object posture of the object carried by the human body at the second time point is one of the at least one target object posture mentioned above, the duration of the object posture of the object carried by the human body at the second time point is counted, and the duration is the duration of S301.

[0142] In some embodiments, after S301, the method further includes: if the object posture of the object carried by the human body at the second time point is a critical object posture, obtaining the object posture corresponding to the object carried by the human body in the captured image at the last third time point among multiple third time points as the object posture of the object carried by the human body at the second time point.

[0143] Among them, the critical object posture refers to the object posture presented by the human body-carrying object when the posture key points on the human body-carrying object are located in the preset critical area. There is at least one posture key point, and the posture key points are located in the preset critical area. It can be that there is one posture key point located in the preset critical area, or a preset proportion of the posture key points in the posture key points are located in the preset critical area.

[0144] The critical object posture can also be understood as the object posture that is close to the target object posture. The critical object posture may be that the object posture of the object carried by the human body is expected to be the target object posture, but is not in the exact target object posture, or the object posture at the second time point is not expected to be the target object posture, but in the process of changing or ending the object posture, the object posture is made to be the target object posture.

[0145] In this case, the object posture of the human body carrying object at the previous time point before the time point when the object posture of the human body carrying object is the critical object posture can be obtained as the object posture at this time point, that is, the object posture of the human body carrying object in the captured image at the last third time point among multiple third time points is obtained as the object posture of the human body carrying object at the second time point.

[0146] After obtaining the object posture of the human body carrying the object in the captured image at the second time point, it is determined whether it is the target object posture. If it is the target object posture, execute S302.

[0147] Through the above process, the problem of jitter caused by repeated changes in the posture of the object carried by the human body between the target object posture and the non-target object posture at multiple time points is solved, which seriously affects the user experience.

[0148] In some implementations, if the object posture of the object carried by the human body at the second time point is neither the target object posture nor the critical object posture, an error prompt message is output.

[0149] S303: Determine the projection content according to the target object's posture and duration.

[0150] In some implementations, a control instruction may be determined based on the posture and duration of the target object, and the projection content may be determined based on the control instruction.

[0151] The projection content may be preset, and the projection content is determined from a plurality of preset projection contents according to a control instruction; the projection content may not be preset, and the corresponding projection content is generated according to the control instruction.

[0152] In some embodiments, a duration threshold may be set, and when the duration is less than the duration threshold or not less than the duration threshold, different projection contents may be selected from a plurality of preset projection contents according to the posture of the target object, or different methods may be selected to generate corresponding projection contents according to the posture of the target object.

[0153] For example, when the duration is not less than the duration threshold, it can be considered that a special event is to be triggered, such as triggering the display of a menu or triggering a projection special effect.

[0154] In this embodiment, the object posture of the object carried by the human body in the captured image at the second time point is obtained; if the object posture of the object carried by the human body at the second time point is the target object posture, the duration of the target object posture is determined according to the object posture of the object carried by the human body in the captured image at multiple consecutive third time points; the multiple third time points are time points in the multiple time points that are located before the second time point; the projection content is determined according to the target object posture and the duration, so that the projection content is determined according to the object posture and the duration. When the user needs to change the projection content, it only needs to change the object posture or adjust the duration, which improves the flexibility of determining the projection content and improves the user experience.

[0155] In one embodiment, taking the human body carrying object as a palm as an example, the application process of the above implementation in the projector is as follows:

[0156] In some embodiments, the state of the palm in the active area can be described by the state of the position state machine, and the position state machine can include 5 states, namely S0 empty state, indicating that there is no palm in the camera area at present; S1 moving state, indicating that there is a moving palm in the active area; S2 stable state, indicating that the movement of the palm has been basically stable; S3 lost state, indicating that the palm is not detected at a time point, but the palm is indeed in the active area and is in a missed detection state.

[0157] Among them, the five states of the position state machine can be converted to each other. As long as the corresponding position trigger condition is met at the second time point, the state conversion of the position state machine can be triggered. The state of the position state machine at the second time point is determined according to the state of the position state machine at the last first time point and the position trigger condition.

[0158] The position trigger conditions include no palm detection, long-distance movement of the palm, and short-distance movement of the palm.

[0159] In some embodiments, if the relative distance between the palm at the second time point and the palm at the last first time point is not less than a relative distance threshold, it is considered that the palm has moved a long distance; if the relative distance between the palm at the second time point and the palm at the last first time point is less than the relative distance threshold, it is considered that the palm has moved a short distance.

[0160] Furthermore, when the state of the position state machine is changed at the second time point, at least one of the palm position information, gesture information and additional information needs to be updated synchronously.

[0161] It can be understood that if only gesture information is updated at the second time point, the position information and additional information at the last first time point are still used as the target position information and target additional information at the second time point. If the position information is updated at the second time point, the updated position information is the target position information of the palm.

[0162] Among them, the position information of the palm is the target object position information corresponding to the object carried by the human body. The updating process of the palm position information can refer to the specific description in S102, and is not limited here. The gesture information includes the target gesture and the duration of the target gesture. The additional information includes information other than the position information and gesture information, such as the system log of the projection system.

[0163] Fig.13 The state of the position state machine at the second time point and the information that needs to be updated in the process of determining the state of the position state machine are shown. Fig.13 In the figure, 0 indicates that the palm is not detected, 1 indicates that the palm moves a long distance, 2 indicates that the palm moves a short distance, a indicates resetting all information, b indicates updating the position information, c indicates updating the gesture information, and d indicates updating the additional information.

[0164] like Fig.13 As shown, in the case where the second time point is the time point for first determining the state of the position state machine, if no palm is detected at this time point, the state of the position state machine at the second time point is determined to be S0, and there is no need to reset all information, update position information, update gesture information, and update additional information, that is, 0-.

[0165] Among them, there is no need to reset all information, update location information, update gesture information, and update additional information, which means that all information is at the initial value or empty.

[0166] like Fig.13As shown, in the case where the second time point is not the time point at which the state of the position state machine is determined for the first time, if no palm is detected at this time point, when the state of the position state machine at the last first time point is S1 or S2, the state of the position state machine at the second time point is determined to be S3, and only the gesture information is updated, that is, 0-b; when the state of the position state machine at the last first time point is S3, all information is reset, that is, 0-a.

[0167] like Fig.13 As shown, when the second time point is not the time point at which the state of the position state machine is determined for the first time, if the palm moves a short distance at this time point, when the state of the position state machine at the last first time point is S0, the state of the position state machine at the second time point is determined to be S1, and the position information, gesture information and additional information are updated, that is, 1-bcd; when the state of the position state machine at the last first time point is S1, S2 and S3, the state of the position state machine at the second time point is determined to be S2, and the gesture information and additional information are updated, that is, 1-bd.

[0168] like Fig.13 As shown, when the second time point is not the time point at which the state of the position state machine is determined for the first time, if the palm moves a long distance at this time point, and the state of the position state machine at the last first time point is S0, S1, S2 or S3, the state of the position state machine at the second time point is determined to be S1, and the position information, gesture information and additional information are updated, that is, 2-bcd.

[0169] After the target position information of the palm at the second time point is determined, the corresponding target content position at the second time point is determined according to the target position information.

[0170] In some implementation methods, when updating gesture information, if there is at least one target gesture, a gesture state machine can be set for each target gesture. The state of the gesture is described by the state of the gesture state machine. The gesture state machine may include three states, namely, a relaxed state V0, indicating that the target gesture is not detected; a bent state V1, indicating that the target gesture exists at a time point; and a maintained state V2, indicating that the target gesture is detected at multiple consecutive time points.

[0171] Among them, the three states of the gesture state machine can be converted to each other. As long as the corresponding gesture trigger condition is met at the second time point, the state conversion of the gesture state machine can be triggered. The state of the gesture state machine at the second time point is determined according to the state of the gesture state machine at the last first time point and the gesture trigger condition.

[0172] Among them, the gesture triggering conditions include that the gesture is not a target gesture, the gesture is a target gesture and the duration of the target gesture is less than a duration threshold, and the gesture is a target gesture and the duration of the target gesture is not less than a duration threshold.

[0173] Fig.14 The process of determining the state of the attitude state machine at the second time point is shown. Fig.14 In , D indicates that the gesture is a target gesture, U indicates that the gesture is not a target gesture, T< indicates that the duration of the target gesture is less than the duration threshold, and T≥ indicates that the duration of the target gesture is not less than the duration threshold.

[0174] like Fig.14 As shown, when the second time point is the time point for first determining the state of the posture state machine, if no palm is detected at this time point, the state of the posture state machine at the second time point is determined to be V0, that is, the posture state machine is at the initial value.

[0175] like Fig.14 As shown, when the second time point is not the time point for first determining the state of the posture state machine, if the gesture at this time point is not the target gesture, and when the state of the posture state machine at the last third time point is V0, V1 or V2, then the state of the posture state machine at the second time point is determined to be V0.

[0176] like Fig.14 As shown, when the second time point is not the time point for first determining the state of the posture state machine, if the gesture at this time point is a target gesture, when the state of the posture state machine at the last third time point is V0 or V1, when the duration of the target gesture is less than the duration threshold, then the state of the posture state machine at the second time point is determined to be V1; when the state of the posture state machine at the last third time point is V1 or V2, when the duration of the target gesture is not less than the duration threshold, then the state of the posture state machine at the second time point is determined to be V2.

[0177] After the state of the gesture state machine at the last third time point and the state of the gesture state machine at the second time point are known, the gesture event collection at the second time point is determined according to the state of the gesture state machine at the next third time point and the state of the gesture state machine at the second time point. The gesture event collection includes gesture events corresponding to multiple target gestures, and the gesture event is the state conversion process of the gesture state machine. For example, if S0 is converted to S1, the gesture event can be recorded as 01.

[0178] The complete gesture update process is as follows Fig.15As shown, first, the position information of the key points of the posture in the captured image at the second time point is input into the gesture recognition algorithm to obtain the gesture at the time point, and then the gesture and at least one target gesture are respectively subjected to gesture input judgment to determine whether the gesture at the second time point is the target gesture and the duration of the target gesture, and then the posture state machine is adjusted according to the judgment result, and thereafter, a collection of gesture events is output.

[0179] The process of gesture input determination is to determine whether the object gesture at the second time point is the target object gesture and the duration of the target object gesture. Please refer to the specific description of S302, which will not be repeated here.

[0180] After the output gesture event set is acquired, the projection content is determined according to the gesture events corresponding to the target gestures in the output gesture event set and according to the gesture events.

[0181] After the projection content is acquired, the projector is controlled to project the projection content according to the target content position.

[0182] In some embodiments, the target object position information includes position information of key positioning points, such as Fig.16 As shown, S103 includes:

[0183] S401: Determine a first transformation relationship according to position information of the positioning key point and position information of the calibrated positioning key point.

[0184] Among them, the first transformation relationship indicates the transformation relationship from the calibrated human body carrying object to the human body carrying object in the camera space of the camera; the calibration positioning key points refer to the positioning key points located on the calibrated human body carrying object in the calibrated captured image; the calibrated captured image refers to the captured image of the active area taken by the camera when the calibrated human body carrying object is located at the calibrated position in the active area.

[0185] Among them, the calibrated human body carrying object and the human body carrying object can be the same object, for example, the palm of object A, the face of object A, etc., or they can be objects of the same type, such as the palm of object A and the palm of object B.

[0186] The position information of the positioning key points and the position information of the calibrated positioning key points can both be coordinates. The first transformation relationship can be represented by a homography matrix (Homography Matrix, H matrix). For the sake of distinction, the resulting homography matrix is ​​referred to as the first homography matrix.

[0187] Take the palm as an example of a human body bearing object. Figure 2It can be seen that there are 6 positioning key points and calibrated positioning key points. Each positioning key point and its corresponding calibrated positioning key point constitute a set of corresponding points. The equation shown in the above formula (1) is constructed to obtain an equation group consisting of 6 equations. The first homography matrix can be obtained by solving the equation group.

[0188] S402: Determine a target transformation relationship according to the pre-calibrated second transformation relationship, the third transformation relationship, and the first transformation relationship.

[0189] The second transformation relationship indicates the transformation relationship from the camera space of the camera to the projection space of the projector; and the third transformation relationship indicates the transformation relationship from the content space where the projection content is located to the camera space.

[0190] The specific process of calibrating the second transformation relationship is as described in S201-S203, which will not be repeated here. The second transformation relationship can be represented by a homography matrix (Homography Matrix, H matrix). For the sake of distinction, the obtained homography matrix is ​​called the second homography matrix.

[0191] In some embodiments, a camera may obtain an image of the activity area when calibrating that the human body carrying object is located in the activity area (hereinafter referred to as the calibration image for the sake of distinction), and determine the third transformation relationship according to the position of the object indication area in the calibration image and the position of the projected content in the content space. The third transformation relationship may be represented by a homography matrix (Homography Matrix, H matrix), and for the sake of distinction, the obtained homography matrix is ​​referred to as the third homography matrix.

[0192] Fig.17 A third transformation relationship is shown, Fig.17 The middle area 1700 is for calibrating the human body bearing object. Fig.17 The middle area 1710 is the object indication area, and the area 1720 is the projection content.

[0193] The position of the object indication area can be represented by corner point coordinates, which are (ltx, lty), (rtx, rty), (lbx, lby) and (rbx, rby).

[0194] Among them, the position of the projected content in the content space can be determined by the resolution of the projected content. Assuming the resolution is (h, w), the projected content in the content space (that is, in its own coordinate system) can be represented as (0, 0), (w, 0), (0, h) and (h, w) respectively.

[0195] After obtaining the position of the object indication area and the position of the projected content in the content space, each corner point of the object indication area and the corresponding corner point of the projected content in the content space constitute a set of corresponding points. For each set of corresponding points, an equation as shown in the above formula (1) is constructed to obtain a system of equations consisting of 4 equations. The third homography matrix can be obtained by solving the system of equations.

[0196] After the first homography matrix, the second homography matrix and the third homography matrix are acquired, a target transformation relationship is determined according to the first homography matrix, the second homography matrix and the third homography matrix.

[0197] S403: Determine the target content position in the projection space according to the target transformation relationship.

[0198] The target transformation relationship can quickly transform the projection content perspectively to the target content position in the projection space, and then the projection content is projected to the actual content display area on the human body carrying object through the projector according to the target content position.

[0199] At this time, the projection content is displayed in the actual content display area on the human body carrying object. If the camera captures the image of the human body carrying object, the area corresponding to the actual content display area on the captured image is the object indication area.

[0200] In this embodiment, the target object position information includes the position information of the positioning key point; the first transformation relationship is determined according to the position information of the positioning key point and the position information of the calibrated positioning key point; the first transformation relationship indicates the transformation relationship from the calibrated human body bearing object to the human body bearing object in the camera space of the camera; the calibrated positioning key point refers to the positioning key point on the calibrated human body bearing object in the captured image of the active area taken by the camera when the calibrated human body bearing object is located in the active area; according to the pre-calibrated second transformation relationship, the third transformation relationship and the first transformation relationship, the target transformation relationship is determined; the second transformation relationship indicates the transformation relationship from the camera space of the camera to the projection space of the projector; the third transformation relationship indicates the transformation relationship from the content space where the projection content is located to the camera space; according to the target transformation relationship, the target content position in the projection space is determined, and the subsequent projection of the projection content according to the target content position can realize real-time projection of the projection image onto the human body bearing object.

[0201] Fig.18 A schematic diagram of a projection control system is shown, which includes a perception module 1810 , a position control module 1820 , a gesture control module 1830 , a content control module 1840 and a projection control module 1850 . The projection control system may also include an additional communication module 1860 and a log module 1870 .

[0202] The perception module 1810 is used to obtain the images taken by the camera of the activity area at multiple consecutive time points and detect whether there are positioning key points in the images. The presence of positioning key points means whether there is a human body carrying an object.

[0203] The position control module 1820 is used to output the target object position information of the human body-carrying object at each time point according to the output of the perception module, that is, to update the position of the human body-carrying object.

[0204] The gesture control module 1830 is used to output a set of posture events (such as a set of gesture events) of the object carried by the human body at each time point according to the output of the perception module.

[0205] The content control module 1840 is used to determine the projection content according to the output of the gesture control module 1830; calculate the first transformation relationship from the calibrated human body bearing object to the human body bearing object in the camera space of the camera according to the target object position information output by the position control module 1820, and determine the target transformation relationship by combining the transformation relationship from the camera space of the camera to the projection space of the projector that is pre-calibrated and the transformation relationship from the content space where the projection content is located to the calibrated human body bearing object in the camera space.

[0206] The projection control module 1850 is used to control the projector to project the projection content based on the target transformation relationship according to the output of the content control module 1840.

[0207] The communication module 1860 is used to communicate with other entities such as mobile devices. For example, in human-computer interaction, the user interacts with the content in the palm of his hand and wants to trigger a control instruction to control other devices to perform related operations. Here, the position, gesture, content and other states of the system need to be encoded and output through the communication protocol, and the message received is used to provide feedback to the system.

[0208] The log module 1870 is used to record various system information.

[0209] In some embodiments, Fig.19 As shown, the content control module 1840 includes a content organization unit and a perspective transformation unit, wherein the content organization unit is used to determine the projection content according to the output of the gesture control module 1830, and the perspective transformation unit is used to calculate the first transformation relationship from the calibrated human body carrying object to the human body carrying object in the camera space of the camera according to the target object position information output by the position control module 1820, and determine the target transformation relationship by combining the transformation relationship from the camera space of the camera to the projection space of the projector that is pre-calibrated and the transformation relationship from the content space where the projection content is located to the calibrated human body carrying object in the camera space.

[0210] See attached Fig. 20 , Fig. 20 The structure block diagram of another projection control device proposed in one embodiment of the present application is shown, which is used in electronic equipment. The device 2000 includes:

[0211] An acquisition module 210 is used for acquiring images of the activity area taken by a camera at a plurality of consecutive time points in response to determining that a human body carrying an object exists in the activity area;

[0212] A first determination module 220 is used to determine the target object position information corresponding to the human body-carrying object in the captured images at a second time point according to the object position information of the human body-carrying object in the captured images at a plurality of consecutive first time points; the second time point is any one of the plurality of time points, and the plurality of first time points are the time points before the second time point in the plurality of time points;

[0213] The second determination module 2030 is used to determine the target content position in the projection space of the projector according to the target object position information;

[0214] The control module 2040 is configured to control the projector to project the projection content according to the target content position in response to the projection time point corresponding to the second time point arriving.

[0215] Optionally, the first determination module 2020 is also used to obtain the object position information of the human body-carrying object in the last captured image at the first time point as the target object position information if the difference between the object position information of the human body-carrying object in the captured images at multiple first time points and the object position information of the human body-carrying object in the captured images at the second time point is less than a difference threshold.

[0216] Optionally, the first determination module 2020 is also used to determine the movement speed corresponding to the human body-bearing object according to the difference between the object position information of the human body-bearing object in the captured images at multiple first time points if the difference between the object position information of the human body-bearing object in the captured images at multiple first time points and the object position information of the human body-bearing object in the captured images at the second time point is not less than a difference threshold; determine the predicted displacement based on the movement speed and the data processing delay; the data processing delay is the time difference between the second time point and the corresponding projection time point; and determine the target object position information based on the object position information of the human body-bearing object in the captured images at the second time point and the predicted displacement.

[0217] Optionally, the first determination module 2020 is also used to obtain, for any one of the multiple time points, the object position information of the human body-bearing object in the captured image at the first reference time point as the object position information of the human body-bearing object in the captured image at the time point if no human body-bearing object is detected in the captured image at the time point, and the time difference between the first reference time point corresponding to the captured image in which the human body-bearing object was detected last time and the time point is less than a time difference threshold.

[0218] Optionally, the first determination module 2020 is also used to determine, for any one of the multiple time points, if there are multiple position information of objects to be detected of the human body-carrying object in the captured image at the time point, from the multiple position information of objects to be detected, the position information of the object to be detected that has the smallest difference with the object position information of the human body-carrying object in the captured image at a second time point before the time point, as the object position information of the human body-carrying object in the captured image at the time point.

[0219] Optionally, the device also includes a third determination module, which is also used to obtain the object posture of the object carried by the human body in the captured image at a second time point; if the object posture of the object carried by the human body at the second time point is the target object posture, determine the duration of the target object posture according to the object posture of the human body in the captured image at multiple consecutive third time points; the multiple third time points are time points among the multiple time points that are located before the second time point; determine the projection content according to the target object posture and the duration.

[0220] Optionally, the third determination module is also used to obtain the object posture corresponding to the human body-carrying object in the captured image at the last third time point among multiple third time points as the object posture of the human body-carrying object at the second time point if the object posture of the human body-carrying object at the second time point is a critical object posture; wherein, the critical object posture refers to the object posture presented by the human body-carrying object when the posture key point on the human body-carrying object is located within a preset critical area.

[0221] Optionally, the second determination module 2030 is also used to determine the first transformation relationship based on the position information of the positioning key points and the position information of the calibration positioning key points; the first transformation relationship indicates the transformation relationship from the calibrated human body bearing object to the human body bearing object in the camera space of the camera; the calibrated positioning key points refer to the positioning key points on the calibrated human body bearing object in the captured image of the active area taken by the camera when the calibrated human body bearing object is located in the active area; the target transformation relationship is determined based on the pre-calibrated second transformation relationship, the third transformation relationship and the first transformation relationship; the second transformation relationship indicates the transformation relationship from the camera space of the camera to the projection space of the projector; the third transformation relationship indicates the transformation relationship from the content space where the projection content is located to the calibrated human body bearing object in the camera space; according to the target transformation relationship, the target content position in the projection space is determined.

[0222] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0223] In addition, each function in each embodiment of the present application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated module can be implemented in the form of hardware or in the form of software function module.

[0224] Please refer to Fig.21 , Fig.21 The structural block diagram of an electronic device provided according to an embodiment of the present application is shown. The electronic device 2100 may include one or more of the following components: a processor 2110, a memory 2120, and one or more application programs, wherein the one or more application programs may be stored in the memory 2120 and configured to be executed by one or more processors 2110, and the one or more programs are configured to execute the method described in the aforementioned method embodiment.

[0225] The processor 2110 may include one or more processing cores. The processor 2110 uses various interfaces and lines to connect various parts of the entire projector 2100, and executes various functions of the projector 2100 and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 2120, and calling data stored in the memory 2120. Optionally, the processor 2110 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 110 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It is understandable that the above-mentioned modem may not be integrated into the processor 2110, but may be implemented separately through a communication chip.

[0226] The memory 2120 may include a random access memory (RAM) or a read-only memory (ROM). The memory 2120 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 2120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as a phone book, audio and video data, chat record data) created by the terminal 2100 during use.

[0227] In addition, each function in each embodiment of the present application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated module can be implemented in the form of hardware or in the form of software function module.

[0228] Please refer to Fig. 22 , Fig. 22The structure block diagram of a computer-readable storage medium in an embodiment of the present application is shown. The computer-readable medium 2200 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0229] Computer readable storage medium 2200 can be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read Only Memory), EPROM, hard disk or ROM. Optionally, computer readable storage medium 2200 includes non-transient computer readable medium (non-transitory computer-readable storage medium). Computer readable storage medium 2200 has storage space for program code 2210 that performs any method step in the above method. These program codes can be read from one or more computer program products or written into one or more computer program products. Program code 2210 can be compressed, for example, in an appropriate form.

[0230] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A projection control method, characterized in that: The method comprises: In response to determining that a human-carrying object exists in the activity area, acquiring images of the activity area taken by a camera at a plurality of consecutive time points; Determine, according to the object position information of the human body-carrying object in the captured images at a second time point, the target object position information corresponding to the human body-carrying object in the captured images at a second time point; the second time point is any one of the multiple time points, and the multiple first time points are the time points before the second time point in the multiple time points; Determining the target content position in the projection space of the projector according to the target object position information; In response to the projection time point corresponding to the second time point arriving, the projector is controlled to project the projection content according to the target content position.

2. The method according to claim 1, characterized in that The determining, based on the object position information of the human body-carrying object in the captured images at the second time point, the target object position information corresponding to the human body-carrying object in the captured images at the second time point includes: If the difference between the object position information of the human body-carrying object in the captured images at the multiple first time points and the object position information of the human body-carrying object in the captured images at the second time point is less than a difference threshold, the object position information of the human body-carrying object in the captured image at the last first time point is obtained as the target object position information.

3. The method according to claim 1, characterized in that The determining, based on the object position information of the human body-carrying object in the captured images at the second time point, the target object position information corresponding to the human body-carrying object in the captured images at the second time point includes: If a difference between the object position information of the human body-carrying object in the captured images at the multiple first time points and the object position information of the human body-carrying object in the captured images at the second time point is not less than a difference threshold, determining a movement speed corresponding to the human body-carrying object according to the difference between the object position information of the human body-carrying object in the captured images at the multiple first time points; Determining a predicted displacement based on the movement speed and a data processing delay, wherein the data processing delay is a time difference between the second time point and a corresponding projection time point; The target object position information is determined based on the object position information of the human body-carrying object in the captured image at the second time point and the predicted displacement.

4. The method according to claim 1, characterized in that Before determining the target object position information corresponding to the human body-carrying object in the captured images at the second time point according to the object position information of the human body-carrying object in the captured images at the consecutive first time points, the method further includes: For any one of the multiple time points, if the human-carrying object is not detected in the captured image at the time point, and the time difference between the first reference time point corresponding to the captured image in which the human-carrying object was detected last time and the time point is less than the time difference threshold, obtain the object position information of the human-carrying object in the captured image at the first reference time point as the object position information of the human-carrying object in the captured image at the time point.

5. The method according to claim 1, characterized in that Before determining the target object position information corresponding to the human body-carrying object in the captured images at the second time point according to the object position information of the human body-carrying object in the captured images at the consecutive first time points, the method further includes: For any one of the multiple time points, if there are multiple object position information to be detected of the human body-carrying object in the captured image at the time point, determine the object position information to be detected with the smallest difference between the object position information of the human body-carrying object in the captured image at a second reference time point before the time point from the multiple object position information to be detected, as the object position information of the human body-carrying object in the captured image at the time point.

6. The method according to claim 1, characterized in that Before the projector is controlled to project the projection content according to the target content position in response to the projection time point corresponding to the second time point arriving, the method further includes: Acquire the object posture of the object carried by the human body in the captured image at the second time point; If the object posture of the object carried by the human body at the second time point is the target object posture, determining the duration of the target object posture according to the object posture of the object carried by the human body in the captured images at multiple consecutive third time points; the multiple third time points are time points before the second time point among the multiple time points; The projection content is determined according to the target object posture and the duration.

7. The method according to claim 6, characterized in that After determining the object posture of the object carried by the human body at the second time point according to the captured image at the second time point, the method further includes: If the object posture of the human body-carrying object at the second time point is a critical object posture, the object posture corresponding to the human body-carrying object in the captured image at the last third time point among the multiple third time points is obtained as the object posture of the human body-carrying object at the second time point; wherein, the critical object posture refers to the object posture presented by the human body-carrying object when the posture key point on the human body-carrying object is located within a preset critical area.

8. The method according to claim 1, characterized in that The target object position information includes position information of a positioning key point; and determining the target content position in the projection space of the projector according to the target object position information includes: Determine a first transformation relationship according to the position information of the positioning key point and the position information of the calibration positioning key point; the first transformation relationship indicates the transformation relationship from the calibration human body carrying object to the human body carrying object in the camera space of the camera; the calibration positioning key point refers to the positioning key point located on the calibration human body carrying object in the calibration captured image; the calibration captured image refers to the captured image of the activity area captured by the camera when the calibration human body carrying object is located at the calibration position in the activity area; Determine a target transformation relationship according to a pre-calibrated second transformation relationship, a third transformation relationship and the first transformation relationship; the second transformation relationship indicates a transformation relationship from the camera space of the camera to the projection space of the projector; the third transformation relationship indicates a transformation relationship from the content space where the projection content is located to the camera space; According to the target transformation relationship, a target content position in the projection space is determined.

9. A projection control device, characterized in that: The device comprises: An acquisition module, configured to acquire images of the activity area taken by a camera at a plurality of consecutive time points in response to determining that a human-bearing object exists in the activity area; A first determination module is used to determine the target object position information corresponding to the human body-carrying object in the captured images at a second time point according to the object position information of the human body-carrying object in the captured images at a plurality of consecutive first time points; the second time point is any one of the plurality of time points, and the plurality of first time points are the time points before the second time point among the plurality of time points; A second determination module, configured to determine a target content position in a projection space of the projector according to the target object position information; The control module is configured to control the projector to project the projection content according to the target content position in response to the projection time point corresponding to the second time point arriving.

10. A projector, characterized in that: include: one or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the method according to any one of claims 1-8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program code executable by a processor, and when the program code is executed by the processor, the processor executes the method according to any one of claims 1 to 8.

Images