Feature surface projection system and feature surface projection method

Abstract

To provide a feature surface projection system including a projector, a camera, and a controller.SOLUTION: A projector projects structured light and an image beam to a feature surface respectively at different times, the structured light forms a structured light pattern on the feature surface, and the image beam forms an image on the feature surface. A camera photographs the structured light pattern and the image at different times. A controller is electrically connected to the projector and the camera, calculates a feature point position of the feature surface on the basis of the structured light pattern, and determines a difference between a feature point position of the image and the feature point position of the feature surface to determine whether or not to adjust the image beam. A projection method of a feature surface is also provided. The feature surface projection system includes an automatic alignment function between a projected content and the feature surface.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to projection systems, and more particularly to feature surface projection systems. [Background technology]

[0002] In recent years, projectors have been widely used in facial projection, such as dynamic facial expressions for service robots, digital makeup for movies, facial art creation, etc. The use of projection content has greatly expanded the extensibility of the application end, allowing dynamic facial expressions to be created, enhancing interaction, simplifying the makeup rehearsal process, and can also be used in art creation and singing performance.

[0003] The principle of face projection is to use a projector to project the projection content onto a human face or a mask, and it can be divided into two types: front projection and back projection. Front projection is when the observer and the projector are on the same side of the screen, and back projection is when they are on different sides.

[0004] In such applications, the projected content needs to be accurately projected onto a real human face or mask, and the relative position between the projector and the human face or mask is very important. In front projection applications, when the human face moves or the human body position changes, the position of the projection module needs to be readjusted, otherwise dynamic image effects cannot be obtained. In both front projection and rear projection, an adjustment mechanism is required to accurately project the projected content onto the human face or mask. However, such adjustment is time-consuming, and when the human body and mask move, re-adjustment is required, and the function of automatic alignment cannot be realized. Summary of the Invention [Problem to be solved by the invention]

[0005] The feature surface projection system provided by the present invention has an automatic alignment function between the projected content and the feature surface.

[0006] The method for projecting a feature surface provided by the present invention can align the projected content with the feature surface. [Means for solving the problem]

[0007] An embodiment of the present invention provides a feature surface projection system, which includes a projector, a camera, and a controller. The projector projects a structured light and an image light beam onto the feature surface at different times, the structured light forms a structured light pattern on the feature surface, and the image light beam forms an image on the feature surface. The camera captures the structured light pattern and the image at different times. The controller is electrically connected to the projector and the camera, calculates the feature point positions of the feature surface according to the structured light pattern, and determines whether to adjust the image light beam by determining the difference between the feature point positions of the image and the feature point positions of the feature surface.

[0008] A method for projecting a feature surface provided by an embodiment of the present invention includes the steps of respectively projecting an output structured light and an image light beam onto the feature surface at different times, such that the structured light forms a structured light pattern on the feature surface and the image light beam forms an image on the feature surface; respectively capturing the structured light pattern and the image at different times; calculating feature point positions on the feature surface based on the structured light pattern; and determining whether to make an adjustment to the image light beam by determining a difference between the feature point positions of the image and the feature point positions on the feature surface.

[0009] In the feature surface projection system and feature surface projection method of the embodiments of the present invention, the structured light and the image light beam are respectively projected onto the feature surface at different times, and the structured light pattern and the image are respectively captured at different times, so as to realize the alignment between the projection content and the feature surface, and further improve the accuracy and visual effect of the projected image. [Brief description of the drawings]

[0010] [Figure 1] 1 is a schematic diagram of a surface projection system featuring one embodiment of the present invention; [Diagram 2]4 is a schematic diagram of a surface projection system featuring another embodiment of the present invention; [Diagram 3] 3 is a schematic diagram of the optical path of the projector when the structured light in FIG. 1 or FIG. 2 is infrared light. [Figure 4] 4 is a schematic diagram of a driving signal for the light-emitting element in FIG. 3. [Diagram 5] 4 is a flow chart illustrating a method for projecting a feature surface according to an embodiment of the present invention. [Figure 6] 4 is a flow chart illustrating a method for projecting a feature surface according to another embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] FIG. 1 is a schematic diagram of a feature surface projection system according to an embodiment of the present invention. Referring to FIG. 1, the feature surface projection system 100 according to this embodiment includes a projector 200, a camera 110, and a controller 120. The projector 200 projects a structured light 201 and an image light beam 202 onto a feature surface 50 at different times, where the structured light 201 forms a structured light pattern on the feature surface 50, and the image light beam 202 forms an image on the feature surface 50. That is, in one embodiment, the projector 200 first projects the structured light 201 onto the feature surface 50, and then projects the image light beam 202 onto the feature surface 50. In this embodiment, the feature surface 50 is, for example, the surface of a mask, that is, the surface of a shell in the shape of a human face, but is not limited thereto. In another embodiment, the feature surface 50 may be any non-planar surface. When the structured light 201 is projected onto a flat surface, for example, it forms a set of dot patterns, a grid pattern, a stripe pattern, or any other possible patterns on the flat surface. On the other hand, when structured light 201 is projected onto a non-flat feature surface 50, a set of dot patterns, a grid pattern, a stripe pattern, or any other possible pattern will be distorted by the unevenness of feature surface 50, and the amount and direction of this deformation can be used to calculate the three-dimensional coordinates of a location on feature surface 50. Furthermore, the image that image beam 202 forms on feature surface 50 is the image that is desired to be shown to an audience and contains the content that is desired to be displayed.

[0012] The camera 110 captures the structured light pattern and the image formed by the image beam 202 on the feature surface 50 at different times, i.e., captures the structured light pattern after projecting the structured light, and then captures the image formed by the image beam 202 after projecting the image beam 202. The controller 120 is electrically connected to the projector 200 and the camera 110, and calculates the feature point positions of the feature surface 50 based on the structured light pattern, and determines whether to adjust the image beam by determining the difference between the feature point positions of the image formed by the image beam 202 on the feature surface 50 and the feature point positions of the feature surface. The so-called adjustment is, for example, changing the display shape or size of the image, which can be clearly understood by those skilled in the art.

[0013] In this embodiment, when the controller 120 determines that the difference between the feature point position of the image and the feature point position of the feature surface 50 exceeds a preset range (e.g., when it is greater than a certain preset distance value or is equal to or greater than a certain preset distance value. For example, the preset distance value may be 1 (cm), but is not limited thereto), the controller 120 adjusts the relative position or orientation (projection direction) of the projector 200 and the feature surface 50 through the actuator 130 to reduce the difference between the feature point position of the image and the feature point position of the feature surface 50. For example, the actuator 130 can adjust the position of the projector 200. On the other hand, when the controller 120 determines that the difference between the feature point position of the image and the feature point position of the feature surface 50 is within a preset range (e.g., when it is less than a certain distance value or is less than a certain distance value. For example, the preset distance value may be 1 (cm), but is not limited thereto), the controller 120 does not adjust the relative position or orientation of the projector 200 and the feature surface 50.

[0014] In this embodiment, the feature points of the feature surface 50 and the image include at least one of the tip of the nose 52, the corners of the eyes, the corners of the mouth, and the earlobes. For example, the controller 120 first determines the position of the feature surface 50 using a central feature point, and then determines the orientation of the feature surface 50 using symmetric feature points, where the central feature point includes the tip of the nose 52 and the symmetric feature points include the corners of the eyes, the corners of the mouth, or the earlobes.

[0015] In this embodiment, the projector 200 and the camera 110 are disposed inside a mask, for example, inside an artificial head having a feature surface 50. That is, the structured light 201 and the image light beam 202 are projected toward the inner surface of the mask, and the camera 110 takes pictures toward the inner surface of the mask. The feature surface 50 is made of a light-transmitting material, so that an audience can view the image displayed by the image light beam 202 from outside the feature surface 50. When the controller 120 determines that the difference between the feature point positions of the image and the feature point positions of the feature surface 50 is within a preset range, and after the controller 120 does not adjust the relative position or orientation of the projector 200 and the feature surface 50, the projector 200 stops projecting the structured light 201, but continues to project the image light beam 202, and the camera 110 stops taking pictures. In this case, the structured light 201 may be visible light, and once the controller 120 determines that the difference between the feature point positions of the image and the feature point positions of the feature surface 50 is within a preset range, the controller 120 also determines that the alignment is complete, so that the structured light 201 does not need to be projected any more, and the audience can see the image but not the structured light pattern. Also, since the projector 200 and the camera 110 are fixed in the artificial head, once the alignment is complete, the relative position between the projector 200 and the feature surface 50 will not change any more, and repeated calibration is not required. Note that in another embodiment, the structured light 201 may be invisible light or infrared light.

[0016] In this embodiment, the projection range of the structured light 201 and the image light beam 202 is larger than the area range of the feature surface 50, which is advantageous for good and efficient calibration when there is a misalignment between the projector 200 and the feature surface 50.

[0017] In another embodiment, the actuator 130 is not used to calibrate the relative position between the projector 200 and the feature surface 50, and the image content may be adjusted by software in the projector 200 through the controller 120. That is, the feature points in the image are directly changed in position in the image to match the feature points of the image with the feature points of the feature surface 50. That is, when the controller 120 determines that the difference between the feature points of the image and the feature points of the feature surface 50 exceeds a preset range, the controller 120 adjusts the image content through the projector 200 to reduce the difference between the feature points of the image and the feature points of the feature surface 50; and when the controller 120 determines that the difference between the feature points of the image and the feature points of the feature surface 50 is within a preset range, the controller does not adjust the image content.

[0018] Fig. 2 is a schematic diagram of a feature surface projection system according to another embodiment of the present invention. Referring to Fig. 2, the feature surface projection system 100a according to this embodiment is similar to the feature surface projection system 100 according to Fig. 1, but the difference between them is that the projector 200 and the camera 110 are disposed outside the human face or mask, and the feature surface 50 is the surface of the human face or the surface of the mask. That is, in this embodiment, the structured light 201 and the image light beam 202 are projected toward the outer surface of the human face, and the camera 110 is directed toward the outer surface of the human face to capture the image.

[0019] In addition, in this embodiment, the projector 200 alternately and repeatedly projects the structured light 201 and the image beam 202, the camera 110 continues to alternately capture the structured light pattern and the image, and the controller 120 continues to judge whether to adjust the image beam 202, so as to continuously project an accurate image onto the human face according to the dynamic movement of the human face. The adjustment to the image beam 202 may be performed by adjusting the position and orientation of the projector 200 through the actuator 130 to change the projection position and orientation of the image beam 202, or by using the controller 120 to adjust the image content in the projector 200 to match the feature point positions of the image with the feature point positions of the feature surface 50. In this way, even if the position of the human body or face is constantly changing, the projector 200 can project an accurate image onto the human face. In this case, the structured light 201 may be infrared, so that even if the structured light 201 is repeatedly projected onto the feature surface 50, the audience can see the image but not the structured light pattern.

[0020] Fig. 3 is a schematic diagram of the optical path of the projector when the structured light in Fig. 1 or Fig. 2 is infrared light, and Fig. 4 is a schematic diagram of the driving signals of the light emitting elements in Fig. 3. Referring to Fig. 3 and Fig. 4, the projector 200 may include light emitting elements 212, 214, 216 and 218 with multiple different emission wavelengths, a plurality of dichroic mirrors 222, 224 and 226, a plurality of lenses 230 and 240, at least one light valve 260 and a projection lens unit 270. In this embodiment, the light-emitting elements 212, 214, 216, and 218 are, for example, red light-emitting diodes (LEDs), green light-emitting diodes, blue light-emitting diodes, and infrared light-emitting diodes, and the dichroic mirrors 222, 224, and 226 combine the red light 213, green light 215, blue light 217, and infrared light 219 emitted by the light-emitting elements 212, 214, 216, and 218, and collect them on at least one light valve 260 through the homogenizing, transmitting, and collecting functions of the plurality of lenses 240. In addition, the plurality of lenses 230 transmit the red light 213, green light 215, blue light 217, and infrared light 219 onto the dichroic mirrors 222, 224, and 226. In another embodiment, instead of the light-emitting elements 212, 214, 216, and 218, laser light sources emitting different colors of light may be used.

[0021] In this embodiment, the light emitting elements 212, 214, 216 and 218 emit red light 213, green light 215, blue light 217 and infrared light 219 in time sequence, the light valve 260 converts the red light 213, the green light 215, the blue light 217 and the infrared light 219 into red image light beams, green image light beams, blue image light beams and structured light 201 in turn, and the projection lens unit 270 projects the red image light beams, green image light beams, blue image light beams and structured light 201 onto the feature surface 50, so that the red image light beams, green image light beams and blue image light beams overlap on the time axis to combine into the image light beam 202, thereby forming a color image on the feature surface 50.

[0022] In addition, the projector 200 further includes a total internal reflection prism (TIR prism) 250, which transmits the red light 213, the green light 215, the blue light 217, and the infrared light 219 to the light valve 260, and transmits the red image light beam, the green image light beam, the blue image light beam, and the structured light 201 from the light valve 260 to the projection lens unit 270. In another embodiment, a field lens or a reflecting mirror may be used instead of the total internal reflection prism 250. In this embodiment, the light valve 260 is, for example, a digital micro-mirror device (DMD). In another embodiment, the light valve 260 may be, but is not limited to, a liquid-crystal-on-silicon panel (LCOS panel) or a transmissive liquid crystal panel.

[0023] 4, in time period T1, a current drives the light emitting elements 212, 214, and 216 in sequence, so that the light emitting elements 212, 214, and 216 emit red light 213, green light 215, and blue light 217 in sequence, and in time period T2, a current drives the light emitting element 218, so that the light emitting element 218 emits infrared light 219. Time period T1 and time period T2 may be alternated continuously.

[0024] The feature surface projection system 100 of the above embodiment can implement a feature surface projection method, which includes the following steps.

[0025] 1. Project structured light 201 and image beam 202 onto feature surface 50 at different times, respectively, where structured light 201 forms a structured light pattern on the feature surface, and image beam 202 forms an image on the feature surface.

[0026] 2. Take structured light patterns and images at different times respectively.

[0027] 3. Calculate feature point locations on the feature surface 50 based on the structured light pattern.

[0028] 4. Determine whether to make an adjustment to the image beam 202 by determining the difference between the feature point locations in the image and the feature point locations on the feature surface 50.

[0029] The detailed process can be performed as shown in the flowchart of FIG. 5 or FIG. 6. The flowchart of FIG. 5 can be applied to the feature surface projection system 100 of FIG. 1, and includes the following steps: First, execute step S111 to project the structured light 201 and irradiate the structured light 201 on the feature surface 50. Then, execute step S112 to capture the structured light pattern with the camera 110. Then, execute step S113 to have the controller 120 calculate the feature point positions of the feature surface 50 based on the structured light pattern. Then, execute step S114 to project the image light beam 202 and project the image light beam 202 onto the feature surface 50 to form an image on the feature surface 50. Then, execute step S115 to have the camera 110 capture the image. Then, execute step S116 to have the controller 120 determine the feature point positions of the image and calculate the difference between the feature point positions of the image and the feature point positions of the feature surface 50. Then, step S117 is executed, and the controller 120 judges whether the difference exceeds a preset range, and if it is "no", the adjustment is terminated (step S130); if it is "yes", the actuator 130 is activated to adjust the relative position or orientation of the projector 200 and the feature surface 50, or the image content is adjusted by software, and then the process loops back to step S111 to continue executing the steps following step S111.

[0030] The flowchart of FIG. 6 can be applied to the feature surface projection system 100a of FIG. 2 and is similar to the flowchart of FIG. 5, but the main differences between them are as follows. In the flowchart of FIG. 6, if the answer is "No" in step S117, execute step S141 to project the structured light 201 and project the structured light pattern onto the feature surface 50. Then, execute step S142 to capture the structured light pattern with a camera. Then, execute step S143 to calculate the feature point positions of the feature surface 50 based on the structured light pattern, and calculate the difference between the feature point positions of the previous image and the feature point positions of the feature surface 50, and then loop back to step S117. This allows for continuous calibration according to the dynamic movement of the human face or mask, and allows the image projected onto the human face or mask to be accurately maintained.

[0031] In one embodiment, the controller 120 may be, for example, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a programmable controller, a programmable logic controller (PLD), or other similar device or combination of devices, but the present invention is not limited thereto. In one embodiment, the functions of the controller 120 may be implemented by a plurality of programs. These programs may be stored in a single memory, and the controller 120 may execute these programs. Alternatively, in one embodiment, the functions of the controller 120 may be implemented by one or more circuits. The present invention is not limited to implementing the functions of the controller 120 in the form of software or hardware. In addition, the execution of each step in the flowcharts of FIG. 5 and FIG. 6 may all be controlled by the controller 120.

[0032] To sum up, in the feature surface projection system and feature surface projection method of the embodiments of the present invention, the structured light and the image light beam are respectively projected onto the feature surface at different times, and the structured light pattern and the image are respectively captured at different times, so as to realize the alignment between the projection content and the feature surface, and further improve the accuracy and visual effect of the projected image.

[0033] The above are merely preferred embodiments of the present invention, and the scope of the present invention should not be limited thereto. In other words, all simple and equivalent changes and modifications made based on the claims and the contents of the specification of the present invention belong to the scope of the present invention. In addition, any embodiment or claim of the present invention does not necessarily achieve all the objectives, advantages or features of the present invention. Furthermore, the abstract and the title of the invention are used for patent search, and do not limit the scope of the present invention. In addition, the terms "first", "second", etc. mentioned in the claims indicate the names of elements or distinguish different embodiments and scopes, and do not limit the upper or lower limits of the number of elements. [Explanation of symbols]

[0034] 50 Characteristic Surfaces 52 Nose tip 100, 100a Features Surface Projection System 110 Camera 120 Controller 130 Actuator 200 projector 201 Structured light 202 Image light flux 212, 214, 216, 218 Light-emitting element 213 Red Light 215 Green Light 217 blue light 219 Infrared 222, 224, 226 Dichroic mirror 230, 240 Lens 250 Total internal reflection prism 260 Light bulb 270 Projection lens unit S111, S112, S113, S114, S115, S116, S117, S120, S130, S141, S142, S143 steps T1, T2 time zone

Claims

1. 1. A feature surface projection system, comprising: the feature surface projection system includes a projector, a camera and a controller; the projector projects a structured light and an image light beam onto the feature surface at different times, the structured light forming a structured light pattern on the feature surface, and the image light beam forming an image on the feature surface; the camera captures the structured light pattern and the image at different times, The controller is electrically connected to the projector and the camera, and calculates feature point positions of the feature surface captured by the camera based on the structured light pattern, and determines whether to make adjustments to the image light beam by determining a difference between the feature point positions of the image captured by the camera and the feature point positions of the feature surface; a feature surface projection system, characterized in that when the controller determines that a difference between the feature point position of the image and the feature point position of the feature surface is within a preset range, and after the controller no longer adjusts the relative position or orientation of the projector and the feature surface, the projector stops projecting the structured light but continues to project the image light beam, and the camera stops capturing images.

2. When the controller determines that the difference between the feature point position of the image and the feature point position of the feature surface exceeds the preset range, the controller adjusts the content of the image through the projector to reduce the difference between the feature point position of the image and the feature point position of the feature surface; 2. The feature surface projection system of claim 1, wherein the controller does not adjust the content of the image when the controller determines that the difference between the feature point positions of the image and the feature point positions of the feature surface is within the preset range.

3. 2. The feature surface projection system of claim 1, wherein when the controller determines that the difference between the feature point position of the image and the feature point position of the feature surface exceeds the preset range, the controller adjusts the relative position or orientation of the projector and the feature surface through an actuator to reduce the difference between the feature point position of the image and the feature point position of the feature surface.

4. 2. The feature surface projection system of claim 1, wherein the feature surface is a surface of a human face or a mask, and the feature points of the feature surface and the image include at least one of the tip of the nose, the corners of the eyes, the corners of the mouth, and the earlobes.

5. the feature surface is a surface of a human face or a mask, and the controller first determines a position of the feature surface using a central feature point, and then determines an orientation of the feature surface using a symmetric feature point; The feature surface projection system of claim 1 , wherein the central feature point includes the tip of the nose, and the symmetrical feature points include the corners of the eyes, the corners of the mouth, or the earlobes.

6. 2. The feature surface projection system of claim 1, wherein the projection range of said structured light and said image light bundle is greater than the range of said feature surface.

7. 2. The feature surface projection system of claim 1, wherein said structured light is infrared.

8. 2. The feature surface projection system of claim 1, wherein said structured light is visible light.

9. 2. The feature surface projection system of claim 1, wherein said feature surface is a surface of a human face, and said projector and said camera are located outside said human face.

10. The feature surface projection system of claim 9, characterized in that the projector alternately and repeatedly projects the structured light and the image light beam, the camera continues to alternately capture the structured light pattern and the image, and the controller continues to judge whether to make adjustments to the image light beam, thereby continuously projecting an accurate image onto the human face according to the dynamic movement of the human face.

11. 2. The feature surface projection system of claim 1, wherein said feature surface is a surface of a mask, and said projector and said camera are located inside said mask.

12. 1. A method for projecting a feature surface for projecting onto a feature surface, comprising the steps of: projecting a structured light beam and an image beam onto the feature surface at different times with a projector, the structured light beam forming a structured light pattern on the feature surface and the image beam forming an image on the feature surface; capturing the structured light pattern and the image at different times with a camera; calculating feature point positions of the feature surface based on the structured light pattern captured by the camera; determining whether to make an adjustment to the image beam by determining a difference between feature point locations of the image captured by the camera and feature point locations of the feature surface; a projector that stops projecting the structured light but continues to project the image light beam, and a camera that stops capturing images, when a difference between the feature point positions of the image and the feature point positions of the feature surface is within a preset range, and after the relative position or orientation of the projector and the feature surface is no longer adjusted, said projector stops projecting the structured light but continues projecting the image light beam, and said camera stops capturing images.

13. adjusting the content of the image to reduce the difference between the feature point positions of the image and the feature point positions of the feature surface when it is determined that the difference between the feature point positions of the image and the feature point positions of the feature surface exceeds the preset range; The method for projecting a feature surface according to claim 12, characterized in that when it is determined that the difference between the feature point positions of the image and the feature point positions of the feature surface is within the preset range, the content of the image is not adjusted.

14. 13. The method for projecting a feature surface according to claim 12, further comprising the steps of: adjusting a relative position or orientation of a projector that projects the structured light and the image light beam and the feature surface through an actuator when it is determined that a difference between the feature point positions of the image and the feature point positions of the feature surface has exceeded the preset range, thereby reducing the difference between the feature point positions of the image and the feature point positions of the feature surface.

15. 13. The method for projecting a feature surface according to claim 12, wherein the feature surface is a surface of a human face or a mask, and the feature points of the feature surface and the image include at least one of the tip of the nose, the corners of the eyes, the corners of the mouth, and the earlobes.

16. the feature surface being a surface of a human face or a mask; calculating the feature point positions of the feature surface based on the structured light pattern includes first determining a position of the feature surface using a central feature point and then determining an orientation of the feature surface using a symmetric feature point; 13. The method of claim 12, wherein the central feature point includes the tip of the nose, and the symmetrical feature points include the corners of the eyes, the corners of the mouth, or the earlobes.

17. 13. The method of claim 12, wherein the projection range of the structured light and the image light bundle is greater than the range of the feature surface.

18. The method for projecting a feature surface according to claim 12, characterized in that the feature surface is a surface of a human face, the structured light and the image light beam are projected toward the outer surface of the human face, and a camera takes an image toward the outer surface of the human face.

19. 13. The method for projecting a feature surface according to claim 12, wherein the feature surface is a surface of a mask, the structured light and the image light beam are projected toward the inner surface of the mask, and a camera takes an image toward the inner surface of the mask.

Images