Real-time projection method, projection equipment and storage medium

Abstract

The invention discloses a real-time projection method, projection equipment and a storage medium, the method is applied to the projection equipment, the projection equipment comprises a shooting module, and the real-time projection method comprises the following steps: in response to a received projection instruction, obtaining a first image corresponding to a projection object at a first moment through the shooting module; wherein the projection instruction is used for indicating a projection object; acquiring a second image corresponding to the projection object at a second moment through the shooting module under the condition of determining that the projection object is not matched with the pre-stored attitude database based on the first image; determining a real-time attitude corresponding to the projection object based on the first image and the second image; and performing real-time projection on the projection object according to the real-time attitude.

Description

Technical Field

[0001] The present invention relates to the field of AR projection technology, and in particular to a real-time projection method, a projection device, and a storage medium. Background Art

[0002] With the popularity of AR (Augmented Reality) projection, AR or VR (Virtual Reality) technology has gradually entered people's field of vision. Controlling smart homes through lightweight VR glasses, intelligent voice, gestures or body language has become the development trend of smart homes in the future.

[0003] In the existing AR projection technology solutions, generally, fixed objects are projected unchanged, and the projection pattern is also a fixed pattern corresponding to the fixed object. In other words, common projection technology has the limitations of projection objects, the delay of projection tracking, and the lack of rich interactivity of the projection content. Summary of the invention

[0004] The present application provides a real-time projection method, a projection device, and a storage medium, which can effectively track unknown objects in real time and interactively project them, thereby realizing real-time AR rendering of unknown objects.

[0005] The technical solution of this application is implemented as follows:

[0006] In a first aspect, the present application provides a real-time projection method, the projection method is applied to a projection device, the projection device includes a shooting module, and the method includes:

[0007] In response to the received projection instruction, acquiring a first image corresponding to the projection object at a first moment through the shooting module; wherein the projection instruction is used to indicate the projection object;

[0008] In a case where it is determined based on the first image that the projection object does not match the pre-stored posture database, acquiring, by the shooting module, a second image corresponding to the projection object at a second moment;

[0009] Determining a real-time posture corresponding to the projection object based on the first image and the second image;

[0010] The projection object is projected in real time according to the real-time posture.

[0011] In a second aspect, the present application provides a terminal device, wherein the projection device comprises: an acquisition unit, a determination unit, and a projection unit.

[0012] The acquisition unit is used to acquire a first image corresponding to the projection object at a first moment through the shooting module in response to the received projection instruction; wherein the projection instruction is used to indicate the projection object;

[0013] The acquisition unit is further configured to acquire, through the shooting module, a second image corresponding to the projection object at a second moment when it is determined based on the first image that the projection object does not match the pre-stored posture database;

[0014] The determining unit is used to determine a real-time posture corresponding to the projection object based on the first image and the second image;

[0015] The projection unit is used to perform real-time projection on the projection object according to the real-time posture.

[0016] In a third aspect, the present application provides a projection device, the projection device comprising: a processor and a memory; wherein:

[0017] The memory is used to store a computer program that can be run on the processor;

[0018] The processor is used to execute the method as described in the first aspect when running the computer program.

[0019] In a fourth aspect, the present application provides a computer-readable storage medium having a program stored thereon, and when the program is executed by a processor, the method described in the first aspect above is implemented.

[0020] The present application provides a real-time projection method, a projection device, and a storage medium, wherein the projection device includes a shooting module, and the projection device responds to the received projection instruction, and obtains the first image corresponding to the projection object at the first moment through the shooting module; wherein the projection instruction is used to indicate the projection object; in the case where it is determined based on the first image that the projection object does not match the pre-stored posture database, the second image corresponding to the projection object is obtained at the second moment through the shooting module; based on the first image and the second image, the real-time posture corresponding to the projection object is determined; and the projection object is projected in real time according to the real-time posture. It can be seen that in the present application, if it is determined that the projection object does not match the pre-stored posture database, the second image corresponding to the projection object can be obtained at the second moment through the shooting module, and then the real-time posture corresponding to the projection object can be determined by using the first image and the second image of the projection object at different moments, and the projection object is projected in real time according to the real-time posture, thereby solving the problems of single projection object, delayed projection tracking, and lack of rich interactivity of projection content. That is to say, in the present application, the projection device can be applied to a variety of projection scenes, and can effectively track unknown objects in real time and interactively project them, so as to realize real-time AR rendering of unknown objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 Schematic diagram of the structure of the projection device proposed in the embodiment of the present application Figure 1 ;

[0022] Figure 2 A schematic diagram of the flow of interaction between the projection device and the shooting module;

[0023] Figure 3 Schematic diagram of the process of implementing the real-time projection method Figure 1 ;

[0024] Figure 4 The projection object of this embodiment is shown as follows: Figure 1 ;

[0025] Figure 5 The projection object of this embodiment is shown as follows: Figure 2 ;

[0026] Figure 6 The projection object of this embodiment is shown as follows: Figure 3 ;

[0027] Figure 7 Schematic diagram of the process of implementing the real-time projection method Figure 2 ;

[0028] Figure 8 Schematic diagram of the process of implementing the real-time projection method Figure 3 ;

[0029] Fig. 9 Schematic diagram of the process of implementing the real-time projection method Figure 4 ;

[0030] Fig.10 Schematic diagram of the process of implementing the real-time projection method Figure 5 ;

[0031] Fig.11 Schematic diagram of the process of implementing the real-time projection method Figure 6 ;

[0032] Fig.12 A schematic diagram of a projector proposed in an embodiment of the present application;

[0033] Fig.13 Schematic diagram of the process of implementing the real-time projection method Figure 7 ;

[0034] Fig.14 Schematic diagram of the structure of the projection device proposed in the embodiment of the present application Figure 2 ;

[0035] Fig.15 Schematic diagram of the structure of the projection device proposed in the embodiment of the present application Figure 3 . DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. It is understood that the specific embodiments described herein are only used to explain the related applications, rather than to limit the applications. It should also be noted that, for ease of description, only the parts related to the related applications are shown in the drawings.

[0037] AR projection is widely used in smart homes and virtual games. AR projection uses key technologies such as image recognition, 3D (3 Dimensions) reconstruction and automatic 3D mapping. It uses a secondary camera to identify the carrier in real time, and uses the powerful computing power of the host to map the preset content to the carrier. In the existing technology, users can only get an unchanging projection pattern for a fixed object, and cannot track the target object in real time. Therefore, the limitations of the projection object, the lack of interactivity of the projection content, and the delay of projection tracking are three problems that need to be solved. To this end, this solution proposes a method that uses segmentation, real-time tracking, posture acquisition and analysis, and changes the projection content according to the posture of the target object, so as to achieve a relatively faster AR projection effect with richer projection content and interactive projection content with the target object.

[0038] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0039] An embodiment of the present application provides a real-time projection method, which can be applied to a projection device, wherein the projection device includes a shooting module.

[0040] Exemplarily, in some embodiments, Figure 1 Schematic diagram of the structure of the projection device proposed in the embodiment of the present application Figure 1 ,like Figure 1 As shown, the projection device may include a shooting module.

[0041] It should be noted that in the embodiments of the present application, the shooting module may be any type of shooting device. For example, the shooting module may be an IR (Infrared) image sensor, a RGB (Red Green Blue) image sensor, or a RGBD (Red Green Blue Depth Map) image sensor, which is not specifically limited in the present application.

[0042] It should be noted that in the embodiments of the present application, IR is infrared, and its wavelength (about 760nm~1mm) is a kind of electromagnetic radiation, which is longer than visible light but shorter than microwave. In dim light conditions, an IR camera is used to enhance the reality effect. An infrared image sensor is a device that can detect and receive infrared radiation. Infrared image sensors are widely used in security monitoring, drone control, smart homes and other fields. In the security field, infrared cameras sense infrared radiation to achieve night monitoring and pedestrian recognition; in the medical field, infrared imaging technology can be used for early detection of diseases, such as breast cancer screening.

[0043] It should be noted that in the embodiments of the present application, the RGB image sensor is a small electronic component that can detect the color and intensity of light in an image. It often uses three LEDs (Light Emitting Diode, red, green and blue) as light sources, and the sensor material is PD (Photo-Diode, photodiode). The RGB image sensor obtains the frequency of light waves and converts them into digital signals, has high precision and stability, and is widely used in consumer electronics fields such as mobile devices, cameras, game consoles and household appliances. In addition to measuring the color average, the RGB image sensor can also convert the color into an RGB color space, thereby facilitating the representation and processing of the color. The RGB color space is composed of three primary colors of red, green and blue, and is an additive color space, that is, more colors are obtained by adding colored light. The TOF image sensor is called a TOF camera or a 3D camera. The principle of the TOF image sensor is to measure the time required for the light irradiated by the TOF sensor to hit the detection object and then return to the TOF sensor, thereby measuring the distance between the TOF sensor and the detection object. Therefore, it has the characteristic of not being easily affected by the reflectivity of the detection object. The image sensor combined with RGB and TOF is also called an RGBD image sensor.

[0044] It should be noted that in the embodiments of the present application, the RGBD sensor is a specific type of depth sensing device used in conjunction with an RGB image sensor camera. It enhances traditional images by using depth information (related to the distance of the sensor) on a per-pixel basis, i.e., RGBD is a combination of RGB and Depth Map. The RGB color model is a color standard in the industry. It obtains a variety of colors by changing the three color channels of red (R), green (G), and blue (B) and superimposing them on each other. The RGB standard includes almost all colors that can be perceived by human vision and is one of the most widely used color systems. Depth Map In 3D computer graphics, Depth Map is an image or image channel containing information related to the distance of the surface of the scene object from the viewpoint. Among them, Depth Map is similar to a grayscale image, except that each of its pixel values ​​is the actual distance of the sensor from the object. Usually, the RGB image and the Depth image are registered, so there is a one-to-one correspondence between the pixels.

[0045] For example, in the embodiments of the present application, Figure 2 It is a flow chart of the interaction between the projection device and the shooting module, such as Figure 2 As shown, in response to the projection instruction, the projection device can send a shooting instruction to the shooting module, and then the shooting module can obtain a real-time image corresponding to the projection object at the corresponding time. The projection instruction is used to indicate the projection object. For example, the projection instruction can be some special signals, and the projection device can determine the projection object according to the special signal. There are many kinds of special signals, which are not limited here, such as voice triggering, gesture triggering, or button triggering.

[0046] Furthermore, in the embodiments of the present application, Figure 3 Schematic diagram of the process of implementing the real-time projection method Figure 1 ,like Figure 3 As shown, the method for real-time projection by a projection device may include the following steps:

[0047] Step 101: In response to a received projection instruction, a first image corresponding to a projection object is acquired at a first moment by a shooting module; wherein the projection instruction is used to indicate the projection object.

[0048] In an embodiment of the present application, the projection device may first respond to the received projection instruction, and use the shooting module to shoot the projection object indicated by the projection instruction at a corresponding time to obtain a corresponding image.

[0049] It should be noted that, in the embodiment of the present application, the projection device can determine the corresponding projection object according to the projection instruction, and then use the shooting module to obtain the first image corresponding to the projection object at the first moment.

[0050] It is understandable that in the embodiments of the present application, the projection instructions may include various forms such as sound, gestures, buttons, etc., which are not specifically limited here.

[0051] Exemplarily, in some embodiments, assuming that the projection instruction is in the form of sound, then when determining the projection object according to the projection instruction, if the projection instruction is a voice "projection to the cube", it can be determined through analysis that the projection object is a cube, or, if the projection instruction is a voice "projection to the refrigerator", it can be determined through analysis that the projection object is a refrigerator. No specific limitation is made here.

[0052] Exemplarily, in some embodiments, assuming that the projection instruction is in the form of a gesture, when determining the projection object according to the projection instruction, if the projection instruction includes a gesture indicating an image area, then the image area can be determined as the projection object, which is not specifically limited here.

[0053] Correspondingly, in an embodiment of the present application, after the projection object is determined by a projection instruction, when using the shooting module to obtain a first image corresponding to the projection object at a first moment, the first real-time image can be first obtained by the shooting module at the first moment. The first real-time image is an image directly obtained by the shooting module. The first real-time image is then segmented to determine the first image corresponding to the projection object.

[0054] It should be noted that in the embodiment of the present application, when the first real-time image is segmented to obtain the first image, that is, the image of the projection object after segmentation, different neural networks can be used for segmentation. There are many models for image segmentation, which are not limited here.

[0055] It should be noted that in the embodiments of the present application, the neural network used for image segmentation processing can meet the characteristics of not requiring knowledge of the target and the interacting subject, so that the segmentation function can be adapted to a wide range of scenes and objects. In other words, this segmentation neural network can also obtain reasonable output for data that has not been encountered in the training (learning) process, which is a common (generalized) neural network, so that the first real-time image can be directly input, and the output obtained is the first image of the segmented projection object.

[0056] Step 102: When it is determined based on the first image that the projection object does not match the pre-stored posture database, a second image corresponding to the projection object is acquired at a second moment by a shooting module.

[0057] In an embodiment of the present application, in response to a received projection instruction, after acquiring a first image corresponding to the projection object at a first moment through a shooting module, when it is determined based on the first image that the projection object does not match a pre-stored posture database, a second image corresponding to the projection object at a second moment is acquired through the shooting module.

[0058] It should be noted that, in the embodiment of the present application, the pre-stored posture database stores the mapping relationship between the projection object and the posture. This mapping relationship may be one-to-one, that is, one object may correspond to one posture; it may also be one-to-many, that is, one object may correspond to many postures.

[0059] Further, in an embodiment of the present application, after acquiring a first image corresponding to the projection object at a first moment through the shooting module, the projection device may first perform matching processing on the projection object and a pre-stored posture database based on the first image.

[0060] Furthermore, in an embodiment of the present application, in the process of performing matching processing, the first feature vector corresponding to the first image can be first determined; based on the first feature vector, the pre-stored posture database is traversed, and when there is a feature vector of a stored object corresponding to the first feature vector in the pre-stored posture database, it is determined that the projected object matches the pre-stored posture database.

[0061] It is understandable that in the embodiment of the present application, when matching the pre-stored posture database, the first feature vector corresponding to the first image can be determined first. Among them, the feature vector can be extracted using a feature extraction algorithm, and there are many algorithms for feature extraction, such as SIFT (Scale-Invariant Feature Transform) algorithm, SURF (Speeded-Up Robust Feature) algorithm, ORB (Oriented FAST and Rotated BRIEF) algorithm, etc., which are not limited here.

[0062] It should be noted that, in an embodiment of the present application, after the extraction of the first feature vector is completed, the pre-stored posture database can be further traversed based on the first feature vector. When traversing the pre-stored posture database, it is possible to find whether there is a feature vector of a stored object corresponding to the first feature vector in the pre-stored posture database. Among them, the traversal can be performed by calculating the similarity of two feature vectors. There are many algorithms for similarity calculation, which are not limited here.

[0063] Exemplarily, in some embodiments, the ORB algorithm can be used to respectively extract descriptor vectors of the first image and the projection object features, and the Euler distance between the descriptor vectors of the first image and the projection object can be calculated to determine their similarity. DBOW (Bags of words for fast place recognition in image sequence) can also be used to perform similarity calculation, etc.

[0064] It can be understood that in an embodiment of the present application, for any feature vector of a stored object in a pre-stored posture database, if the similarity between the first feature vector of the projection object and the feature vector of the stored object is less than a preset similarity threshold, then it can be considered that there is no feature vector of the stored object corresponding to the first feature vector in the pre-stored posture database, that is, it can be determined that the projection object does not match the pre-stored posture database; if the similarity between the first feature vector of the projection object and the feature vector of the stored object is greater than or equal to the preset similarity threshold, then it can be considered that there is a feature vector of the stored object corresponding to the first feature vector in the pre-stored posture database, that is, it can be determined that the projection object matches the pre-stored posture database.

[0065] It should be noted that, in the embodiments of the present application, the preset similarity threshold may be any value greater than or equal to 0 and less than or equal to 1. For example, the preset similarity threshold may be 0.01 or 0.1, and the embodiments of the present application do not specifically limit this.

[0066] Furthermore, in an embodiment of the present application, when traversing the pre-stored posture database, if there is no feature vector of a stored object with a high similarity to the first feature vector in the pre-stored posture database, it can be considered that the database does not store the posture information corresponding to the projection object, and it is necessary to obtain the second image corresponding to the projection object at the second moment through the shooting module, and the interval between obtaining the two images is slightly longer to ensure that the posture of the projection object has a more obvious change.

[0067] Exemplarily, in some embodiments, the first moment is acquired at the third second, and the second moment is acquired at the sixth second.

[0068] Further, in an embodiment of the present application, when it is determined based on the first image that the projection object matches a pre-stored posture database, the corresponding posture information of the stored object is directly used.

[0069] Furthermore, in an embodiment of the present application, when it is determined based on the first image that the projection object does not match the pre-stored posture database, it is also possible to choose to obtain a stored object with the highest similarity to the projection object in the pre-stored database, obtain at least one posture information corresponding to the stored object, and continue to obtain a second image corresponding to the projection object at a second moment through the shooting module.

[0070] Correspondingly, in an embodiment of the present application, a second real-time image is acquired at a second moment through a shooting module. The second real-time image is an image directly acquired through the shooting module. After segmenting the second real-time image, the second image corresponding to the projection object can be determined.

[0071] It should be noted that in the embodiment of the present application, when the second real-time image is segmented to obtain the second image, that is, the segmented image of the projection object, different neural networks can be used for segmentation. There are many models for image segmentation, which are not limited here.

[0072] It should be noted that in the embodiments of the present application, the neural network used for image segmentation processing can meet the characteristics of not requiring knowledge of the target and the interacting subject, so that the segmentation function can be adapted to a wide range of scenes and objects. In other words, this segmentation neural network can also obtain reasonable output for data that has not been encountered in the training (learning) process, which is a common (generalized) neural network, so that the second real-time image can be directly input, and the output obtained is the second image of the segmented projection object.

[0073] It should be noted that, in the embodiments of the present application, the projection device can continuously obtain real-time images corresponding to the projection object through the shooting module, wherein the images corresponding to the projection object refer to the segmented images.

[0074] Step 103: Determine a real-time posture corresponding to the projection object based on the first image and the second image.

[0075] In an embodiment of the present application, when it is determined based on the first image that the projection object does not match the pre-stored posture database, after the shooting module acquires a second image corresponding to the projection object at a second moment, the real-time posture corresponding to the projection object is determined based on the first image and the second image.

[0076] Furthermore, in an embodiment of the present application, when determining the real-time posture corresponding to the projection object based on the first image and the second image, the posture change value corresponding to the projection object can be first determined based on the first image, the second image and the posture estimation model, and then the posture change value is input into the posture optimization model to output the real-time posture corresponding to the projection object.

[0077] Furthermore, in an embodiment of the present application, the posture change value corresponding to the projection object is determined based on the first image, the second image and the posture estimation model. At this time, the data of the first image and the second image have been stored in the memory, and the interval between obtaining the two images is slightly longer to ensure that the posture of the projection object has a more obvious change.

[0078] It should be noted that in an embodiment of the present application, when determining the posture change value corresponding to the projection object based on the first image, the second image and the posture estimation model, the first image and the second image can be input into the posture estimation model, so that the posture change value can be output.

[0079] It should be noted that, in the embodiment of the present application, the posture change value represents the posture change of the projection object between the first moment and the second moment.

[0080] It can be understood that, in the embodiments of the present application, the posture estimation model can be used for preliminary posture estimation processing.

[0081] Exemplarily, in some embodiments, when performing preliminary posture estimation on the projection object based on the posture estimation model, the projection device memory stores the target object data after segmentation of the first image and the second image, and the interval between the two frames is slightly longer to ensure that the posture of the target object has a more obvious change.

[0082] Exemplarily, in some embodiments, based on the posture estimation model, by performing feature matching on two frames of images and estimating their posture at the second moment through epipolar constraints, it is brought into the posture estimation method based on RANSAC (Random Sample Consensus) as the initial value, so as to obtain the posture with the most votes and the most likely one. Assume that the first moment is the initial posture of the target object, the posture at the second moment is the posture change value of the target object at the second moment relative to the posture at the first moment, and the subsequent posture values ​​are also changes relative to the posture at the first moment, and then the most likely posture is obtained, and the posture and the corresponding image are stored in the memory.

[0083] It can be understood that in the embodiments of the present application, in order to improve the accuracy of posture estimation, after the posture change value is obtained using the posture estimation model, the posture change value is input into the posture optimization model to obtain a more accurate real-time posture.

[0084] It can be understood that in the embodiments of the present application, in order to improve the accuracy of the real-time posture, the posture change value can be further optimized, that is, the posture change value is input into the posture optimization model to obtain a more accurate real-time posture.

[0085] Furthermore, in an embodiment of the present application, when the posture change value is input into the posture optimization model and the real-time posture corresponding to the projection object is output, the posture change value can be updated through the posture optimization model using the historical posture change data corresponding to the projection object to determine the real-time posture.

[0086] It should be noted that, in the embodiments of the present application, the real-time posture corresponding to the projection object may be posture information corresponding to any time, or posture change information between any two time points, which is not specifically limited in the present application.

[0087] It can be understood that, in the embodiments of the present application, the posture optimization model can be used to obtain a more accurate real-time posture.

[0088] Exemplarily, in some embodiments, when performing posture optimization processing on the projection object based on the posture optimization model, the specific process of the posture optimization algorithm is to perform posture graph optimization calculation on the image and posture of the projection object after segmentation obtained at the current moment, that is, the second image and posture change value corresponding to the current frame, and the n historical images corresponding to the projection object stored in the memory. Here, the n historical images stored in the memory need to be smaller than a preset threshold. Assuming that the memory records the postures of n projection objects, recorded as T1, ..., T n , when the corresponding posture T at the i-th moment is obtained i and the pose T corresponding to the jth moment j The estimated pose change T ij After that, the cost equation can be constructed. The following formula is the cost equation:

[0089]

[0090] The optimized value Δξ is calculated ij , and use it to update the current estimated T ij The posture of the i-th moment is thus completed. i and the pose T corresponding to the jth moment j The estimated pose change T ij Optimization.

[0091] It should be noted that, in the embodiments of the present application, the preset threshold value may be any value greater than or equal to 0. For example, the preset threshold value may be 100 or 200, and the embodiments of the present application do not specifically limit this.

[0092] It can be understood that in an embodiment of the present application, after the real-time posture of the projection object is determined, the projection object and the real-time posture are stored correspondingly in a pre-stored posture database, so that the next time the projection object is projected, the real-time posture corresponding to the projection object can be found in the pre-stored database.

[0093] Step 104: Project the projection object in real time according to the real-time posture.

[0094] In an embodiment of the present application, after the real-time posture corresponding to the projection object is determined based on the first image and the second image, the projection object can be projected in real time according to the real-time posture.

[0095] Furthermore, in an embodiment of the present application, when the projection object is projected in real time according to the real-time posture, the position information of at least one pixel of the projection surface corresponding to the projection object is determined based on the real-time posture; the position information is coordinate-converted according to a preset calibration relationship to determine the homography matrix of at least one pixel; and the projection object is projected in real time based on the homography matrix of at least one pixel.

[0096] It should be noted that, in an embodiment of the present application, when determining the position information of at least one pixel of the projection surface corresponding to the projection object based on the real-time posture, it is possible to choose to calculate the three-dimensional information of the surface of the projection object based on the real-time posture and RGBD information corresponding to the projection object, and divide the surface of the projection object into N voxels based on the three-dimensional information, and then determine the four three-dimensional coordinate corner points of each voxel, that is, determine the position information of at least one pixel of the projection surface.

[0097] It should be noted that in the embodiments of the present application, when the coordinate transformation of the position information is performed according to the preset calibration relationship to determine the homography matrix of at least one pixel, the preset calibration relationship may include the transformation matrix from the RGBD camera coordinate system to the optical machine coordinate system. And the intrinsic parameter matrix K of the RGBD camera and optical machine RGBD , K projector .

[0098] For example, in the embodiments of the present application, Figure 4 The projection object of this embodiment is shown as follows: Figure 1 ,like Figure 4 As shown in the figure, after confirming the target object (projection object) through the camera, the three-dimensional information of the target object surface is obtained by estimating the object's posture and acquiring the RGBD information, and the three-dimensional object is divided into N voxels, whose projection surface consists of m×n voxels, and the four three-dimensional coordinate corner points of each voxel are obtained. The transformation matrix from the known RGBD camera coordinate system to the optomechanical coordinate system is And the intrinsic parameter matrix K of the RGBD camera and optical machine RGBD , K projector , assuming K RGBD Known.

[0099]

[0100]

[0101] The boundary points of the projectable surface of the four voxels are calculated by the above formula, and the rendered target image is also divided into m×n grids, and the homography matrix from each grid to the voxel surface is calculated. Figure 5 The projection object of this embodiment is shown as follows: Figure 2 ,like Figure 5 As shown, the pixels in the grid of the rendered image are then mapped to the pixel plane of the optical machine through their corresponding homography matrix, thereby completing the rendering of the surface of the target object.

[0102] Furthermore, in an embodiment of the present application, when the projection object is projected in real time according to the real-time posture, the border position information corresponding to the projection object can be first determined based on the real-time posture, and then the coordinates of the border position information can be transformed according to a preset calibration relationship to determine the depth information corresponding to the projection object. Finally, the homography matrix corresponding to the projection object can be determined according to the depth information, and the projection object can be projected in real time based on the homography matrix.

[0103] For example, in the embodiments of the present application, Figure 6 The projection object of this embodiment is shown as follows: Figure 3 ,like Figure 6 As shown, after confirming the projection object through the shooting device, the pixel coordinates of the four corner points of the projection object are obtained, A(u,v), B(u,v), C(u,v), and D(u,v). Through the TOF (Time of Flight) data and the calibration relationship between TOF and the shooting device and the projection device optical machine, the above four pixel points are first mapped to the TOF coordinate system and the depth is obtained, and then the obtained result is converted to the coordinate system of the optical machine, and the homography relationship between the normalized plane of the optical machine and the projection plane composed of ABCD is calculated, so as to project one side of the projection object.

[0104] Furthermore, in the embodiments of the present application, Figure 7 Schematic diagram of the process of implementing the real-time projection method Figure 2 ,like Figure 7 As shown, after acquiring the first image corresponding to the projection object at the first moment by the shooting module in response to the received projection instruction, that is, after step 101, and before performing real-time projection on the projection object according to the real-time posture, that is, before step 104, the method for real-time projection by the projection device may include the following steps:

[0105] Step 105: When it is determined based on the first image that the projection object matches the pre-stored posture database, at least one posture information corresponding to the projection object is obtained from the pre-stored database.

[0106] In an embodiment of the present application, in response to a received projection instruction, after acquiring a first image corresponding to the projection object at a first moment through a shooting module, when it is determined based on the first image that the projection object matches the pre-stored posture database, at least one posture information corresponding to the projection object can be selected from the pre-stored database.

[0107] Furthermore, in an embodiment of the present application, when the first image determines that the projection object matches a pre-stored posture database, at least one posture information corresponding to the projection object is obtained from the pre-stored database. The pre-stored posture database stores a mapping relationship between the projection object and the posture. This mapping relationship may be one-to-one or one-to-many. An object may correspond to one posture, or an object may correspond to many postures. The closest posture information is found in the pre-stored posture database.

[0108] Furthermore, in an embodiment of the present application, after traversing the pre-stored posture database, if there is a feature vector of a stored object in the pre-stored posture database whose similarity with the first feature vector is greater than a preset similarity threshold, it can be considered that the database has stored the posture information corresponding to the projection object, and therefore at least one posture information corresponding to the projection object can be directly obtained from the pre-stored database.

[0109] Step 106: Determine a real-time projection of the projection object according to at least one piece of posture information.

[0110] In an embodiment of the present application, when it is determined based on the first image that the projection object matches a pre-stored posture database, after obtaining at least one posture information corresponding to the projection object in the pre-stored database, the real-time projection of the projection object can be further determined based on the at least one posture information.

[0111] Furthermore, in an embodiment of the present application, when it is determined based on the first image that the projection object matches a pre-stored posture database, when obtaining at least one posture information corresponding to the projection object in the pre-stored database, any one of the posture information can be selected, or part of the posture information can be selected for processing (such as fusion processing, etc.) to generate a real-time projection.

[0112] Furthermore, in the embodiments of the present application, Figure 8 Schematic diagram of the process of implementing the real-time projection method Figure 3 ,like Figure 8 As shown, after acquiring the first image corresponding to the projection object at the first moment by the shooting module in response to the received projection instruction, that is, after step 101, and before performing real-time projection on the projection object according to the real-time posture, that is, before step 104, the method for real-time projection by the projection device may include the following steps:

[0113] Step 107: When it is determined based on the first image that the projection object does not match the pre-stored posture database, obtain a stored object with the highest similarity to the projection object in the pre-stored database, obtain at least one posture information corresponding to the stored object, and determine the real-time projection of the projection object according to the at least one posture information corresponding to the stored object.

[0114] In an embodiment of the present application, after obtaining a first image corresponding to the projection object at a first moment through a shooting module in response to a received projection instruction, when it is determined based on the first image that the projection object does not match a pre-stored posture database, it is possible to select a stored object with the highest similarity to the projection object in the pre-stored database, and then obtain at least one posture information corresponding to the stored object, and determine the real-time projection of the projection object based on the at least one posture information corresponding to the stored object.

[0115] Furthermore, in an embodiment of the present application, if there is a feature vector of a stored object with high similarity to the first feature vector in the pre-stored posture database, it can be considered that the database has stored the posture information corresponding to the projection object; if there is no feature vector of a stored object corresponding to the first feature vector in the pre-stored posture database, when traversing the pre-stored posture database, the similarities between the stored objects in the pre-stored posture database and the first feature vector are sorted to obtain the stored objects in the pre-stored posture database with the highest similarity to the projection object, so that the real-time projection of the projection object can be determined based on at least one posture information corresponding to the stored object with the highest similarity to the projection object.

[0116] In summary, through the real-time projection method proposed in steps 101 to 107, the projection device responds to the received projection instruction and obtains the first image corresponding to the projection object at the first moment through the shooting module; wherein the projection instruction is used to indicate the projection object. One real-time method is to obtain the second image corresponding to the projection object at the second moment through the shooting module when it is determined based on the first image that the projection object does not match the pre-stored posture database. Based on the first image and the second image, the real-time posture corresponding to the projection object is determined; another real-time method is to obtain at least one posture information corresponding to the projection object in the pre-stored database when it is determined based on the first image that the projection object matches the pre-stored posture database, and determine the real-time projection of the projection object according to the at least one posture information; another real-time method is to obtain the stored object with the highest similarity to the projection object in the pre-stored database when it is determined based on the first image that the projection object does not match the pre-stored posture database, obtain at least one posture information corresponding to the stored object, and determine the real-time projection of the projection object according to the at least one posture information corresponding to the stored object. Finally, the projection object can be projected in real time according to the real-time posture.

[0117] The embodiment of the present application provides a real-time projection method, which is applied to a projection device, wherein the projection device includes a shooting module, and the shooting module responds to a received projection instruction, and obtains a first image corresponding to the projection object at a first moment through the shooting module, wherein the projection instruction is used to indicate the projection object, and in the case where it is determined based on the first image that the projection object does not match the pre-stored posture database, the stored object with the highest similarity to the projection object is obtained in the pre-stored database, and at least one posture information corresponding to the stored object is obtained, and a second image corresponding to the projection object is obtained at a second moment through the shooting module; based on the first image and the second image, the real-time posture corresponding to the projection object is determined, and the projection object is projected in real time according to the real-time posture; in the case where it is determined based on the first image that the projection object matches the pre-stored posture database, at least one posture information corresponding to the projection object is obtained in the pre-stored database, and the real-time projection of the projection object is determined according to the at least one posture information; it can be seen that in the embodiment of the present application, the image corresponding to the projection object is continuously obtained through the shooting module, so as to overcome the problems of the projection method of single projection object, delayed projection tracking and lack of rich interactivity of projection content, and can be applied to a variety of projection scenes, and thus can effectively track unknown objects in real time and interactively project them, so as to realize the real-time AR rendering of unknown objects by the projector.

[0118] Based on the above embodiments, another embodiment of the present application provides a real-time projection method, which is applied to a projection device. The projection device includes a shooting module. The shooting module responds to a received projection instruction and obtains a first image corresponding to the projection object at a first moment through the shooting module, wherein the projection instruction is used to indicate the projection object. After the first image is determined to be the projection object and matched with a pre-stored posture database, a second image corresponding to the projection object is obtained at a second moment through the shooting module. Based on the first image and the second image, the real-time posture corresponding to the projection object is determined, and the projection object is projected in real time according to the real-time posture.

[0119] Furthermore, in the embodiments of the present application, Fig. 9 Schematic diagram of the process of implementing the real-time projection method Figure 4 ,like Fig. 9 As shown, the real-time projection method may include the following steps:

[0120] Step 201: Receive a projection instruction.

[0121] The projection device includes a shooting module, and the shooting module can be any type of shooting device. For example, the shooting module can include an IR image sensor, an RGB image sensor, or an RGBD image sensor. The shooting module obtains images through the IR image sensor and / or the RGB image sensor and / or the RGBD image sensor. The shooting module may be configured with multiple shooting modules, and different image sensors are used for different scenes. It is possible to use an IR image sensor at night and an RGB image sensor during the day.

[0122] It should be noted that, in the embodiments of the present application, the projection device can determine the corresponding projection object according to the projection instruction, and the projection instruction can include various forms such as sound, gesture, and button, which are not specifically limited here.

[0123] Step 202: Acquire a first image corresponding to the projection object at a first moment through a shooting module.

[0124] In an embodiment of the present application, after the projection object is determined by a projection instruction, the shooting module acquires a first real-time image at a first moment. The first real-time image is an image directly acquired by the shooting module. After segmenting the first real-time image, the first image corresponding to the projection object can be determined.

[0125] It should be noted that in the embodiment of the present application, when the first real-time image is segmented to obtain the first image, that is, the image of the projection object after segmentation, different neural networks can be used for segmentation. There are many models for image segmentation, which are not limited here.

[0126] It should be noted that in the embodiments of the present application, the neural network used for image segmentation processing can meet the characteristics of not requiring knowledge of the target and the interacting subject, so that the segmentation function can be adapted to a wide range of scenes and objects. In other words, this segmentation neural network can also obtain reasonable output for data that has not been encountered in the training (learning) process, which is a common (generalized) neural network, so that the first real-time image can be directly input, and the output obtained is the first image of the segmented projection object.

[0127] Step 203: Match the projection object determined by the first image with a pre-stored posture database.

[0128] In an embodiment of the present application, after a first image corresponding to a projection object is acquired at a first moment by a shooting module, the projection object determined by the first image is matched with a pre-stored posture database.

[0129] It should be noted that, in an embodiment of the present application, a mapping relationship between projection objects and postures is stored in a pre-stored posture database. This mapping relationship may be one-to-one or one-to-many. One object may correspond to one posture, or one object may correspond to many postures. When the first image determines that the projection object does not match the pre-stored posture database, the stored object with the highest similarity to the projection object is obtained in the pre-stored database, and at least one posture information corresponding to the stored object is obtained.

[0130] It can be understood that in an embodiment of the present application, before matching the pre-stored posture database, the first feature vector corresponding to the first image is first determined, and the pre-stored posture database is traversed based on the first feature vector. A feature extraction algorithm can be used to extract the feature vector. There are many algorithms for feature extraction, which are not limited here, such as SIFT algorithm, SURF algorithm, ORB algorithm, etc.

[0131] It should be noted that, in the embodiment of the present application, the pre-stored posture database is traversed based on the first feature vector, and the search for whether there is a feature vector of a stored object corresponding to the first feature vector in the pre-stored posture database is performed by calculating the similarity of the two feature vectors. There are many algorithms for similarity calculation, which are not limited here. For example, the ORB algorithm is used to extract the descriptor vector of the first image and the projection object feature, and the Euler distance of the descriptor between the first image and the projection object is calculated to determine their similarity. DBOW can also be used for similarity calculation, etc. The feature vector of the stored object corresponding to the first feature vector refers to a difference in similarity. If the difference in similarity is not within the preset threshold range, it is considered that there is no feature vector of the stored object corresponding to the first feature vector in the pre-stored posture database. If the difference in similarity is within the preset threshold range, it is considered that there is a feature vector of the stored object corresponding to the first feature vector in the pre-stored posture database.

[0132] It should be noted that, in the embodiments of the present application, the preset similarity threshold may be any value greater than or equal to 0 and less than or equal to 1. For example, the preset similarity threshold may be 0.01 or 0.001, and the embodiments of the present application do not specifically limit this.

[0133] Furthermore, in an embodiment of the present application, when the first image determines that the projection object matches a pre-stored posture database, at least one posture information corresponding to the projection object is obtained from the pre-stored database. The pre-stored posture database stores a mapping relationship between the projection object and the posture. This mapping relationship may be one-to-one or one-to-many. An object may correspond to one posture, or an object may correspond to many postures. The closest posture information is found in the pre-stored posture database.

[0134] Further, in an embodiment of the present application, when it is determined in the first image that the projection object does not match the pre-stored gesture database, a stored object with the highest similarity to the projection object is obtained in the pre-stored database.

[0135] Step 204: when it is determined based on the first image that the projection object does not match the pre-stored posture database, a second image corresponding to the projection object is acquired at a second moment by a shooting module.

[0136] In an embodiment of the present application, after the projection object determined by the first image is matched with the pre-stored posture database, when it is determined based on the first image that the projection object does not match the pre-stored posture database, it is necessary to obtain a second image corresponding to the projection object at a second moment through the shooting module.

[0137] Furthermore, in an embodiment of the present application, when it is determined based on the first image that the projection object does not match the pre-stored posture database, it is also possible to choose to obtain a stored object with the highest similarity to the projection object in the pre-stored database, obtain at least one posture information corresponding to the stored object, and determine the real-time projection of the projection object based on the at least one posture information corresponding to the stored object.

[0138] Furthermore, in an embodiment of the present application, if there exists a feature vector of a stored object in the pre-stored posture database that has a high similarity with the first feature vector, it can be considered that the database has stored the posture information corresponding to the projection object; if there is no feature vector of a stored object corresponding to the first feature vector in the pre-stored posture database, when traversing the pre-stored posture database, the similarities between the stored objects in the pre-stored posture database and the first feature vector are sorted to obtain the stored objects in the pre-stored database that have the highest similarity with the projection object.

[0139] It should be noted that in the embodiment of the present application, when the second real-time image is segmented to obtain the second image, that is, the segmented image of the projection object, different neural networks can be used for segmentation. There are many models for image segmentation, which are not limited here.

[0140] It should be noted that in the embodiments of the present application, the neural network used for image segmentation processing can meet the characteristics of not requiring knowledge of the target and the interacting subject, so that the segmentation function can be adapted to a wide range of scenes and objects. In other words, this segmentation neural network can also obtain reasonable output for data that has not been encountered in the training (learning) process, which is a common (generalized) neural network, so that the second real-time image can be directly input, and the output obtained is the second image of the segmented projection object.

[0141] It should be noted that, in the embodiments of the present application, the projection device can continuously obtain real-time images corresponding to the projection object through the shooting module, wherein the images corresponding to the projection object refer to the segmented images.

[0142] Furthermore, in the embodiments of the present application, Fig.10 Schematic diagram of the process of implementing the real-time projection method Figure 5 ,like Fig.10 As shown, in an embodiment of the present application, the real-time projection method may include the following steps:

[0143] Step 205: Determine a real-time posture corresponding to the projection object based on the first image and the second image.

[0144] In an embodiment of the present application, after determining that the projection object does not match the pre-stored posture database based on the first image, the first image and the second image corresponding to the projection object are obtained, and the real-time posture corresponding to the projection object is determined. That is, the posture change value corresponding to the projection object is determined based on the first image, the second image and the posture estimation model, the posture change value is input into the posture optimization model, and the real-time posture corresponding to the projection object is output.

[0145] Furthermore, in an embodiment of the present application, the posture change value corresponding to the projection object is determined based on the first image, the second image and the posture estimation model. At this time, the data of the first image and the second image have been stored in the memory, and the interval between obtaining the two images is slightly longer to ensure that the posture of the projection object has a more obvious change.

[0146] It should be noted that in the embodiment of the present application, the first image and the second image are subjected to feature matching processing and constraint estimation processing, the posture change value corresponding to the projection object is determined, the posture change value represents the posture change of the projection object between the first moment and the second moment, the posture change value is input into the posture estimation model, the posture change value is output, and the projection object and the real-time posture are stored in a pre-stored posture database. Different posture estimation models can be used for estimation, and there are many models for posture estimation, which are not limited here, such as the RANSAC posture estimation method, etc.

[0147] Furthermore, in an embodiment of the present application, the posture change value is input into the posture optimization model, and the real-time posture corresponding to the projection object is output, that is, the posture data of the stored key frame is brought into the posture optimization algorithm to obtain a more accurate real-time posture.

[0148] It should be noted that, in the embodiments of the present application, the real-time posture corresponding to the projection object may be posture information corresponding to any time, or posture change information between any two time points, which is not specifically limited in the present application.

[0149] Further, in an embodiment of the present application, when the posture optimization processing is performed on the projection object based on the posture optimization model, the specific process of the posture optimization algorithm is to optimize the posture graph by using the image and posture of the segmented projection object obtained at the current moment, that is, the second image and posture change value corresponding to the current frame, and then performing posture graph optimization calculation with the n historical images corresponding to the projection object stored in the memory. Here, the n historical images stored in the memory need to be less than a preset threshold. Assuming that the postures of n projection objects are recorded in the memory, recorded as T1, ..., T n , when the corresponding posture T at the i-th moment is obtained i and the pose T corresponding to the jth moment j The estimated pose change T ij After that, the cost equation can be constructed as shown in the above formula (1).

[0150] The optimized value Δξ is calculated ij , and use it to update the current estimated T ij The posture of the i-th moment is thus completed. i and the pose T corresponding to the jth moment j The estimated pose change T ij Optimization.

[0151] It should be noted that, in the embodiments of the present application, the preset threshold value may be any value greater than or equal to 0. For example, the preset threshold value may be 100 or 200, and the embodiments of the present application do not specifically limit this.

[0152] Step 207: When it is determined based on the first image that the projection object matches the pre-stored posture database, at least one posture information corresponding to the projection object is obtained from the pre-stored database.

[0153] In an embodiment of the present application, in response to a received projection instruction, after acquiring a first image corresponding to the projection object at a first moment through a shooting module, when it is determined based on the first image that the projection object matches a pre-stored posture database, at least one posture information corresponding to the projection object can be selected from the pre-stored database.

[0154] Further, in an embodiment of the present application, after traversing the pre-stored posture database, if there is a feature vector of a stored object in the pre-stored posture database that has a high similarity to the first feature vector, it can be considered that the database has stored the posture information corresponding to the projection object.

[0155] Step 208: Determine the real-time projection of the projection object according to at least one piece of posture information.

[0156] In an embodiment of the present application, when it is determined based on the first image that the projection object matches a pre-stored posture database, after obtaining at least one posture information corresponding to the projection object in the pre-stored database, the real-time projection of the projection object can be further determined based on the at least one posture information.

[0157] Furthermore, in an embodiment of the present application, when it is determined based on the first image that the projection object matches a pre-stored posture database, when obtaining at least one posture information corresponding to the projection object in the pre-stored database, any one of the posture information can be selected, or part of the posture information can be selected for processing (fusion, etc.) to generate a real-time projection.

[0158] Furthermore, in an embodiment of the present application, when the first image determines that the projection object matches a pre-stored posture database, at least one posture information corresponding to the projection object is obtained from the pre-stored database. The pre-stored posture database stores a mapping relationship between the projection object and the posture. This mapping relationship may be one-to-one or one-to-many. An object may correspond to one posture, or an object may correspond to many postures. The closest posture information is found in the pre-stored posture database.

[0159] Furthermore, in the embodiments of the present application, Fig.11 Schematic diagram of the process of implementing the real-time projection method Figure 6 ,like Fig.11 As shown, in an embodiment of the present application, the real-time projection method may include the following steps:

[0160] Step 209: when it is determined based on the first image that the projection object does not match the pre-stored posture database, obtain a stored object with the highest similarity to the projection object in the pre-stored database, and obtain at least one posture information corresponding to the stored object.

[0161] In an embodiment of the present application, in response to a received projection instruction, after obtaining a first image corresponding to the projection object at a first moment through a shooting module, when it is determined based on the first image that the projection object does not match the pre-stored posture database, it is possible to select a stored object in the pre-stored database with the highest similarity to the projection object, and obtain at least one posture information corresponding to the stored object.

[0162] Step 206: Project the projection object in real time according to the real-time posture.

[0163] In an embodiment of the present application, after determining the real-time posture corresponding to the projection object based on the first image and the second image, the border position information corresponding to the projection object is determined based on the real-time posture, the coordinates of the border position information are transformed according to a preset calibration relationship, the depth information corresponding to the projection object is determined, the homography matrix corresponding to the projection object is determined according to the depth information, and the projection object is projected in real time based on the homography matrix.

[0164] Furthermore, in an embodiment of the present application, when the projection object is projected in real time according to the real-time posture, the position information of at least one pixel of the projection surface corresponding to the projection object is determined based on the real-time posture; the position information is coordinate-converted according to a preset calibration relationship to determine the homography matrix of at least one pixel; and the projection object is projected in real time based on the homography matrix of at least one pixel.

[0165] For example, in the embodiments of the present application, Figure 4 As shown in the figure, after confirming the target object (projection object) through the camera, the three-dimensional information of the target object surface is obtained by estimating the object's posture and acquiring the RGBD information, and the three-dimensional object is divided into N voxels, whose projection surface consists of m×n voxels, and the four three-dimensional coordinate corner points of each voxel are obtained. The transformation matrix from the known RGBD camera coordinate system to the optomechanical coordinate system is And the intrinsic parameter matrix K of the RGBD camera and optical machine RGBD , K projector , assuming K RGBD The boundary points of the projectable surfaces of the four voxels are calculated by using the above equations (2) and (3), and the rendered target image is also divided into m×n grids. The homography matrix from each grid to the voxel surface is calculated, as follows: Figure 5 As shown, the pixels in the grid of the rendered image are then mapped to the pixel plane of the optical machine through their corresponding homography matrix, thereby completing the rendering of the surface of the target object.

[0166] For example, in the embodiments of the present application, Figure 6 As shown in the figure, after confirming the target object through the camera, the pixel coordinates of the four corner points of the object are obtained, A(u,v), B(u,v), C(u,v), and D(u,v). Through the TOF data and the calibration relationship between TOF, camera, and projector optical machine, the above four pixel points are first mapped to the TOF coordinate system and the depth is obtained, and then the obtained result is converted to the coordinate system of the optical machine, and the homography relationship between the normalized plane of the optical machine and the projection plane composed of ABCD is calculated, so as to project one side of the target object.

[0167] Furthermore, in an embodiment of the present application, when it is determined based on the first image that the projection object does not match the pre-stored posture database, a stored object with the highest similarity to the projection object is obtained in the pre-stored database, at least one posture information corresponding to the stored object is obtained, and the real-time projection of the projection object is determined based on the at least one posture information corresponding to the stored object.

[0168] Further, in an embodiment of the present application, when it is determined based on the first image that the projection object matches a pre-stored posture database, at least one posture information corresponding to the projection object is obtained in the pre-stored database, and the real-time projection of the projection object is determined according to the at least one posture information.

[0169] The embodiment of the present application provides a real-time projection method, which is applied to a projection device. The projection device includes a shooting module. The shooting module of the projection device responds to a received projection instruction. The projection device obtains a first image corresponding to a projection object at a first moment through the shooting module. The first image corresponding to the projection object is obtained by inputting a first real-time image obtained by the shooting module at the first moment into an image segmentation model. The projection instruction is used to indicate the projection object. The projection instruction determines the projection object according to some special signals. After matching the first feature vector corresponding to the first image of the projection object with a mapping relationship between the projection object and the posture stored in a pre-stored posture database, the projection object is determined by the first image and matched with the pre-stored posture database. At a second moment, a second image corresponding to the projection object is obtained through the shooting module. Based on the first image, the second image and the posture estimation model, a posture change value corresponding to the projection object is determined. The posture change value is then input into a posture optimization model, and a real-time posture corresponding to the projection object is output. Based on the real-time posture, the frame position information corresponding to the projection object is determined. The frame position information is converted into coordinates according to a preset calibration relationship. The depth information corresponding to the projection object is determined. The homography matrix corresponding to the projection object is determined according to the depth information. The projection object is projected in real time based on the homography matrix.

[0170] Based on the above embodiment, another embodiment of the present application proposes a real-time projection method, which is applied to a projection device. First, visual information is obtained through a camera (shooting module) carried by the projection device. When a special signal (projection instruction) is detected, the frame at this moment is recorded as F. t-1 , segment the signal area or the indicated object in the frame, and thus determine the target object (projection object). Secondly, after determining the target object, match the target object with the database (pre-stored posture database). If the match is unsuccessful or the database is empty, after determining the target object, F t-1 and F t The image data of the frame (i.e., the first image and the second image) are input into the pose estimation algorithm (pose estimation model) to obtain a more accurate current pose of the target object.t ∈SE(3) (the attitude change value corresponding to the projection object). Again, the acquired attitude data of the target object is input into the attitude graph optimization algorithm (attitude optimization model) to update the currently estimated attitude, thereby obtaining a more accurate estimated attitude (the real-time attitude corresponding to the projection object). In addition, after obtaining the real-time attitude of the target object, the projection device saves the attitude and its corresponding number of images to obtain an attitude map (pre-stored attitude database). If the match is successful, the target object is matched with the attitude map historically stored in the database to obtain the real-time attitude of the target object. Finally, the homography matrix between the surface of the target object and the optical-mechanical normalized plane of the projector is calculated based on the acquired attitude of the object and the point cloud data of the target object, thereby realizing the rendering of the target object.

[0171] Furthermore, in an embodiment of the present application, the projection device may be a projector, and the shooting device may be an RGBD camera. The method proposed in the embodiment of the present application is described below using a projector and an RGBD camera as an example.

[0172] For example, in the embodiments of the present application, Fig.12 A schematic diagram of a projector proposed in an embodiment of the present application, such as Fig.12 As shown, this embodiment is a projector equipped with a front RGBD camera.

[0173] Fig.13 Schematic diagram of the process of implementing the real-time projection method Figure 7 ,like Fig.13 As shown, the method for real-time projection by a projector may include the following steps:

[0174] Step 301: The projector finds and segments the target object specified by the user through the front camera and the neural network.

[0175] For example, in the embodiment of the present application, the projector needs to know the approximate position of the target object specified by the user through voice triggering, gesture triggering or key triggering, and the target object can be the projection object. t-1 The target object of the frame is segmented out (the neural network needs to conform to the characteristics of not requiring knowledge of the target and the interacting subject, so that the segmentation function can be adapted to a wide range of scenes and objects) and recorded.

[0176] Step 302: Determine whether the target object has been recorded.

[0177] Exemplarily, in an embodiment of the present application, after the projector finds and segments the target object specified by the user through the front camera and the neural network, the projection will match the currently acquired target object image A with the first image B in the posture map, and the similarity can be calculated by the following method. The descriptor vectors of the features A and B are extracted using the ORB algorithm, and the Euler distance of the descriptors between A and B is calculated to determine their similarity. DBOW can also be used for similarity calculation. If the difference in similarity is not within the preset threshold range, it is considered that the target object has not been recorded, that is, there is no feature vector of the stored object corresponding to the first feature vector of the target object in the pre-stored posture database. If the difference in similarity is within the preset threshold range, it is considered that the target object has been recorded, that is, there is a feature vector of the stored object corresponding to the first feature vector of the target object in the pre-stored posture database.

[0178] It should be noted that, in the embodiments of the present application, the preset similarity threshold may be any value greater than or equal to 0 and less than or equal to 1. For example, the preset similarity threshold may be 0.01 or 0.001, and the embodiments of the present application do not specifically limit this.

[0179] Step 303: The projector obtains two frames of image data of the target object and preliminarily estimates the posture of the target object.

[0180] Exemplarily, in the embodiment of the present application, the projector obtains F t Image data, and roughly estimate the position of the target object relative to the projector. At this time, the projector memory stores F t-1 Points and F t The target object data obtained by segmentation in the above two frames is slightly longer to ensure that the position and posture of the target object have a more obvious change. By matching the features of the two frames and estimating their position in F through epipolar constraints t The pose of the frame ξ t ∈SE(3), and bring it into the RANSAC-based pose estimation method as the initial value, so as to obtain the most votes and the most likely ξ t ∈SE(3). Assume that time t-1 is the initial position of the target object, ξ t is the change value of the target object’s posture at time t relative to the initial posture, and the subsequent posture values ​​are also changes relative to the initial posture. Then get the most likely ξ t , store the posture and the corresponding image into memory.

[0181] After the projector acquires two frames of image data of the target object and preliminarily estimates the posture of the target object, if the target object has not been recorded, step 304 is executed, otherwise step 306 is executed.

[0182] Step 304: Bring the acquired posture data of the target object key frame into the posture optimization algorithm to obtain a more accurate real-time posture.

[0183] Exemplarily, in an embodiment of the present application, the posture data of the stored key frame is brought into the posture optimization algorithm to obtain a more accurate real-time posture. When the projection object is subjected to posture optimization processing based on the posture optimization model, the specific process of the posture optimization algorithm is to use the image of the segmented projection object obtained at the current moment (the image successfully matches the stored image, which means that a loop has occurred between the current frame and the historical frame) and the posture, that is, the second image and posture change value corresponding to the current frame, and then perform posture graph optimization calculation with the n historical images corresponding to the projection object stored in the memory. Here, the n historical images stored in the memory need to be less than a preset threshold. Assume that the postures of n projection objects are recorded in the memory, which are recorded as T1, ..., T n , when the corresponding posture T at the i-th moment is obtained i and the pose T corresponding to the jth moment j The estimated pose change T ij After that, the cost equation can be constructed as shown in the above formula (1).

[0184] The optimized value Δξ is calculated ij , and use it to update the current estimated T ij The posture of the i-th moment is thus completed. i and the pose T corresponding to the jth moment j The estimated pose change T ij Optimization.

[0185] Step 305: record the key frame posture of the object and construct a posture map of the object.

[0186] Exemplarily, in an embodiment of the present application, the projector records the key frame image of the target object and its corresponding posture, thereby forming a database consisting of images and the posture of the target object, and the first image of the generated database is constructed into a bag-of-words library. The pre-stored posture database stores the mapping relationship between the projected object and the posture. This mapping relationship may be one-to-one or one-to-many. One object may correspond to one posture, or one object may correspond to many postures.

[0187] Step 306: Match the target object with the posture map to obtain the real-time posture.

[0188] Exemplarily, in an embodiment of the present application, the posture of the current image is obtained by searching and matching image A with the bag-of-words database. Before matching the pre-stored posture database, the first feature vector corresponding to image A is first determined, and the pre-stored posture database is traversed based on the first feature vector. The feature vector can be extracted using a feature extraction algorithm. There are many algorithms for feature extraction, which are not limited here, such as SIFT algorithm, SURF algorithm, ORB algorithm, etc.

[0189] It should be noted that, in the embodiment of the present application, the pre-stored posture database is traversed based on the first feature vector, and the search for whether there is a feature vector of a stored object corresponding to the first feature vector in the pre-stored posture database is performed by calculating the similarity of the two feature vectors. There are many algorithms for similarity calculation, which are not limited here. For example, the ORB algorithm is used to extract the descriptor vectors of the A and B features, and the Euler distance between the descriptors of A and B is calculated to determine their similarity. DBOW can also be used for similarity calculation, etc. The feature vector of the stored object corresponding to the first feature vector refers to a difference in similarity. If the difference in similarity is not within the preset threshold range, it is considered that there is no feature vector of the stored object corresponding to the first feature vector in the pre-stored posture database. If the difference in similarity is within the preset threshold range, it is considered that there is a feature vector of the stored object corresponding to the first feature vector in the pre-stored posture database.

[0190] It should be noted that, in the embodiments of the present application, the preset similarity threshold may be any value greater than or equal to 0 and less than or equal to 1. For example, the preset similarity threshold may be 0.01 or 0.001, and the embodiments of the present application do not specifically limit this.

[0191] Furthermore, in an embodiment of the present application, when the first image determines that the projection object matches a pre-stored posture database, at least one posture information corresponding to the projection object is obtained from the pre-stored database. The pre-stored posture database stores a mapping relationship between the projection object and the posture. This mapping relationship may be one-to-one or one-to-many. An object may correspond to one posture, or an object may correspond to many postures. The closest posture information is found in the pre-stored posture database.

[0192] Further, in an embodiment of the present application, when the first image determines that the projection object does not match the pre-stored posture database, a stored object with the highest similarity to the projection object is obtained in the pre-stored database, and at least one posture information corresponding to the stored object is obtained.

[0193] Step 307: Calculate the homography matrix from the surface of the target object to the optomechanical normalized plane, and perform projection rendering on it.

[0194] For example, in the embodiments of the present application, Figure 4 As shown in the figure, after confirming the target object (projection object) through the camera, the three-dimensional information of the target object surface is obtained by estimating the object's posture and acquiring the RGBD information, and the three-dimensional object is divided into N voxels, whose projection surface consists of m×n voxels, and the four three-dimensional coordinate corner points of each voxel are obtained. The transformation matrix from the known RGBD camera coordinate system to the optomechanical coordinate system is And the intrinsic parameter matrix K of the RGBD camera and optical machine RGBD , K projector , assuming K RGBD The boundary points of the projectable surfaces of the four voxels are calculated by using the above equations (2) and (3), and the rendered target image is also divided into m×n grids. The homography matrix from each grid to the voxel surface is calculated, as follows: Figure 5 As shown, the pixels in the grid of the rendered image are then mapped to the pixel plane of the optical machine through their corresponding homography matrix, thereby completing the rendering of the surface of the target object.

[0195] Exemplarily, in an embodiment of the present application, a homography matrix from a projector to a target object surface is calculated to achieve rendering of the target object surface. Figure 6 As shown in the figure, after confirming the target object through the camera, the pixel coordinates of the four corner points of the object are obtained, A(u,v), B(u,v), C(u,v), and D(u,v). Through the TOF data and the calibration relationship between TOF, camera, and projector optical machine, the above four pixel points are first mapped to the TOF coordinate system and the depth is obtained, and then the obtained result is converted to the coordinate system of the optical machine, and the homography relationship between the normalized plane of the optical machine and the projection plane composed of ABCD is calculated, so as to project one side of the target object.

[0196] The embodiment of the present application provides a real-time projection method, which is applied to a projection device. The projection device includes a shooting module. The shooting module of the projection device responds to a received projection instruction. The projection instruction determines a projection object according to some special signals. The projection device obtains a first image corresponding to the projection object at a first moment through the shooting module. The first image corresponding to the projection object is obtained by inputting a first real-time image obtained by the shooting module at the first moment into an image segmentation model. After matching a first feature vector corresponding to the first image of the projection object with a mapping relationship between the projection object and the posture stored in a pre-stored posture database, the projection object is determined by matching the first image with the pre-stored posture database, and then a second image corresponding to the projection object is obtained at a second moment through the shooting module. Based on the first image, the second image and the posture estimation model, a posture change value corresponding to the projection object is determined, and then the posture change value is input into a posture optimization model, and the real-time posture corresponding to the projection object is output. Based on the real-time posture, the frame position information corresponding to the projection object is determined, the frame position information is converted into coordinates according to a preset calibration relationship, the depth information corresponding to the projection object is determined, the homography matrix corresponding to the projection object is determined according to the depth information, and the projection object is projected in real time based on the homography matrix.

[0197] Based on the above embodiment, in another embodiment of the present application, Fig.14 Schematic diagram of the structure of the projection device proposed in the embodiment of the present application Figure 2 ,like Fig.14 As shown, the projection device 10 proposed in the embodiment of the present application may include an acquisition unit 110, a determination unit 111, and a projection unit 112, wherein:

[0198] The determination unit 110 is used to obtain a first image corresponding to the projection object at a first moment through the shooting module in response to a received projection instruction; wherein the projection instruction is used to indicate the projection object; and is also used to obtain a second image corresponding to the projection object at a second moment through the shooting module when it is determined based on the first image that the projection object does not match the pre-stored posture database.

[0199] The determining unit 111 is used to determine a real-time posture corresponding to the projection object based on the first image and the second image.

[0200] The projection unit 112 is used to project the projection object in real time according to the real-time posture.

[0201] In the embodiments of the present application, further, Fig.15 Schematic diagram of the structure of the projection device proposed in the embodiment of the present application Figure 3 ,like Fig.15As shown, the projection device 10 proposed in the embodiment of the present application may also include a processor 121 , a memory 122 , a communication interface 123 , and a bus 124 for connecting the processor 121 , the memory 122 and the communication interface 123 .

[0202] In the embodiment of the present application, the processor 121 may be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, and a microprocessor. It is understandable that for different devices, the electronic device used to implement the function of the processor may also be other, and the embodiment of the present application is not specifically limited. The projection device 10 may also include a memory 122, which may be connected to the processor 121, wherein the memory 122 is used to store executable program code, the program code includes computer operation instructions, and the memory 122 may include a high-speed RAM memory, and may also include a non-volatile memory, for example, at least two disk memories.

[0203] In the embodiment of the present application, the bus 124 is used to connect the communication interface 123, the processor 121 and the memory 122, and the mutual communication between these devices.

[0204] In practical applications, the memory 122 may be a volatile memory, such as a random access memory (RAM); or a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or a combination of the above types of memories, and provide instructions and data to the processor 121.

[0205] Further, in an embodiment of the present application, the processor 121 is used to store a computer program that can be run on the processor;

[0206] The memory 122 is used to execute the method according to any one of claims 1 to 12 when running the computer program.

[0207] In addition, each functional module in this embodiment can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or software functional modules.

[0208] If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method of this embodiment. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk, etc., which can store program code.

[0209] An embodiment of the present application provides a computer-readable storage medium having a program stored thereon, and when the program is executed by a processor, the real-time projection method as described above is implemented.

[0210] Specifically, the program instructions corresponding to a real-time projection method in this embodiment can be stored in a storage medium such as a CD, a hard disk, a USB flash drive, etc. When the program instructions corresponding to a real-time projection method in the storage medium are read or executed by an electronic device, the following steps are included:

[0211] The shooting module responds to the received projection instruction and acquires a first image corresponding to the projection object at a first moment through the shooting module; wherein the projection instruction is used to indicate the projection object;

[0212] In a case where it is determined based on the first image that the projection object does not match the pre-stored posture database, acquiring, by the shooting module, a second image corresponding to the projection object at a second moment;

[0213] Determining a real-time posture corresponding to the projection object based on the first image and the second image;

[0214] The projection object is projected in real time according to the real-time posture.

[0215] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of hardware embodiments, software embodiments, or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0216] The present application is described with reference to implementation flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0217] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which is implemented in the implementation flow diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0218] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing the steps in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0219] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application.

Claims

1. A real-time projection method, characterized in that: The projection method is applied to a projection device, the projection device includes a shooting module, and the method includes: In response to the received projection instruction, acquiring a first image corresponding to the projection object at a first moment through the shooting module; wherein the projection instruction is used to indicate the projection object; In a case where it is determined based on the first image that the projection object does not match the pre-stored posture database, acquiring, by the shooting module, a second image corresponding to the projection object at a second moment; Determining a real-time posture corresponding to the projection object based on the first image and the second image; The projection object is projected in real time according to the real-time posture.

2. The method according to claim 1, characterized in that After acquiring, by the shooting module, a first image corresponding to the projection object at a first moment in response to the received projection instruction, and before performing real-time projection on the projection object according to the real-time posture, the method further includes: In a case where it is determined based on the first image that the projection object matches the pre-stored posture database, obtaining at least one posture information corresponding to the projection object in the pre-stored database; The real-time projection of the projection object is determined according to the at least one posture information.

3. The method according to claim 1, characterized in that The determining, based on the first image and the second image, a real-time posture corresponding to the projection object includes: The first image and the second image are input into a posture estimation model, and a posture change value corresponding to the projection object is output to determine the real-time posture corresponding to the projection object; wherein the posture change value represents the posture change of the projection object between the first moment and the second moment.

4. The method according to claim 3, characterized in that The method further comprises: The posture change value is updated by using the historical posture change data corresponding to the projection object through a posture optimization model to determine the real-time posture.

5. The method according to claim 4, characterized in that The method further comprises: The projection object and the real-time posture are stored correspondingly in the pre-stored posture database.

6. The method according to any one of claims 1 to 5, characterized in that: The performing real-time projection on the projection object according to the real-time posture comprises: Based on the real-time posture, determining position information of at least one pixel of the projection surface corresponding to the projection object; Performing coordinate transformation on the position information according to a preset calibration relationship to determine a homography matrix of the at least one pixel; The projection object is projected in real time based on the homography matrix of the at least one pixel.

7. The method according to claim 2, characterized in that The method further comprises: determining a first eigenvector corresponding to the first image; The pre-stored gesture database is traversed based on the first feature vector, and when a feature vector of a stored object corresponding to the first feature vector exists in the pre-stored gesture database, it is determined that the projected object matches the pre-stored gesture database.

8. A projection device, characterized in that: The projection device comprises: an acquisition unit, a determination unit, and a projection unit. The acquisition unit is used to acquire a first image corresponding to the projection object at a first moment through the shooting module in response to the received projection instruction; wherein the projection instruction is used to indicate the projection object; The acquisition unit is further configured to acquire, through the shooting module, a second image corresponding to the projection object at a second moment when it is determined based on the first image that the projection object does not match the pre-stored posture database; The determining unit is used to determine a real-time posture corresponding to the projection object based on the first image and the second image; The projection unit is used to perform real-time projection on the projection object according to the real-time posture.

9. A projection device, characterized in that: The projection device comprises: a processor and a memory; wherein, The memory is used to store a computer program that can be run on the processor; The processor is configured to execute the method according to any one of claims 1 to 7 when running the computer program.

10. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images