Projection control method, apparatus, and system

Abstract

The application discloses a projection control method, device and system. The method comprises: acquiring a shooting image corresponding to a projection image, the shooting image comprising an identification motion track corresponding to an identification marking the projection image; determining a marking track corresponding to the identification motion track in the projection image according to a mapping parameter and the shooting image; performing skeleton extraction on the marking track to obtain a skeleton track; acquiring position information of a midpoint of the skeleton track as marking position information; responding to an instruction event corresponding to the identification; and controlling the projection image to change based on the instruction event and the marking position information. In the application, the marking position information corresponding to the marking is determined according to the skeleton track corresponding to the marking track, the marking position information can accurately reflect the position indicated by the identification, thereby improving the positioning accuracy of the identification and the interactive accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image projection, and more particularly, to a projection control method, device and system. BACKGROUND

[0002] At present, in a teaching or conference scenario, a user marks a projection image projected by a projector by means of a marker device such as a laser remote controller, or sends an interactive event to the projection image by means of the marker device such as a remote controller, and then a projection controller controls the projection image to change according to position information of the marker and the sent interactive event, so as to realize interaction between the user and the projection image.

[0003] However, when interacting with the projection image by means of the marker device such as a remote controller by using the conventional projection control method, the interaction accuracy is low. SUMMARY

[0004] In view of the above problems, the present application provides a projection control method, device and system, which can improve the interaction accuracy.

[0005] In a first aspect, an embodiment of the present application provides a projection control method, and the method comprises:

[0006] obtaining a shooting image corresponding to a projection image, the shooting image comprising a marker motion trail corresponding to a marker marking the projection image;

[0007] determining a marked trail corresponding to the marker motion trail in the projection image according to a mapping parameter and the shooting image, the mapping parameter being used to represent a mapping relationship between the projection image and the shooting image;

[0008] performing skeleton extraction on the marked trail to obtain a skeleton trail;

[0009] obtaining position information of a midpoint of the skeleton trail as marker position information;

[0010] responding to an instruction event corresponding to the marker;

[0011] controlling the projection image to change based on the instruction event and the marker position information.

[0012] In a second aspect, an embodiment of the present application provides a projection control device, and the device comprises:

[0013] an image obtaining module, configured to obtain a shooting image corresponding to a projection image, the shooting image comprising a marker motion trail corresponding to a marker marking the projection image;

[0014] determining, according to the mapping parameter and the photographed image, a marked trajectory corresponding to the identified motion trajectory in the projection image, the mapping parameter being used to represent a mapping relationship between the projection image and the photographed image;

[0015] extracting, by an extracting module, a skeleton of the marked trajectory to obtain a skeleton trajectory;

[0016] obtaining, by a position obtaining module, position information of a midpoint of the skeleton trajectory as marker position information;

[0017] responding, by a responding module, to an instruction event corresponding to the marker;

[0018] controlling, by a controlling module, the projection image to change based on the instruction event and the marker position information.

[0019] In a third aspect, an embodiment of the present application provides a projection control device, comprising:

[0020] one or more processors;

[0021] a memory;

[0022] one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method described above.

[0023] In a fourth aspect, an embodiment of the present application provides a projection control system, comprising at least a camera, a projector, a remote controller and a projection control device, wherein:

[0024] the projector is configured to project a projection image;

[0025] the remote controller is configured to obtain a photographed image corresponding to the projection image;

[0026] the remote controller is configured to emit a marker;

[0027] the projection control device is configured to execute the method of the first aspect described above.

[0028] The projection control method, device and system provided by the embodiments of the present application can determine a marked trajectory corresponding to a marker motion trajectory in a projection image, and then determine marker position information corresponding to the marker according to a skeleton trajectory corresponding to the marked trajectory. The marker position information can accurately reflect a position indicated by the marker, thereby improving positioning accuracy of the marker and further improving interactive accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments description will be briefly introduced. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0030] Figure 1 A schematic diagram of a projection control system suitable for an application environment of the embodiments of the present application is shown.

[0031] Figure 2 Projection schematic diagrams of different scenarios in the embodiments of the present application are shown.

[0032] Figure 3 A flowchart of a projection control method according to an embodiment of the present application is shown.

[0033] Figure 4 A schematic diagram of a skeleton track in the embodiments of the present application is shown.

[0034] Figure 5 A flowchart of a method for obtaining mapping parameters in the embodiments of the present application is shown.

[0035] Figure 6 A flowchart of another method for obtaining mapping parameters in the embodiments of the present application is shown.

[0036] Figure 7 A flowchart of still another method for obtaining mapping parameters in the embodiments of the present application is shown.

[0037] Figure 8 A schematic diagram of a process for obtaining a second decoding result in the embodiments of the present application is shown.

[0038] Figure 9 A flowchart of still another method for obtaining mapping parameters in the embodiments of the present application is shown.

[0039] Figure 10a A schematic diagram of a fourth calibration image in the embodiments of the present application is shown.

[0040] Figure 10b A schematic diagram of still another fourth calibration image in the embodiments of the present application is shown.

[0041] Figure 11 A schematic diagram of a fourth calibration image in the embodiments of the present application is shown. Figure 10a A schematic diagram of a fourth calibration image in the embodiments of the present application is shown.

[0042] Figure 12 A block diagram of a projection control device according to an embodiment of the present application is shown.

[0043] Figure 13A block diagram of a projection control device is shown, which illustrates another embodiment of the present application.

[0044] Figure 14 A structural block diagram of a computer readable storage medium is shown, which illustrates an embodiment of the present application. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without creative work under the premise that the embodiments in the present application are within the scope of protection of the present application.

[0046] In the following description, the terms "first" and "second" are only to distinguish similar objects, and do not represent the specific order of the objects. It can be understood that the "first" and "second" can be interchanged in the specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0048] In the conventional projection control method, the position information of the mark in the projection image is determined in the photographed image corresponding to the projection image, and the position information is taken as the position information of the mark in the projection image. Generally, when the mark moves too fast, the mark in the photographed image appears in the form of a mark motion trajectory, and the position information of the centroid of the mark motion trajectory can be taken as the position information of the mark. However, when the position information of the centroid of the mark motion trajectory is taken as the position information of the mark, the position information is difficult to accurately reflect the position indicated by the mark, resulting in poor accuracy of the position information, and further resulting in low interactive accuracy.

[0049] Therefore, the inventors propose a projection control method, device and system. The projection control method comprises: obtaining a shooting image corresponding to a projection image, the shooting image comprising an identification motion track corresponding to an identification marking the projection image; determining a marking track corresponding to the identification motion track in the projection image according to a mapping parameter and the shooting image, the mapping parameter being used to represent a mapping relationship between the projection image and the shooting image; performing skeleton extraction on the marking track to obtain a skeleton track; obtaining position information of a midpoint of the skeleton track as marking position information; in response to an instruction event corresponding to the identification; and controlling the projection image to change based on the instruction event and the marking position information. In the present application, the marking position information corresponding to the identification is determined according to the skeleton track corresponding to the marking track, and the marking position information can accurately reflect the position indicated by the identification, thereby improving the positioning accuracy of the identification and the interactive accuracy.

[0050] With reference to Figure 1 , Figure 1 A schematic diagram of a projection control system suitable for an application environment of embodiments of the present application is shown. The projection control system comprises a projector 101, a camera 102, a projection control device 103 and a remote controller 104.

[0051] The projector 101 can be any type or structure of projector, and the resolution of the projector can be 1080p, 2K or 4K, etc. The projector can be placed in a fixed position, and the projector can project a to-be-projected image obtained by the projector onto a projection area 106 (the area corresponding to the dashed box). The projection area can be provided with a white projection screen, and the white projection screen displays the to-be-projected image projected thereon. The to-be-projected image displayed on the white projection screen is the projection image in the present application.

[0052] The camera 102 can be a high-definition camera or a digital camera, and the camera 102 can have a zoom function. The camera 102 is used to capture the projection image of the projection area. The shooting image captured by the camera can at least include the projection image in the projection area, and the shooting image captured by the camera can also include other contents outside the projection area. The shooting area 105 (the area corresponding to the solid box) of the camera includes the projection area 106, and the shooting area 105 is larger than the projection area 106.

[0053] The remote controller 103 is used to mark the projection image in the projection area. The remote controller is used to emit an identification 107, and the identification 107 is used to mark the projection image. The remote controller can be a remote controller emitting laser or colored rays.

[0054] The projection control device 104 is configured to perform the steps of the projection control method. The projection control device can be any type of electronic device including a processor and a memory, such as a computer, a tablet, and the like. The projection control device can receive the projection image sent by the projector and the captured image captured by the camera, and perform positioning according to the projection image and the captured image. The projection control device 104 can also control the projection image to change based on an interaction event. The interaction event can be an event of withdrawing, restoring, editing, drawing a track, and viewing attributes of the position indicated by the marker.

[0055] In some embodiments, the projector implements part or all of the projection control device through software or hardware functions, that is, the projector can also serve as the projection control device to perform the steps of the projection control method.

[0056] Reference is made to Figure 2 , Figure 2 The projection schematic diagrams of different scenarios in the embodiments of the present application are shown. In Figure 2 three different scenarios are involved: scenario 1, scenario 2, and scenario 3 (the different projection surfaces are represented by thin lines), wherein the projection surface of scenario 3 is a curved surface. Figure 2 The angle between the two dashed lines in the figure represents the projection visual angle of the projector. The thick solid lines in the projection surfaces of different scenarios represent the projection regions in the projection surfaces.

[0057] Reference is made to Figure 3 , Figure 3 The projection control method flowchart of one embodiment of the present application is shown. The method is used for a projection control device, and the method includes the following steps.

[0058] S110, a captured image corresponding to a projection image is acquired. The captured image includes a marker motion track corresponding to a marker used to mark the projection image.

[0059] In the present application, the projector projects the acquired to-be-projected image on a projection region, and the projection region displays the projection image. A user can emit a marker through a remote controller, and the emitted marker is used to mark the content in the projection image. The marker emitted by the remote controller can be of any shape, such as a dot, a square dot, a cursor, a palm-shaped light dot, and the like. For example, Figure 1 as shown, the marker is a dot.

[0060] When the marker emitted by the remote controller moves too fast, the marker in the captured image corresponding to the marker is a motion track of the marker, and the motion track is used as the marker motion track. When the marker is a dot, the marker motion track can be an elliptical track, a linear track, a curved track, and the like.

[0061] In some embodiments, after the projector projects the projection image, the camera captures the projection image of the projection area in real time, and the projection control device analyzes the captured image in real time. If the remote controller emits the mark to mark the projection image at a certain time, the captured image obtained has the track corresponding to the mark, and if the remote controller does not emit any mark at a certain time, the captured image obtained does not have the track corresponding to the mark.

[0062] In other embodiments, when the remote controller emits the mark, the remote controller generates a trigger signal, the remote controller sends the trigger signal to the projection control device, the projection control device controls the camera to obtain the captured image of the projection area, and then the projection control device obtains the captured image and analyzes the captured image. In this embodiment, the camera does not need to capture at all times, and the projection control device does not need to analyze the captured image at all times, which can effectively reduce the energy consumption of the projection control device and the camera.

[0063] S120, determining a marked track corresponding to the mark motion track in the projection image according to the mapping parameter and the captured image, the mapping parameter being used to represent the mapping relationship between the projection image and the captured image.

[0064] It should be noted that the mapping parameter can represent the parameter of the mapping relationship between the pixel point in the projection image and the pixel point in the captured image. The pixel point in the projection image can be mapped in the captured image according to the mapping parameter, or the pixel point in the captured image can be mapped in the projection image according to the mapping parameter.

[0065] The mapping parameter can be obtained according to the calibration image and the captured image corresponding to the calibration image, wherein the calibration image is the image displayed by the projection area when the projector projects the calibration template on the projection area, and the captured image obtained by capturing the capturing area by the camera is the captured image corresponding to the calibration image. The mapping parameter is determined according to the corresponding relationship between the calibration image projected by the projector and the captured image corresponding to the calibration image, and the mapping parameter can reflect the mapping relationship between the projection image projected by the projector and the corresponding captured image. The marked track can refer to the track obtained by mapping each pixel point in the mark motion track in the projection image.

[0066] Optionally, when the resolution of the photographed image is greater than the resolution of the projection image, the determining the annotation trajectory corresponding to the marker trajectory in the projection image according to the mapping parameter and the photographed image comprises: determining a region corresponding to the projection image in the photographed image as a target region; mapping pixel points in the target region to a first target image having the same size as the projection image according to the mapping parameter to obtain an aligned image; and determining the annotation trajectory corresponding to the marker trajectory in the aligned image.

[0067] When the resolution of the photographed image is greater than the resolution of the projection image, a region corresponding to the projection region in the photographed image is determined as a target region, and then all pixel points in the target region (including the pixel points corresponding to the marker trajectory) are mapped to a first target image having the same size as the projection image according to the mapping parameter to obtain an aligned image, the aligned image having the same size as the projection image, and then a trajectory corresponding to the marker trajectory in the aligned image is determined as the annotation trajectory.

[0068] In the present application, the marker trajectory in the aligned image can be determined by using a conventional projection control method, for example, when the marker is a laser, a conventional laser point detection method can be used to determine the annotation trajectory corresponding to the marker trajectory in the aligned image.

[0069] Optionally, when the resolution of the photographed image is less than or equal to the resolution of the projection image, the determining the annotation trajectory corresponding to the marker trajectory in the projection image according to the mapping parameter and the photographed image comprises: determining a projection boundary according to the resolution of the projection image, the boundary of the projection image and a preset offset; mapping the projection boundary to a coordinate system in which the photographed image is located to obtain a mapped projection boundary according to the mapping parameter; determining a second target image from a part of the photographed image located in the mapped projection boundary; determining an initial annotation trajectory corresponding to the marker trajectory in the second target image; and mapping the initial annotation trajectory on the projection image according to the mapping parameter to obtain the annotation trajectory.

[0070] The preset offset can be set by the user based on their needs, for example, a preset offset of 5. Specifically, based on the boundary of the projected image, the row pixels are expanded upwards and downwards by the preset offset, and the column pixels are expanded left and right by the preset offset, respectively, to obtain the projection boundary. Using the mapping parameters and the specific pixel coordinates of the projection boundary (the pixel coordinates of the projection boundary in the projector coordinate system), the projection boundary is mapped to the coordinate system of the camera image (that is, the camera coordinate system of the camera corresponding to the captured image). The resulting boundary is the mapped projection boundary. The area bounded by the mapped projection boundary in the captured image is used as the second target image. Then, the trajectory corresponding to the identified motion trajectory is determined in the second target image as the initial motion trajectory.

[0071] After determining the initial motion trajectory corresponding to the identified motion trajectory in the second target image, the initial motion trajectory is then mapped to the coordinate system of the projector using mapping parameters. The resulting trajectory is the labeled trajectory corresponding to the identified motion trajectory in the projected image.

[0072] In this embodiment, the initial motion trajectory of the marker motion trajectory in the second target image can be determined using a conventional projection control method. For example, when the marker is a laser, the initial motion trajectory of the marker motion trajectory in the second target image can be determined using a conventional laser spot detection method.

[0073] S130. Extract the skeleton from the labeled trajectory to obtain the skeleton trajectory.

[0074] S140. Obtain the position information of the midpoint of the skeleton trajectory as the marker position information.

[0075] After obtaining the labeled trajectory, the skeleton is extracted from the labeled trajectory, and the resulting skeleton is used as the skeleton trajectory. When the labeled trajectory is a circular or elliptical trajectory, the centroid can be used as the skeleton trajectory. When the labeled trajectory is a bar (straight line or curved trajectory), the center line of the bar is used as the skeleton trajectory.

[0076] After determining the skeleton trajectory corresponding to the labeled trajectory, the position of the midpoint of the skeleton trajectory is used as the position indicated by the marker, and the position information of the midpoint is used as the position information of the marker indicated by the label.

[0077] Reference Figure 4 , Figure 4 A schematic diagram of the skeleton trajectory in an embodiment of this application is shown. Figure 4 The diagram shows three different labeled trajectories: elliptical trajectories, linear trajectories, and curved trajectories. Figure 4 The area enclosed by the solid line is used as the marked trajectory.

[0078] For the elliptical trajectory, the skeleton trajectory is the center (indicated by the dark dot in Figure 4 the center is the marked position information, and the skeleton trajectory of the linear trajectory and the curved trajectory is the center line (indicated by the dashed line in Figure 4 the center line, and the position information of the midpoint of the center line (indicated by the dark dot in Figure 4 the center line) is the marked position information.

[0079] S150, in response to an instruction event corresponding to the mark.

[0080] S160, based on the instruction event and the marked position information, controlling the projection image to change.

[0081] The instruction event can be an instruction sent to the position indicated by the mark. The instruction event can be a selection, deselection, trajectory drawing, editing, deletion, undo, clearing, recovery, and attribute viewing instruction event.

[0082] The remote controller can be installed with physical buttons or virtual buttons. The buttons can be explicit, always displayed on the projection image, or implicit, displayed or not displayed on the projection image and activated directly through a certain way, such as a laser staying at a certain position for a certain time or a special pen gesture design. The buttons can include multiple buttons, and each button corresponds to an instruction event.

[0083] After obtaining the instruction event, corresponding operations are performed on the position indicated by the mark according to the marked position information of the position indicated by the mark, so that the projection image changes. For example, the instruction event is attribute viewing, and attribute viewing is performed on the position indicated by the mark according to the instruction event and the marked position information.

[0084] In this embodiment, first, a projection image corresponding to a projection image is obtained, the projection image includes a mark motion trajectory corresponding to a mark of the projection image; a mark trajectory corresponding to the mark motion trajectory is determined in the projection image according to a mapping parameter and the projection image, the mapping parameter is used to represent a mapping relationship between the projection image and the projection image, then a skeleton of the mark trajectory is extracted to obtain a skeleton trajectory; position information of a midpoint of the skeleton trajectory is obtained as marked position information; then, in response to an instruction event corresponding to the mark; based on the instruction event and the marked position information, the projection image is controlled to change. The marked position information corresponding to the mark is determined according to the skeleton trajectory corresponding to the mark trajectory, the marked position information can accurately reflect the position indicated by the mark, thereby improving the positioning accuracy of the mark and further improving the interaction accuracy.

[0085] Reference is made to Figure 5, Figure 5 A flowchart of a method for obtaining a mapping parameter in an embodiment of the present application is shown. The method can be used in a projection control device. The method can include the following steps.

[0086] In S210, a first calibration image and a first photographed image corresponding to the first calibration image are obtained. The identification values of the pixel points in the first calibration image are different.

[0087] In this embodiment, the projection surface on which the projection area of the projector is located is a curved surface. The first calibration image can be one of the calibration images described above. The first photographed image is obtained by photographing the projection area by the camera when the projector projects the first calibration image on the projection area. The identification values of the pixel points in the first calibration image are different. The identification values can be gray values or combinations of other numbers and letters. The different pixel points in the first calibration image can be distinguished by the identification values.

[0088] In S220, the mapping parameter is determined according to the coordinates of a first pixel point and the coordinates of a second pixel point having the same identification value as the first pixel point. The first pixel point is a pixel point in the first calibration image, and the second pixel point is a pixel point in the first photographed image.

[0089] The corresponding homography matrix can be solved according to the coordinates of the first pixel point and the coordinates of the second pixel point having the same identification value as the first pixel point, and the homography matrix is taken as the mapping parameter. At least four first pixel points and at least four second pixel points corresponding to the at least four first pixel points can be selected. The corresponding homography matrix can be solved according to the coordinates of the at least four first pixel points and the coordinates of the at least four second pixel points, and the homography matrix is taken as the mapping parameter. The coordinates of the first pixel point can be the coordinates in the coordinate system of the projector, and the coordinates of the second pixel point can be the coordinates in the corresponding camera coordinate system of the camera.

[0090] In this embodiment, when the projection surface on which the projection area of the projector is located is a curved surface, the mapping parameter is determined by the first calibration image having different identification values of the pixel points and the first photographed image corresponding to the first calibration image. The mapping parameter can accurately reflect the mapping relationship between the calibration image and the photographed image. The accuracy of the mapping parameter is high. When the mapping parameter is used for interaction, the accuracy of the interaction is high.

[0091] Please refer to Figure 6 , Figure 6 A flowchart of another method for obtaining a mapping parameter in an embodiment of the present application is shown. The method can be used in a projection control device. The method can include the following steps.

[0092] S310, acquire a second calibration image and a second photographed image corresponding to the second calibration image, the identification value of each pixel point in the second calibration image is different, the second calibration image comprises a plurality of calibration regions, and the second photographed image comprises photographed regions corresponding to the plurality of calibration regions respectively.

[0093] In the embodiment, the projection surface on which the projection region of the projector is located is a curved surface. The second calibration image can be one of the calibration images described above, and the second photographed image is a photographed image obtained by the camera when the camera photographs the photographed region while the projector projects the second calibration image on the projection region. The identification value of each pixel point in the second calibration image is different, and the identification value can be a gray value or a combination of other numerals and letters. The different pixel points in the second calibration image can be distinguished by the identification value.

[0094] The second calibration image is divided into a plurality of regions, each region serving as a calibration region, the second photographed image is divided into a plurality of photographed regions corresponding to the plurality of calibration regions one by one, one calibration region corresponds to one photographed region in the photographed image, and the calibration region and the photographed region correspond to each other in terms of positional relationship. For example, the second calibration image is divided into 3*4 regions, and the second photographed image is also divided into 3*4 regions, wherein the third calibration region in the first row from left in the second calibration image corresponds to the third photographed region in the first row from left in the second photographed image.

[0095] S320, determine a sub-mapping parameter corresponding to each calibration region according to the coordinates of the third pixel point in each calibration region and the coordinates of the fourth pixel point having a matching relationship with the third pixel point in the photographed region, wherein the matching relationship includes that the identification values of the pixel points are the same, and the calibration region in which the third pixel point is located corresponds to the photographed region in which the fourth pixel point is located.

[0096] For one calibration region A, one photographed region B corresponding to the calibration region A is determined, at least four pixel points in the calibration region A are selected as third pixel points, the pixel points having the same identification value as each third pixel point in the photographed region B corresponding to the calibration region A are obtained as fourth pixel points, and one third pixel point corresponds to one fourth pixel point. Then, the homographic matrix corresponding to the calibration region A is solved according to the coordinates of the at least four third pixel points and the coordinates of the at least four fourth pixel points, as the sub-mapping parameter corresponding to the calibration region A. The homographic matrix corresponding to each calibration region is determined by traversing each calibration region, as the sub-mapping parameter corresponding to each calibration region, and finally a plurality of sub-mapping parameters corresponding to the plurality of calibration regions are obtained. The coordinates of the third pixel point can be the coordinates in the coordinate system of the projector, and the coordinates of the fourth pixel point can be the coordinates in the corresponding camera coordinate system of the camera.

[0097] S330, performing a merging operation on the plurality of sub-mapping parameters to obtain the mapping parameter.

[0098] The plurality of sub-mapping parameters corresponding to the plurality of calibration regions are obtained, and a merging operation is performed on the plurality of sub-mapping parameters to obtain the mapping parameter.

[0099] In this embodiment, the second calibration image is divided into a plurality of calibration regions, and a sub-mapping parameter corresponding to each calibration region is determined. The range corresponding to each calibration region is small, so that each sub-mapping parameter can accurately reflect the mapping relationship between the pixel points in the calibration region and the pixel points in the mapping region. The mapping parameter obtained by performing a merging operation on the plurality of sub-mapping parameters has high accuracy, and the interaction accuracy is high when the mapping parameter is used for interaction.

[0100] Referring to Figure 7 , Figure 7 A flowchart of another method for obtaining a mapping parameter in an embodiment of the application is shown. The method can be used in a projection control device. The method can include:

[0101] S410, obtaining a third calibration image and a third photographed image corresponding to the third calibration image, the third calibration image including a target pattern.

[0102] When the projection surface on which the projection region of the projector is located is a curved surface, the third calibration image can be one of the calibration images described above, and the third photographed image is a photographed image obtained by the camera when the camera photographs the projection region after the projector projects the third calibration image on the projection region. The target pattern can be a sine stripe or a Gray code coded light pattern.

[0103] S420, performing phase main value extraction on the third calibration image to obtain a first extraction result, and performing Gray code decoding on the third calibration image to obtain a first decoding result.

[0104] S430, determining a product of the first decoding result and a target value as a first product, and adding the first product to the first extraction result to obtain a second decoding result, the identification values of the pixel points in the second decoding result being different.

[0105] First, the phase main value extraction is performed on the third calibration image to obtain an extracted result as a first extraction result, and the Gray code decoding is performed on the third calibration image to obtain a decoded result as a first decoding result. Then, a product of the first decoding result and a target value is determined as a first product, and the first product is added to the first extraction result to obtain a second decoding result, the identification values of the pixel points in the second decoding result being different. The target value can be 2π.

[0106] Referring to Figure 8 ,Figure 8 The obtaining process of the second decoding result in the embodiment of the application is shown. The target pattern is a vertical sine stripe (four-step phase shift pattern, the phase of 11 is 0, the phase of 12 is 90, the phase of 13 is 180, and the phase of 14 is 270), the phase principal value of the vertical sine stripe is extracted to obtain a first extraction result, and the vertical sine stripe is subjected to Golay code decoding to obtain a first decoding result after decoding. The first decoding result is multiplied by 2π to obtain a first product. The first product is added to the first extraction result to obtain the second decoding result. In the second decoding result, the gray values of the pixel points increase from left to right in sequence.

[0107] However, Figure 8 The gray values of the pixel points in the shown second decoding result are not completely different. The gray values of the pixel points in the same column are the same. Therefore, the gray value is not accurate enough as an identification value. A target pattern is a horizontal sine stripe (four-step phase shift pattern) calibration image. The horizontal sine stripe is processed according to the processing process of the vertical sine stripe to obtain a corresponding second decoding result. At this time, the gray values of the pixel points in the second decoding result increase from top to bottom in sequence.

[0108] The second decoding result corresponding to the vertical sine stripe is superimposed on the second decoding result corresponding to the horizontal sine stripe to obtain a combination of the gray values of the pixel points. At this time, the identification value can be the combination of the gray values of the pixel points. For example, the gray value combination of a pixel point C can be (c1, c2), where c1 can refer to the gray value of the pixel point C when the target pattern is a vertical sine stripe, and c2 can refer to the gray value of the pixel point C when the target pattern is a horizontal sine stripe.

[0109] S440, phase principal value extraction is performed on the third photographed image to obtain a second extraction result, and Golay code decoding is performed on the third photographed image to obtain a third decoding result.

[0110] S450, the product of the third decoding result and the target value is determined as a second product, and the second product is added to the second extraction result to obtain a fourth decoding result.

[0111] The processing process of the third photographed image is similar to the processing process of the third calibration image described above, and will not be described again.

[0112] S460, the mapping parameter is determined according to the coordinates of the fifth pixel point and the coordinates of the sixth pixel point having the same identification value as the fifth pixel point, wherein the fifth pixel point is a pixel point in the second decoding result, and the sixth pixel point is a pixel point in the fourth decoding result.

[0113] The homography matrix corresponding to the fifth pixel point and the sixth pixel point with the same identification value of the fifth pixel point can be solved as the mapping parameter. At least four fifth pixel points and at least four corresponding sixth pixel points can be selected, and the homography matrix corresponding to the at least four fifth pixel points and the at least four sixth pixel points can be solved as the mapping parameter. The coordinates of the fifth pixel point can be the coordinates in the projector coordinate system, and the coordinates of the sixth pixel point can be the coordinates in the camera coordinate system corresponding to the camera.

[0114] In some embodiments, the second decoding result can be divided into a plurality of calibration decoding regions, the fourth decoding result can be divided into a plurality of shooting decoding regions corresponding to the plurality of calibration decoding regions one by one, the sub-mapping parameter corresponding to each calibration decoding region can be obtained according to the coordinates of the seventh pixel point in each calibration decoding region and the coordinates of the eighth pixel point having a matching relationship with the seventh pixel point in the shooting decoding region, and the mapping parameter can be obtained by performing a merging operation on the sub-mapping parameters corresponding to the plurality of calibration decoding regions one by one. The matching relationship includes that the identification values of the pixel points are the same, and the calibration decoding region where the seventh pixel point is located corresponds to the shooting decoding region where the eighth pixel point is located.

[0115] In this embodiment, the mapping parameter with high accuracy can be solved through the third calibration image of the target pattern, so that the interaction accuracy is high when the mapping parameter is used for interaction.

[0116] Reference Figure 9 , Figure 9 A flowchart of another method for obtaining a mapping parameter in an embodiment of the application is shown, which can be used in a projection control device. The method can include:

[0117] S510, obtaining a fourth calibration image and a fourth shooting image corresponding to the fourth calibration image, the fourth calibration image including a projection feature point, and the fourth shooting image including a shooting feature point corresponding to the projection feature point.

[0118] In this embodiment, when the projection area of the projector corresponds to a planar projection surface, the projector projects a calibration template on the projection area to obtain a fourth calibration image, and the camera shoots the fourth calibration image projected on the projection area to obtain a fourth shooting image. The fourth calibration image is provided with a plurality of projection feature points, which can be one of a circle center and an angle point. As shown in Figure 10a , the projection feature point is an angle point of the edge part of the frame (the point surrounded by a small circle in Figure 10a ), and as shown in Figure 10b , the projection feature point is the center of the edge circle (the point surrounded by a small circle in Figure 10b ).

[0119] In the implementation of the present application, the selection rules of the projection feature points can be set according to the accuracy requirement, for example Figure 10a and Figure 10b As shown in the fourth calibration image, four points can be selected as the projection feature points every certain number of rows or columns.

[0120] The projection feature points can be determined in the fourth calibration image through the traditional OpenCV algorithm library, and the shooting feature points can be determined in the fourth shooting image corresponding to the fourth calibration image.

[0121] The effective area to which the mapping parameter is applicable can be the area covered by the projection feature points, for example Figure 10a and Figure 10b In the four projection feature points, the area covered by the rectangle formed by the four projection feature points in sequence is the effective area of the mapping parameter. If the mark is in other areas outside the effective area of the mapping parameter, the accuracy of the determined position information of the mark in the projection image can be slightly low, and if the mark is in the effective area of the mapping parameter, the accuracy of the determined position information of the mark in the projection image is higher. Therefore, in order to improve the accuracy and scope of application of the mapping parameter, the projection feature points can be selected to be closer to the edge of the fourth calibration image.

[0122] As shown in Figure 10a , the four points circled by the solid small circles in the fourth calibration image are the projection feature points, as shown in Figure 11 , the four points circled by the solid small circles in the fourth shooting image are the corresponding shooting feature points, and in Figure 10a and Figure 11 , the shooting feature points correspond to the projection feature points one by one: the shooting feature point at the top left corner corresponds to the projection feature point at the top left corner, the shooting feature point at the top right corner corresponds to the projection feature point at the top right corner, the shooting feature point at the bottom right corner corresponds to the projection feature point at the bottom right corner, and the shooting feature point at the bottom left corner corresponds to the projection feature point at the bottom left corner.

[0123] S520, according to the coordinates of the projection feature points and the coordinates of the shooting feature points, determine the mapping parameter.

[0124] In the present application, the mapping parameter can be calculated by perspective transformation or homography matrix algorithm according to the pixel coordinates of the shooting feature points and their corresponding projection feature points. The mapping parameter can be a homography matrix, which is obtained according to at least four projection feature points and four corresponding shooting feature points, for example, according to Figure 10a and Figure 11 The four projection feature points and four shooting feature points involved can obtain the mapping parameter. In some embodiments, more projection feature points and corresponding shooting feature points can be set to obtain a homography matrix with higher accuracy by a larger number of projection feature points and corresponding shooting feature points.

[0125] The pixel coordinates of the photographed feature points in the photographed image can refer to the coordinates of the photographed feature points in a camera coordinate system with the camera as a reference.

[0126] In the embodiment, for the projection surface of the plane, the mapping parameter with a higher accuracy is obtained according to the coordinates of the projected feature points in the fourth calibration image and the coordinates of the photographed feature points in the fourth photographed image, the mapping parameter can accurately reflect the mapping relationship between the fourth calibration image and the fourth photographed image, and thus accurate conversion between the projection image and the photographed image is realized.

[0127] Reference Figure 12 , Figure 12 A block diagram of a projection control device according to an embodiment of the present application is shown, the device 1100 includes:

[0128] An image acquisition module 1110 is configured to acquire a photographed image corresponding to a projection image, the photographed image including an identification trajectory corresponding to an identification of the projection image;

[0129] A determination module 1120 is configured to determine an annotation trajectory corresponding to the identification trajectory in the projection image according to a mapping parameter and the photographed image, the mapping parameter being used to represent a mapping relationship between the projection image and the photographed image;

[0130] An extraction module 1130 is configured to perform skeleton extraction on the annotation trajectory to obtain a skeleton trajectory;

[0131] A position acquisition module 1140 is configured to acquire position information of a midpoint of the skeleton trajectory as marking position information;

[0132] A response module 1150 is configured to respond to an instruction event corresponding to the identification;

[0133] A control module 1160 is configured to control the projection image to change based on the instruction event and the marking position information.

[0134] Optionally, when the projection surface is a curved surface, the device 1100 further includes:

[0135] A parameter acquisition module is configured to acquire a first calibration image and a first photographed image corresponding to the first calibration image, identification values of all pixel points in the first calibration image being different; and determine the mapping parameter according to coordinates of a first pixel point and coordinates of a second pixel point with the same identification value as the first pixel point, wherein the first pixel point is a pixel point in the first calibration image, and the second pixel point is a pixel point in the first photographed image.

[0136] Optionally, when the projection surface is a curved surface, the apparatus 1100 further comprises:

[0137] The parameter obtaining module is configured to: acquire a second calibration image and a second captured image corresponding to the second calibration image, the second calibration image comprising different identification values of each pixel point, the second calibration image comprising a plurality of calibration regions, and the second captured image comprising a plurality of captured regions corresponding to the plurality of calibration regions respectively; determine a sub-mapping parameter corresponding to each of the calibration regions according to a coordinate of a third pixel point in each of the calibration regions and a coordinate of a fourth pixel point having a matching relationship with the third pixel point in the captured region, wherein the matching relationship comprises that the identification values of the pixel points are the same, and the calibration region in which the third pixel point is located corresponds to the captured region in which the fourth pixel point is located; and perform a merging operation on the plurality of sub-mapping parameters to obtain the mapping parameter.

[0138] Optionally, when the projection surface is a curved surface, the apparatus 1100 further comprises:

[0139] The parameter obtaining module is configured to: acquire a third calibration image and a third captured image corresponding to the third calibration image, the third calibration image comprising a target pattern; perform phase residence value extraction on the third calibration image to obtain a first extraction result, and perform Golay code decoding on the third calibration image to obtain a first decoding result; determine a product of the first decoding result and a target value as a first product, and add the first product and the first extraction result to obtain a second decoding result, the second decoding result comprising different identification values of each pixel point; perform phase residence value extraction on the third captured image to obtain a second extraction result, and perform Golay code decoding on the third captured image to obtain a third decoding result; determine a product of the third decoding result and the target value as a second product, and add the second product and the second extraction result to obtain a fourth decoding result; and determine the mapping parameter according to a coordinate of a fifth pixel point and a coordinate of a sixth pixel point having the same identification value as the fifth pixel point, wherein the fifth pixel point is a pixel point in the second decoding result, and the sixth pixel point is a pixel point in the fourth decoding result.

[0140] Optionally, when the projection surface is a flat surface, the apparatus 1100 further comprises:

[0141] The parameter obtaining module is configured to: acquire a fourth calibration image and a fourth captured image corresponding to the fourth calibration image, the fourth calibration image comprising a projection feature point, and the fourth captured image comprising a captured feature point corresponding to the projection feature point; and determine the mapping parameter according to a coordinate of the projection feature point and a coordinate of the captured feature point.

[0142] Optionally, when the resolution of the photographed image is greater than the resolution of the projected image, the determining module 1120 is further configured to determine a region corresponding to the projected image in the photographed image as a target region; map the pixel points in the target region to a first target image having the same size as the projected image according to the mapping parameter to obtain an aligned image; and determine a labeled trajectory corresponding to the identified motion trajectory on the aligned image.

[0143] Optionally, when the resolution of the photographed image is less than or equal to the resolution of the projected image, the determining module 1120 is further configured to determine a projected boundary according to the resolution of the projected image, the boundary of the projected image, and a preset offset; map the projected boundary to a coordinate system in which the camera image is located to obtain a mapped projected boundary through the mapping parameter; determine a second target image in the photographed image located within the mapped projected boundary; determine an initial labeled trajectory corresponding to the identified motion trajectory in the second target image; and map the initial labeled trajectory on the projected image through the mapping parameter to obtain the labeled trajectory.

[0144] Those skilled in the art can clearly understand the specific working process of the described device and module for the convenience and brevity of the description, and can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0145] In several embodiments provided in the present application, the coupling or direct coupling or communication connection between the modules displayed or discussed can be indirect coupling or communication connection between the interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0146] In addition, each functional module in each embodiment of the present application can be integrated in one processing module, or each module can exist physically independently, or two or more modules can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software functional module.

[0147] Reference Figure 13 , Figure 13 A block diagram of a projection control device according to yet another embodiment of the present application is shown. The projection control device 1200 can include one or more of the following components: a processor 1210, a memory 1220, and one or more application programs, wherein the one or more application programs can be stored in the memory 1220 and configured to be executed by the one or more processors 1210, and the one or more programs are configured to perform the methods described in the foregoing method embodiments.

[0148] The processor 1210 can include one or more processing cores. The processor 1210 connects various parts within the projection control device 1200 by running or executing instructions, programs, code sets or instruction sets stored in the memory 1220, and calling data stored in the memory 1220, to perform various functions and process data of the projection control device 1200. Alternatively, the processor 1210 can be implemented in at least one of a hardware form of a digital signal processing (DSP), a field-programmable gate array (FPGA), a programmable logic array (PLA). The processor 110 can be integrated with a combination of one or more of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU is mainly used to process operating systems, user interfaces, and application programs; the GPU is used to render and draw display content; and the modem is used to process wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor 1210, but can be realized by a separate communication chip.

[0149] The memory 1220 can include a random access memory (RAM) and can also include a read-only memory (ROM). The memory 1220 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 1220 can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing each of the methods described below, etc. The data storage area can also store data created by the terminal 1200 in use (such as a phone book, audio and video data, chat record data, etc.).

[0150] Please refer to Figure 14 , Figure 14 A structural block diagram of a computer readable storage medium of an embodiment of the present application is shown. The computer readable medium 800 stores program codes therein, which can be called and executed by a processor to perform the methods described in the above method embodiments.

[0151] The computer-readable storage medium 800 can be an electronic storage memory such as a flash memory, an EEPROM (Electrically Erasable Programmable Read-Only Memory), an EPROM, a hard disk or a ROM. Optionally, the computer-readable storage medium 800 comprises a non-transitory computer-readable medium. The computer-readable storage medium 800 has a storage space for program codes 810 to execute any of the method steps described above. These program codes can be read from or written to one or more computer program products. The program codes 810 can be compressed, for example, in an appropriate form.

[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them. Although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art will understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent replacements for some of the technical features, without the essence of the corresponding technical solutions deviating from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A projection control method, characterized in that, The method includes: Acquire a captured image corresponding to the projected image, the captured image including the motion trajectory of the marker corresponding to the marker marked on the projected image; Based on the mapping parameters and the captured image, a labeled trajectory corresponding to the marked motion trajectory is determined in the projected image. The mapping parameters are used to characterize the mapping relationship between the projected image and the captured image. The skeleton of the labeled trajectory is extracted to obtain the skeleton trajectory; The position information of the midpoint of the skeleton trajectory is obtained as the marker position information; In response to an instruction event corresponding to the identified identifier; Based on the instruction event and the marker position information, the projected image is controlled to change.

2. The method as described in claim 1, characterized in that, When the projection surface is a curved surface, the method for obtaining the mapping parameters includes: Acquire a first calibration image and a first captured image corresponding to the first calibration image, wherein the identifier values ​​of each pixel in the first calibration image are different; The mapping parameters are determined based on the coordinates of the first pixel and the coordinates of the second pixel that has the same identifier value as the first pixel, wherein the first pixel is a pixel in the first calibration image and the second pixel is a pixel in the first captured image.

3. The method as described in claim 1, characterized in that, When the projection surface is a curved surface, the method for obtaining the mapping parameters includes: A second calibration image and a second captured image corresponding to the second calibration image are obtained. The identification values ​​of each pixel in the second calibration image are different. The second calibration image includes multiple calibration regions. The second captured image includes a captured region corresponding to each of the multiple calibration regions. Based on the coordinates of the third pixel in each calibration region and the coordinates of the fourth pixel in the shooting region that has a matching relationship with the third pixel, the sub-mapping parameters corresponding to each calibration region are determined. The matching relationship includes that the pixel identification values ​​are the same and that the calibration region where the third pixel is located corresponds to the shooting region where the fourth pixel is located. The mapping parameters are obtained by merging multiple sub-mapping parameters.

4. The method as described in claim 1, characterized in that, When the projection surface is a curved surface, the method for obtaining the mapping parameters includes: Acquire a third calibration image and a third captured image corresponding to the third calibration image, wherein the third calibration image includes a target pattern; Phase principal value extraction is performed on the third calibration image to obtain a first extraction result, and Gray code decoding is performed on the third calibration image to obtain a first decoding result; The product of the first decoding result and the target value is determined as the first product, and the first product is added to the first extraction result to obtain the second decoding result, wherein the identifier values ​​of each pixel in the second decoding result are different; Phase principal value extraction is performed on the third captured image to obtain a second extraction result, and Gray code decoding is performed on the third captured image to obtain a third decoding result; The product of the third decoding result and the target value is determined as the second product, and the second product is added to the second extraction result to obtain the fourth decoding result; the target value is 2π. The mapping parameters are determined based on the coordinates of the fifth pixel and the coordinates of the sixth pixel, which has the same identifier value as the fifth pixel, wherein the fifth pixel is a pixel in the second decoding result and the sixth pixel is a pixel in the fourth decoding result.

5. The method as described in claim 1, characterized in that, When the projection plane is a plane, the method for obtaining the mapping parameters includes: Acquire a fourth calibration image and a fourth captured image corresponding to the fourth calibration image, wherein the fourth calibration image includes projected feature points and the fourth captured image includes captured feature points corresponding to the projected feature points; The mapping parameters are determined based on the coordinates of the projected feature points and the coordinates of the captured feature points.

6. The method as described in claim 1, characterized in that, When the resolution of the captured image is greater than the resolution of the projected image, determining the annotation trajectory corresponding to the identified motion trajectory in the projected image based on the mapping parameters and the captured image includes: The region in the captured image that corresponds to the projected image is determined as the target region. According to the mapping parameters, the pixels in the target region are mapped onto a first target image with the same size as the projected image to obtain an aligned image; Determine the labeled trajectory on the aligned image corresponding to the identified motion trajectory.

7. The method as described in claim 1, characterized in that, When the resolution of the captured image is less than or equal to the resolution of the projected image, determining the annotation trajectory corresponding to the identified motion trajectory in the projected image based on the mapping parameters and the captured image includes: The projection boundary is determined based on the resolution of the projected image, the boundary of the projected image, and a preset offset. The mapped projection boundary is obtained by mapping the projection boundary to the coordinate system of the captured image using the mapping parameters. The portion of the captured image located within the mapping projection boundary is identified as the second target image; In the second target image, determine the initial labeled trajectory corresponding to the identified motion trajectory; The initial labeled trajectory is mapped onto the projected image using the mapping parameters to obtain the labeled trajectory.

8. A projection control device, characterized in that, The device includes: The image acquisition module is used to acquire a captured image corresponding to the projected image, wherein the captured image includes the motion trajectory of the marker corresponding to the marker marked on the projected image; The determining module is used to determine the labeled trajectory corresponding to the identified motion trajectory in the projected image based on the mapping parameters and the captured image; the mapping parameters are used to characterize the mapping relationship between the projected image and the captured image. The extraction module is used to extract the skeleton of the labeled trajectory to obtain the skeleton trajectory; The position acquisition module is used to acquire the position information of the midpoint of the skeleton trajectory as the marker position information; A response module, used to respond to an instruction event corresponding to the identifier; The control module is used to control the projected image to change based on the instruction event and the marker position information.

9. A projection control device, characterized in that, include: One or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs being configured to perform the method as described in any one of claims 1-7.

10. A projection control system, characterized in that, The projection control system includes at least a camera, a projector, a remote controller, and a projection control device, wherein... The projector is used to project images; The remote control is used to acquire the captured image corresponding to the projected image; The remote control is used to transmit an identifier; The projection control device is used to perform the method as described in any one of claims 1-7.

Images