Control site determination method and device, electronic equipment and storage medium

Abstract

The invention discloses a control site determination method and device, electronic equipment and a storage medium, and the method is based on a target image of a display screen, a calibration light source on the display screen and an inertial measurement unit in remote control equipment. Determining first attitude information of the shooting device in combination with a conversion relationship between a camera coordinate system of the shooting device and an inertial measurement unit coordinate system of the inertial measurement unit, the first attitude information representing a conversion relationship between the camera coordinate system and a world coordinate system; and determining a control site of the remote control equipment in the display screen based on the first attitude information and a plane where the display screen is located under the world coordinate system. Through the conversion relation between the camera coordinate system and the inertial measurement unit coordinate system and the conversion relation between the camera coordinate system and the world coordinate system, the information in the target image is converted so as to determine the control site of the remote control equipment in the display screen, the UWB technology is avoided, and the accuracy of determining the control site by the remote control equipment is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote control device control, in particular to a control point determination method and device, an electronic device and a storage medium. BACKGROUND

[0002] The most widely used remote control device at present is a UWB remote control device. The UWB remote control device determines the control point of the remote control device by using UWB (Ultra-Wide band) technology, and is used for marking, indicating or controlling content on a corresponding screen, interacting with the screen or annotating the screen.

[0003] However, the environment used by the remote control device is complex, and the remote control device using the UWB technology is affected by factors such as multipath effect, channel attenuation and electromagnetic interference, thereby reducing the accuracy of the determination of the control point of the remote control device.

[0004] Therefore, how to improve the accuracy of the determination of the control point of the remote control device is a problem to be solved. SUMMARY

[0005] The embodiments of the present application provide a control point determination method and device, an electronic device and a storage medium, which can improve the accuracy of the determination of the control point of the remote control device.

[0006] In a first aspect, the embodiments of the present application provide a control point determination method, which is applied to a remote control device. The remote control device is used to emit laser to a display device to display a control point for indication on a display screen of the display device. The remote control device comprises a shooting device and an inertial measurement unit. The conversion relationship between the camera coordinate system of the shooting device and the inertial measurement unit coordinate system of the inertial measurement unit is calibrated in advance. The method comprises the following steps:

[0007] Obtaining a target image collected by the shooting device on the display screen; the display screen comprises a plurality of preset calibration light sources;

[0008] Extracting a plurality of target feature regions containing the calibration light sources from the target image;

[0009] When the number of the extracted target feature regions is greater than or equal to the number of the calibration light sources, determining first attitude information of the shooting device according to first position information of the target feature regions, second position information of the calibration light sources and first motion state information of the inertial measurement unit in the remote control device;

[0010] determine, based on the first attitude information of the photographing apparatus and a plane in which the display screen is located in a world coordinate system determined based on the calibration light source, a control point of a laser point displayed in the display screen by the laser emitted by the remote control device to the display device.

[0011] The first position information is position information of the target feature region in a camera coordinate system corresponding to the target image; the second position information is position information of the calibration light source in the world coordinate system; the first motion state information includes motion information and attitude information of the inertial measurement unit in an inertial measurement unit coordinate system; and the first attitude information represents a conversion relationship between a camera coordinate system of the photographing apparatus and the world coordinate system.

[0012] In a second aspect, an embodiment of the present application provides a remote control device positioning apparatus, including:

[0013] The acquisition module is configured to acquire a target image collected by the photographing apparatus on the display screen; and the display screen includes a plurality of preset calibration light sources.

[0014] The extraction module is configured to extract a plurality of target feature regions including the calibration light sources from the target image.

[0015] The first determination module is configured to, when the number of the target feature regions extracted is greater than or equal to the number of the calibration light sources, determine first attitude information of the photographing apparatus according to first position information of the target feature regions, second position information of the calibration light sources, and first motion state information of an inertial measurement unit in the remote control device.

[0016] The second determination module is configured to determine, based on the first attitude information of the photographing apparatus and a plane in which the display screen is located in a world coordinate system determined based on the calibration light source, a control point of a laser point displayed in the display screen by the laser emitted by the remote control device to the display device.

[0017] In a third aspect, an embodiment of the present application further provides an electronic device, which is configured with a processor and a memory; the memory stores a plurality of computer programs; and the processor loads the computer programs from the memory to execute steps of any one of the control point determination methods provided in the embodiments of the present application.

[0018] In a fourth aspect, an embodiment of the present application further provides a computer readable storage medium, which stores a plurality of computer programs; and the computer programs are suitable for being loaded by a processor to execute steps of any one of the control point determination methods provided in the embodiments of the present application.

[0019] In a fifth aspect, the embodiments of the present application further provide a computer program product, comprising a computer program or a computer program, which, when executed by a processor, implements the steps of any of the control point determination methods provided by the embodiments of the present application.

[0020] By using the scheme of the embodiments of the present application, the target image collected by the shooting device on the display screen and the plurality of target feature regions containing the calibration light source in the target image are acquired in a visual positioning manner. The first position information of the target feature region, the second position information of the calibration light source, and the first motion state information of the inertial measurement unit in the remote control device are combined to determine the first attitude information of the shooting device based on the conversion relationship between the camera coordinate system of the shooting device and the inertial measurement unit coordinate system of the inertial measurement unit. The first attitude information represents the conversion relationship between the camera coordinate system of the shooting device and the world coordinate system. Then, the control point of the remote control device in the display screen is determined based on the first attitude information of the shooting device and the plane in which the display screen is located in the world coordinate system determined based on the calibration light source. The information in the target image collected by the shooting device on the display screen is converted to determine the control point of the remote control device in the display screen based on the conversion relationship between the camera coordinate system of the shooting device and the inertial measurement unit coordinate system of the inertial measurement unit, and the conversion relationship between the camera coordinate system of the shooting device and the world coordinate system. This avoids the influence of factors such as multipath effect, channel attenuation, and electromagnetic interference when the control point is determined using UWB technology, thereby improving the accuracy of the control point determined by the remote control device. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0022] Figure 1 is a first embodiment flowchart of the control point determination method provided by the present application;

[0023] Figure 2 is a calibration ruler diagram of the control point determination method provided by the present application;

[0024] Figure 3 is a relationship diagram of the display screen and the calibration light source of the control point determination method provided by the present application;

[0025] Figure 4 is a feature region detection diagram of the control point determination method provided by the present application;

[0026] Figure 5is a schematic diagram of feature region sorting of a control point determination method provided in the present application;

[0027] Figure 6 is a schematic diagram of display screen calibration light source sorting of a control point determination method provided in the present application;

[0028] Figure 7 is a schematic diagram of determining a control point of a control point determination method provided in the present application;

[0029] Figure 8 is a schematic diagram of a second embodiment flow of a control point determination method provided in the present application;

[0030] Figure 9 is a schematic diagram of another determining a control point of a control point determination method provided in the present application;

[0031] Figure 10 is a schematic diagram of a third embodiment flow of a control point determination method provided in the present application;

[0032] Figure 11 is a structural schematic diagram of a remote control device positioning apparatus provided in an embodiment of the present application;

[0033] Figure 12 is a structural schematic diagram of a remote control device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person skilled in the art without creative work fall within the scope of protection of the present application. Meanwhile, in the description of the embodiments of the present application, the terms “first”, “second”, etc. are only used for distinguishing description, and cannot be understood as indicating or implying relative importance. Therefore, the features with “first” and “second” can explicitly or implicitly include one or more features. In the description of the embodiments of the present application, the meaning of “multiple” is two or more, unless otherwise specifically limited.

[0035] The most widely used remote control device at present is a UWB remote control device. The UWB remote control device realizes positioning of the remote control device by using UWB (Ultra-Wide band) technology. However, the environment used by the remote control device is complex, and the remote control device using the UWB technology is affected by factors such as multipath effect, channel attenuation, and electromagnetic interference, thereby reducing the accuracy of determining the control point of the remote control device.

[0036] To solve the above problems, the embodiment provides a control point determination method and device, electronic equipment and a storage medium. The control point determination method comprises the following steps: acquiring a target image collected by a shooting device on a display screen and a plurality of target feature regions containing a calibration light source in the target image by using a visual positioning method; determining first attitude information of the shooting device based on first position information of the target feature regions, second position information of the calibration light source, first motion state information of an inertial measurement unit in a remote control device, and a conversion relationship between a camera coordinate system of the shooting device and an inertial measurement unit coordinate system of the inertial measurement unit, the first attitude information representing a conversion relationship between the camera coordinate system of the shooting device and a world coordinate system; and determining a control point of the remote control device in the display screen based on the first attitude information of the shooting device and a plane in which the display screen is located in the world coordinate system determined based on the calibration light source. The information in the target image collected by the shooting device on the display screen is converted to determine the control point of the remote control device in the display screen based on the conversion relationship between the camera coordinate system of the shooting device and the inertial measurement unit coordinate system of the inertial measurement unit, and the conversion relationship between the camera coordinate system of the shooting device and the world coordinate system, so that the control point determined by the remote control device is not affected by factors such as multipath effect, channel attenuation and electromagnetic interference, and the accuracy of the control point determined by the remote control device is improved.

[0037] The embodiments of the present application will be described in detail below with reference to the accompanying drawings. In the embodiments of the present application, the execution subject is taken as an example of a remote control device, which can be a laser pointer or the like. It should be noted that the sequence of the following embodiments is not limited to the preferred sequence of the embodiments. Although a logical sequence is shown in the flowchart, in some cases, the steps shown or described can be performed in a sequence different from that shown in the accompanying drawings.

[0038] Please refer to Figure 1 The first embodiment of the control point determination method specifically comprises the following steps:

[0039] It should be noted that the remote control device is used to emit laser light to the display device to display a control point for indication on the display screen of the display device. The remote control device comprises a shooting device and an inertial measurement unit. The conversion relationship between the camera coordinate system of the shooting device and the inertial measurement unit coordinate system of the inertial measurement unit is calibrated in advance.

[0040] In step 101, a target image collected by the shooting device on the display screen is acquired. The display screen contains a plurality of preset calibration light sources.

[0041] In this step, the display screen contains a plurality of pre-assembled calibration light sources. The remote control device shoots the display screen by the shooting device, and then acquires the target image collected by the shooting device on the display screen.

[0042] It should be noted that the camera parameters and the distortion parameters of the photographing device have been calibrated before the remote control device captures the target image of the display screen by the photographing device. The remote control device corrects the real-time image according to the calibrated camera parameters and the distortion parameters of the photographing device when capturing the image, so as to obtain the target image.

[0043] The process of calibrating the camera parameters of the photographing device is as follows: first, prepare a calibration ruler, such as Figure 2 As shown in the figure, the calibration ruler is a rigid tool, and the material surface is a light-absorbing material that does not reflect infrared light. The white dots represent infrared lamps. The calibration ruler is quickly moved in the field of view of the photographing device, and the photographing device captures the image of the calibration ruler quickly moving in the field of view of the photographing device. According to these images, the infrared lamp is tracked and matched, and the camera parameters K and the distortion parameters D of the camera are output.

[0044] Exemplarily, the display screen contains four preset calibration light sources, which can be installed on any edge of the display screen or any fixed position on the display screen, and at least one calibration light source is placed on each edge of the display screen. As shown in Figure 3 Figure 3 The four exemplary assembly diagrams of the calibration light sources on the display screen are shown in Figure 3 The four exemplary assembly diagrams of the calibration light sources on the display screen are shown in

[0045] Step 102, extracting a plurality of target feature regions containing the calibration light sources from the target image.

[0046] In this step, after the remote control device obtains the target image of the display screen, it extracts a plurality of target feature regions containing the calibration light sources from the target image. It should be noted that the target image will include a plurality of feature regions, and the plurality of feature regions will include feature regions corresponding to the calibration light sources and feature regions formed by interference. The remote control device needs to screen a plurality of target feature regions from the plurality of feature regions.

[0047] Specifically, the step of extracting a plurality of target feature regions containing the calibration light sources from the target image includes:

[0048] Step 1021, performing feature region detection on the target image to determine the shape information and size information of each feature region in the target image.

[0049] At step 1022, the aspect ratio of each feature region is determined according to the aspect information of each feature region.

[0050] At step 1023, the pixel number of each feature region is determined according to the size information of each feature region.

[0051] At step 1024, the feature region with an aspect ratio less than a preset ratio and a pixel number greater than a preset number is determined as a target feature region containing the calibration light source in the target image.

[0052] At step 1025, a plurality of target feature regions containing the calibration light source are extracted from the target image.

[0053] In steps 1021 to 1025, after the remote control device captures the target image through the photographing device, the aspect information and the size information of each feature region in the target image are determined through feature region detection. Specifically, as shown in FIG. 10, the target image is divided into a plurality of feature regions, and the aspect information and the size information of each feature region are determined. Figure 4 Figure 4 As shown in FIG. 11, an enlarged pixel representation of a feature region in the target image is a square representing one pixel, the pixels are traversed from the top-left corner of the feature region, and whether the pixels are visited is recorded. When the gray value {gray>white_enough_threshold, white_enough_threshold≥100} appears, the four adjacent pixels not visited are found to be judged, and a quadtree is formed. If all the pixels satisfy the above gray condition, they are put into a container recording the current feature region. When the visited quadtree has no adjacent gray value satisfying the threshold white_enough_threshold, the detection of the feature region is completed, and the aspect information and the size information of the feature region are determined. After the detection of all the feature regions is completed, the aspect information and the size information of each feature region are obtained.

[0054] The remote control device determines the aspect ratio of each feature region according to the aspect information of each feature region, determines the pixel number of each feature region according to the size information of each feature region, and determines the feature region with an aspect ratio less than a preset ratio and a pixel number greater than a preset number as a target feature region. Exemplarily, the feature region with an aspect ratio greater than 2 is removed, the feature region with a pixel number less than 20 pixels is removed, and the remaining feature region is the target feature region.

[0055] ​Step 103, when the number of the target feature regions extracted is greater than or equal to the number of the calibration light sources, the first attitude information of the photographing device is determined according to the first position information of the target feature regions, the second position information of the calibration light sources and the first motion state information of the inertial measurement unit in the remote control device.

[0056] In this step, after the remote control device extracts the target feature regions containing the calibration light sources from the target image, the number of the target feature regions is compared with the number of the calibration light sources on the display screen. When the number of the target feature regions extracted is greater than or equal to the number of the calibration light sources, the first attitude information of the photographing device is determined according to the first position information of the target feature regions, the second position information of the calibration light sources and the first motion state information of the inertial measurement unit in the remote control device.

[0057] Specifically, since the conversion relationship between the coordinate system of the display screen and the world coordinate system is determined in advance and stored in the remote control device, the remote control device converts the position information of the calibration light sources under the display screen into the position information of the calibration light sources under the world coordinate system according to the conversion relationship between the coordinate system of the display screen and the world coordinate system, to obtain the second position information of the calibration light sources. The conversion relationship between the image coordinate system of the image collected by the photographing device in the remote control device and the camera coordinate system of the photographing device is set in advance. The remote control device determines the position information of the target feature regions under the camera coordinate system according to the position information of the target feature regions on the target image, in combination with the conversion relationship between the image coordinate system of the image collected by the photographing device and the camera coordinate system of the photographing device, to obtain the first position information of the target feature regions. The first motion state information of the inertial measurement unit can be obtained by the photographing device when collecting the target image. The first motion state information includes the motion information and the attitude information of the inertial measurement unit under the inertial measurement unit coordinate system. The remote control device determines the first attitude information of the photographing device according to the first position information of the target feature regions, the second position information of the calibration light sources and the first motion state information of the inertial measurement unit in the remote control device, in combination with the conversion relationship between the camera coordinate system of the photographing device and the inertial measurement unit coordinate system of the inertial measurement unit, which is calibrated in advance. The first attitude information represents the conversion relationship between the camera coordinate system of the photographing device and the world coordinate system.

[0058] Step 104, based on the first attitude information of the photographing device and the plane where the display screen is located under the world coordinate system determined based on the calibration light sources, the control position of the laser point displayed in the display screen by the laser emitted by the remote control device to the display device is determined.

[0059] In this step, after the remote control device determines the first attitude information of the shooting device, the remote control device determines a target ray of the shooting device in the world coordinate system based on the first attitude information of the shooting device, and calculates an intersection point of the target ray and a plane equation corresponding to a plane of the display screen determined by the calibration light source in the world coordinate system. The intersection point is a control point of the laser point displayed on the display screen by the laser emitted by the remote control device to the display device.

[0060] The remote control device of the embodiment uses a visual positioning method to obtain a target image collected by the shooting device on the display screen and a plurality of target feature regions containing the calibration light source in the target image. The remote control device determines first attitude information of the shooting device based on first position information of the target feature regions, second position information of the calibration light source, first motion state information of an inertial measurement unit in the remote control device, and a conversion relationship between a camera coordinate system of the shooting device and an inertial measurement unit coordinate system of the inertial measurement unit. The first attitude information represents a conversion relationship between the camera coordinate system of the shooting device and the world coordinate system. The remote control device determines a control point in the display screen based on the first attitude information of the shooting device and a plane in which the display screen is located in the world coordinate system determined based on the calibration light source. The remote control device converts information in the target image collected by the shooting device on the display screen to determine the control point in the display screen based on the conversion relationship between the camera coordinate system of the shooting device and the inertial measurement unit coordinate system of the inertial measurement unit, and the conversion relationship between the camera coordinate system of the shooting device and the world coordinate system. This avoids the control point determined by the remote control device from being affected by factors such as multipath effect, channel attenuation, and electromagnetic interference when the UWB technology is used to determine the control point, and improves the accuracy of the control point determined by the remote control device.

[0061] In an embodiment, step 103 comprises:

[0062] In step 1031, second attitude information corresponding to the inertial measurement unit in the remote control device is extracted from the first motion state information. The second attitude information represents a conversion relationship between an inertial measurement unit coordinate system of the inertial measurement unit and the world coordinate system.

[0063] In this step, the remote control device extracts second attitude information corresponding to the inertial measurement unit in the remote control device from the first motion state information. The second attitude information represents a conversion relationship between an inertial measurement unit coordinate system of the inertial measurement unit and the world coordinate system. It can be understood that the conversion relationship between the inertial measurement unit coordinate system of the inertial measurement unit and the world coordinate system is calibrated in advance. The conversion relationship between the inertial measurement unit coordinate system of the inertial measurement unit and the world coordinate system includes a rotation matrix of the inertial measurement unit relative to the world coordinate system.

[0064] Step 1032, the plurality of target feature regions are sorted to obtain a target feature region sequence.

[0065] In this step, the remote control device sorts the target feature regions after determining the plurality of target feature regions in the target image to obtain a target feature region sequence.

[0066] Preferably, the remote control device calculates a maximum bounding box based on the target feature regions by a convex hull algorithm, and then selects an arbitrary point of the convex hull center to perform polar coordinate transformation to sort the target feature regions. Exemplarily, as shown in Figure 5 the remote control device determines 5 target feature regions, and the middle position of the long dashed line in the figure is the origin of the polar coordinate. The 5 target feature regions determined are sorted in a counterclockwise order to obtain the number in the frame, and the target feature region sequence is [px0, px1, px2, px3, px4].

[0067] Alternatively, the remote control device can also sort the plurality of target feature regions in a counterclockwise, clockwise or other manner to obtain a target feature region sequence; which is not limited herein.

[0068] Step 1033, according to the preset target feature region sequence and the arrangement order of the target feature regions in the target feature region sequence, a target feature region subsequence set corresponding to the target feature region sequence is determined, and the target feature region subsequence set includes a plurality of target feature region subsequences.

[0069] In this step, the remote control device determines the target feature region subsequence set corresponding to the target feature region sequence according to the preset target feature region sequence and the arrangement order of the target feature regions in the target feature region sequence, and the target feature region subsequence set includes a plurality of target feature region subsequences.

[0070] Specifically, the remote control device obtains the number of target feature regions in the target feature region sequence and the number of target feature regions in the preset target feature region sequence, and determines the target feature region subsequence set in the target feature region sequence by C(n, m) times selection according to the number of target feature regions, the number of target feature regions, and the arrangement order of the target feature regions in the target feature region sequence, where n is the number of target feature regions, and m is the preset number of target feature regions.

[0071] Exemplarily, the number of target feature regions is 4, as shown in Figure 5As shown, the target feature region sequence is [px0, px1, px2, px3, px4], that is, the number of target feature regions is 5. The remote control device combines the arrangement order of the target feature regions in the target feature region sequence, uses the RANSAC algorithm to determine the target feature region subsequence set corresponding to the target feature region sequence through C(5, 4) times selection, and the five target feature region subsequences in the target feature region subsequence set are [px0, px1, px2, px3], [px0, px1, px2, px4], [px0, px2, px3, px4], [px1, px2, px3, px4], and [px1, px2, px3, px4]. It should be noted that the RANSAC (Random Sample Consensus) algorithm is an iterative method for solving the model fitting problem of outliers in data. The basic idea is to estimate the model parameters by randomly selecting data samples, and to judge which data points conform to the model and which data points are outliers according to the fitting degree of the estimated model and the data points.

[0072] In step 1034, the first reference pose information of the photographing device is determined according to the target feature region subsequence set, the preset calibration light source sequence, the first position information, and the second position information. The first reference pose information represents the reference conversion relationship between the camera coordinate system of the photographing device and the world coordinate system.

[0073] In this step, the remote control device determines the first reference pose information of the photographing device corresponding to each target feature region subsequence in the target feature region subsequence set according to the target feature region subsequence, the preset calibration light source sequence, the first position information, and the second position information. Then, the first reference pose information of the photographing device corresponding to each target feature region subsequence in the target feature region subsequence set is obtained. The first reference pose information represents the reference conversion relationship between the camera coordinate system of the photographing device and the world coordinate system in the case of a specific target feature region subsequence.

[0074] Specifically, step 1034 includes:

[0075] In step 10341, the target feature region subsequence and the preset calibration light source sequence are matched for each target feature region subsequence in the target feature region subsequence set, and a target feature region matching sequence set corresponding to the target feature region subsequence is obtained.

[0076] In this step, after the remote control device determines the target feature region subsequence set, for each target feature region subsequence in the target feature region subsequence set, the target feature region subsequence is matched with the preset calibration light source sequence respectively, to obtain a target feature region matching sequence set corresponding to the target feature region subsequence, and then the first reference attitude information of the shooting device corresponding to each target feature region subsequence in the target feature region subsequence set is determined.

[0077] As shown in the example, Figure 6 , Figure 6 It is shown in the example that the order of the preset calibration light source sequence is a clockwise order, the calibration light source sequence is [p0, p1, p2, p3], and the target feature region subsequence [px0, px1, px2, px3] is taken as an example. [px0, px1, px2, px3] is matched with [p0, p1, p2, p3], and four target feature region matching sequences in the target feature region matching sequence set can be obtained, which are [px0, px1, px2, px3], [px1, px2, px3, x0], [px2, px3, px0, px1], and [px3, px0, px1, px2]. That is, the feature region px0 can be matched with the calibration light sources p0, p1, p2, and p3 respectively, and the other feature regions are matched in turn according to the order. In the coordinates of each calibration light source, W and H represent the width and height of the display screen respectively, as shown in the example, Figure 6 , the x-axis of the world coordinate system is perpendicular to the display screen, and the plane coordinate system of the display screen is not coincident with the yoz plane coordinate system of the world coordinate system. Therefore, the calibration light source on the display screen must have a certain distance from the origin of the x-axis, which is z d . The above method can be used to determine the target feature region matching sequence set corresponding to each target feature region subsequence in the target feature region subsequence set.

[0078] In step 10342, for each target feature region subsequence, the first reference attitude information of the shooting device is determined based on the target feature region matching sequence set, the first position information, and the second position information.

[0079] In this step, after the remote control device determines the target feature region subsequence set, for each target feature region subsequence in the target feature region subsequence set, the target feature region subsequence is matched with the preset calibration light source sequence respectively, to obtain a target feature region matching sequence set corresponding to the target feature region subsequence, and then the first reference attitude information of the shooting device corresponding to each target feature region subsequence in the target feature region subsequence set is determined.

[0080] Specifically, the remote control device matches the target feature region first position information of each target feature region matching sequence in the target feature region matching sequence set with the second position information of each calibration light source in the preset calibration light source sequence, and calculates a homography matrix under a matching relationship between the target feature region corresponding to the target feature region matching sequence set and the calibration light source. The homography matrix includes the first reference attitude information of the photographing device corresponding to the target feature region subsequence. The first reference attitude information includes a rotation matrix and a translation matrix between a camera coordinate system of the photographing device and a world coordinate system.

[0081] It can be understood that for each target feature region matching sequence, the first reference attitude information corresponding to the photographing device can be determined.

[0082] In step 1035, the first attitude information of the photographing device is determined according to the extrinsic parameter between the inertial measurement unit and the photographing device, the first reference attitude information, and the second attitude information.

[0083] In this step, after the remote control device determines the plurality of first reference attitude information, the remote control device converts each first reference attitude information into second reference attitude information according to the extrinsic parameter between the inertial measurement unit and the photographing device. The second reference attitude information includes a rotation matrix and a translation matrix between an inertial measurement unit coordinate system of the inertial measurement unit and a world coordinate system. The remote control device determines the first attitude information of the photographing device based on the plurality of second reference attitude information and the second attitude information.

[0084] Specifically, step 1035 includes:

[0085] In step 10351, each first reference attitude information is converted into second reference attitude information of the inertial measurement unit based on the extrinsic parameter between the inertial measurement unit and the photographing device.

[0086] In this step, after the remote control device determines the first reference attitude information corresponding to each target feature region matching sequence, the remote control device converts each first reference attitude information into second reference attitude information of the inertial measurement unit based on the extrinsic parameter between the inertial measurement unit and the photographing device. Specifically, the extrinsic parameter between the inertial measurement unit and the photographing device is determined in advance by calibration. The extrinsic parameter is represented as a rotation matrix between a camera coordinate system of the photographing device and an inertial measurement unit coordinate system of the inertial measurement unit. Each first reference attitude information is converted into second reference attitude information of the inertial measurement unit by formula, wherein, is the second reference attitude information of the inertial measurement unit. The second reference attitude information includes a reference rotation matrix between the inertial measurement unit coordinate system of the inertial measurement unit and the world coordinate system, The first reference attitude information of the photographing device is a rotation matrix of the photographing device relative to the world coordinate system, R ic T represents a transpose matrix.

[0087] In step 10352, the error between each second reference attitude information and the second attitude information is calculated respectively to obtain an error set.

[0088] In this step, the remote control device calculates the error between each second reference attitude information and the second attitude information respectively to obtain an error set. Specifically, the inertial measurement unit can measure the second attitude information of itself relative to the world coordinate system, and the second attitude information includes a rotation matrix between the inertial measurement unit coordinate system of the inertial measurement unit and the world coordinate system. The remote control device calculates the error between the second reference attitude information and the second attitude information through a function function The second reference attitude information of the inertial measurement unit is R wi The second attitude information of the inertial measurement unit is R is decomposed into roll_a and pitch_a, and R wi is decomposed into roll_b and pitch_b, roll is the roll of the inertial measurement unit, and pitch is the pitch angle of the inertial measurement unit; the remote control device further calculates the sum of the absolute values of the angle differences between roll_a and roll_b and between pitch_a and pitch_b as the error of the second reference attitude information and the second attitude information.

[0089] In step 10353, the first attitude information of the photographing device is determined based on the error set.

[0090] In this step, the remote control device compares each error in the error set with a first preset error threshold, and determines the first attitude information of the photographing device according to the comparison result.

[0091] Specifically, step 10353 includes:

[0092] In step 103531, if there is an error in the error set that is less than or equal to the first preset error threshold, the second reference attitude information corresponding to the minimum error in the error set is determined as target second reference attitude information, and the first reference attitude information corresponding to the target second reference attitude information before coordinate conversion is determined as the first attitude information of the photographing device.

[0093] In this step, the remote control device compares each error in the error set with the first preset error threshold, and if it is determined that there is an error in the error set that is less than or equal to the first preset error threshold, the second reference attitude information corresponding to the smallest error among all errors less than or equal to the first preset error threshold is determined as target second reference attitude information, and the first reference attitude information corresponding to the target second reference attitude information before coordinate conversion is determined as the first attitude information of the photographing device.

[0094] In step 103532, if each error in the error set is greater than the first preset error threshold, the first attitude information of the photographing device is determined based on the target images of the continuous preset number of frames.

[0095] In this step, the remote control device compares each error in the error set with the first preset error threshold, and if it is determined that each error in the error set is greater than the first preset error threshold, the first attitude information of the photographing device is determined based on the target images of the continuous preset number of frames.

[0096] The remote control device of the embodiment determines the first reference attitude information of the photographing device through matching of the target feature region and the calibration light source, converts each first reference attitude information into second reference attitude information of the inertial measurement unit in combination with the external parameter between the inertial measurement unit and the photographing device, and finally determines the first attitude information of the photographing device in each first reference attitude information according to the error of the second attitude information of the inertial measurement unit from the second reference attitude information, thereby improving the accuracy of determining the first attitude information of the photographing device and further helping to improve the accuracy of the remote control device in determining the control point.

[0097] In an embodiment, determining the first attitude information of the photographing device based on the target images of the continuous preset number of frames comprises:

[0098] In step 1035321, the target images of the continuous preset number of frames are acquired, and adjacent two target images are divided into an image group.

[0099] In this step, the remote control device acquires the target images of the continuous preset number of frames, and divides adjacent two target images in the target images of the continuous preset number of frames into an image group. For example, 10 continuous target images are acquired, and correspondingly, the first frame and the second frame target images are divided into a group, the second frame and the third frame target images are divided into a group, the third frame and the fourth frame target images are divided into a group, and so on.

[0100] In step 1035322, for each image group, target feature regions in the target images of two adjacent frames are determined according to the angular velocity measured by the corresponding IMU and the extrinsic parameters between the IMU and the shooting device.

[0101] In this step, for each image group, the remote control device determines the target feature region corresponding to each target image according to the angular velocity measured by the corresponding IMU and the extrinsic parameters between the IMU and the shooting device.

[0102] Specifically, for each image group, the remote control device respectively acquires the angular velocity w measured by the IMU when shooting the image group t , and determines the target inter-frame reference attitude information of the IMU corresponding to the target images in the image group according to w t . is an anti-symmetric matrix of w t , and the target inter-frame reference attitude information of the IMU corresponding to the target images in the image group is calculated based on the formula . The remote control device acquires the extrinsic parameters R between the IMU and the shooting device ic , and calculates the target inter-frame reference attitude information of the shooting device corresponding to the target images in the image group based on the formula . The rotation of the z-axis of is decomposed to obtain The target images in the image group are rotated by to align the z-axis of the target images as much as possible, and then a combined optimization algorithm is used to match the nearest neighbors and similar surrounding geometries, which can complete the inter-frame matching through a bipartite graph to determine the target feature region corresponding to each target image in the image group.

[0103] In step 1035323, for each image group, the first inter-frame reference attitude information of the shooting device corresponding to the image group is determined based on the third position information of the target feature regions in the target images of two adjacent frames and the second position information of the calibrated light source.

[0104] In this step, for each image group, the remote control device determines the first inter-frame reference attitude information of the shooting device corresponding to the image group based on the third position information of the target feature regions in the target images of two adjacent frames and the second position information of the calibrated light source. It can be understood that the third position information is the position information of the target feature region in the camera coordinate system corresponding to the target image; for each image group, the first inter-frame reference attitude information of the shooting device corresponding to the image group can be determined.

[0105] Specifically, the remote control device calculates, according to third position information of a target feature region in target images of two adjacent frames and in combination with second position information of each calibration light source in a preset calibration light source sequence, a homography matrix under a matching relationship between the target feature region and the calibration light source corresponding to the image group, the homography matrix including first interframe reference pose information of a photographing device corresponding to the image group, the first interframe reference pose information being a reference conversion relationship between a camera coordinate system and a world coordinate system when the photographing device photographs the two adjacent target images, the reference conversion relationship including a reference rotation matrix and a reference translation matrix.

[0106] In this step, the remote control device determines the first pose information of the photographing device according to the first interframe reference pose information of each image group and preset interframe reference pose information of the inertial measurement unit.

[0107] In this step, the remote control device determines the first pose information of the photographing device according to the first interframe reference pose information of each image group and preset interframe reference pose information of the inertial measurement unit. The preset interframe reference pose information is a preset conversion relationship between an inertial measurement unit coordinate system and a world coordinate system when the inertial measurement unit photographs two adjacent target images, the preset conversion relationship including a preset rotation matrix and a preset translation matrix.

[0108] Specifically, step 1035324 includes:

[0109] Step 10353241, for each image group, converts each first interframe reference pose information into second interframe reference pose information of the inertial measurement unit based on external parameters between the inertial measurement unit and the photographing device, the second interframe reference pose information being a reference conversion relationship between the inertial measurement unit coordinate system and the world coordinate system when the inertial measurement unit photographs two adjacent target images of the photographing device.

[0110] In this step, for each image group, the remote control device converts each first interframe reference pose information into second interframe reference pose information of the inertial measurement unit based on external parameters between the inertial measurement unit and the photographing device. The second interframe reference pose information is a reference conversion relationship between the inertial measurement unit coordinate system and the world coordinate system when the inertial measurement unit photographs two adjacent target images of the photographing device, the reference conversion relationship including a reference rotation matrix and a reference translation matrix.

[0111] Specifically, for each image group, after the remote control device determines the first interframe reference pose information of the photographing device corresponding to the image group, the remote control device determines, according to external parameters between the inertial measurement unit and the photographing device and the first interframe reference pose information, second interframe reference pose information of the inertial measurement unit corresponding to the image group. Specifically, the remote control device determines the second interframe reference pose information of the inertial measurement unit corresponding to the image group based on the formula determining second inter-frame reference attitude information of the inertial measurement unit corresponding to the image group, the second inter-frame reference attitude information represents a rotation matrix of the inertial measurement unit relative to the world coordinate system, the first inter-frame reference attitude information is R ic the extrinsic parameter is between the inertial measurement unit and the photographing apparatus.

[0112] In step 10353242, for each image group, a difference value between the second inter-frame reference attitude information and the corresponding preset inter-frame reference attitude information is calculated.

[0113] In this step, for each image group, the remote control device calculates a difference value between the image group and the second inter-frame reference attitude information and the corresponding preset inter-frame reference attitude information, and then determines the difference value corresponding to each image group. Specifically, the remote control device calculates the difference value of the second inter-frame reference attitude information and the preset inter-frame reference attitude information corresponding to each image group through the formula In this step, for each image group, the remote control device calculates a difference value between the image group and the second inter-frame reference attitude information and the corresponding preset inter-frame reference attitude information, and then determines the difference value corresponding to each image group. Specifically, the remote control device calculates the difference value of the second inter-frame reference attitude information and the preset inter-frame reference attitude information corresponding to each image group through the formula

[0114] In step 10353243, target inter-frame reference attitude information is obtained, the target inter-frame reference attitude information is determined as the first attitude information of the photographing apparatus, and the target inter-frame reference attitude information is the first inter-frame reference attitude information corresponding to the second inter-frame reference attitude information with the smallest difference value and less than or equal to a second preset error threshold.

[0115] In this step, the remote control device compares the difference value of the second inter-frame reference attitude information and the preset inter-frame reference attitude information corresponding to each image group with a preset difference threshold, determines the first inter-frame reference attitude information corresponding to the second inter-frame reference attitude information with the smallest difference value and less than or equal to a second preset error threshold as the target inter-frame reference attitude information, and then determines the target inter-frame reference attitude information as the first attitude information of the photographing apparatus. The preset difference threshold is a unit matrix

[0116] When the remote control device of this embodiment determines that the error of each second reference attitude information and the second attitude information of the inertial measurement unit is greater than the preset error threshold, the observation of the target image of the plurality of continuous two frames is obtained, and the first attitude information of the photographing apparatus is determined, which improves the accuracy of determining the first attitude information of the photographing apparatus, and further helps to improve the accuracy of the remote control device in determining the control point.

[0117] In an embodiment, step 104 includes:

[0118] Step 1041, obtaining a vertical axis direction of a camera coordinate system of the photographing device and a rotation matrix of the photographing device relative to a world coordinate system in the first attitude information.

[0119] In this step, the remote control device obtains a vertical axis direction of a camera coordinate system of the photographing device and a rotation matrix of the photographing device relative to a world coordinate system in the first attitude information.

[0120] Step 1042, rotating the vertical axis direction based on the rotation matrix to obtain a ray direction of the photographing device relative to the world coordinate system.

[0121] In this step, the remote control device rotates the vertical axis direction of the camera coordinate system of the photographing device based on the rotation matrix of the photographing device relative to the world coordinate system to obtain a ray direction of the photographing device relative to the world coordinate system.

[0122] Step 1043, determining a target ray according to the ray direction and position information of the photographing device.

[0123] In this step, the remote control device obtains position information of the photographing device, takes the position information of the photographing device as an origin, and determines a target ray based on the origin and the ray direction.

[0124] Step 1044, calculating an intersection point between the target ray and a plane in which the display screen determined based on the calibration light source is located in the world coordinate system to obtain a reference control point of the remote control device in the world coordinate system.

[0125] In this step, the remote control device calculates an intersection point between the target ray and a plane equation of the display screen determined based on the calibration light source in the world coordinate system to obtain a reference control point of the remote control device in the world coordinate system. It should be noted that the plane equation of the display screen relative to the world coordinate system is set in advance, as shown in the following formula: Figure 7 R wc is a rotation matrix of the photographing device in the world coordinate system, t wc is a translation matrix of the photographing device in the world coordinate system, dir(R wc .z) refers to the target ray, and nx+d=0 refers to the plane equation of the display screen in the world coordinate system. The black dot in the figure is a reference control point of a laser point displayed in the display screen by the laser emitted by the remote control device to the display device in the world coordinate system.

[0126] Step 1045, determining a control point of a laser point displayed in the display screen by the laser emitted by the remote control device to the display device according to a coordinate of the reference control point and a conversion matrix between a coordinate system of the display screen and the world coordinate system.

[0127] In this step, the remote control device obtains the coordinates of the reference control point after determining the reference control point in the world coordinate system. Since the ultimate goal is to determine the projection point coordinates of the remote control device on the display screen, the remote control device converts the coordinates of the reference control point to determine the coordinates of the control point of the remote control device on the display screen according to the conversion matrix of the coordinate system of the display screen and the world coordinate system, that is, the control point of the laser point displayed in the display screen by the laser emitted by the remote control device to the display device. The conversion matrix of the coordinate system of the display screen and the world coordinate system is set in advance.

[0128] The remote control device of the embodiment determines the control point of the laser point displayed in the display screen by the laser emitted by the remote control device to the display device according to the calculated first attitude information of the shooting device, the plane equation of the display screen relative to the world coordinate system, and the conversion matrix of the coordinate system of the display screen and the world coordinate system, avoiding the influence of factors such as multipath effect, channel attenuation, and electromagnetic interference when using UWB technology, and using visual positioning to improve the accuracy of the remote control device in determining the control point.

[0129] Reference Figure 8 The second embodiment of the present application is proposed, and the difference between the second embodiment of the present application and the first embodiment is that the control point determination method further comprises:

[0130] In step 105, when the number of the extracted target feature regions is less than the number of the calibration light sources, the first attitude information of the shooting device is determined according to the first position information of the target feature region, the second position information of the calibration light source, the camera parameter of the shooting device, and the first motion state information of the inertial measurement unit in the remote control device.

[0131] In this step, if the remote control device determines that the number of the extracted target feature regions is less than the number of the calibration light sources, the remote control device uses the two-point PNP algorithm to determine the first attitude information of the shooting device in combination with the first position information of the target feature region, the second position information of the calibration light source, the camera parameter of the shooting device, and the gravity vector and angular velocity measured by the inertial measurement unit in the first motion state information of the remote control device.

[0132] It should be noted that the camera parameter of the shooting device is the camera parameter of the shooting device after calibration, including the distortion parameter of the camera, the principal point of the camera, the focal length of the camera, etc.

[0133] Specifically, step 105 comprises:

[0134] In step 1051, the gravity vector and angular velocity measured by the inertial measurement unit in the first motion state information are extracted.

[0135] In this step, the remote control device extracts the gravity vector and the angular velocity measured by the inertial measurement unit in the first motion state information.

[0136] In this step, the remote control device extracts the gravity vector and the angular velocity measured by the inertial measurement unit in the first motion state information.

[0137] In this step, the remote control device extracts the gravity vector and the angular velocity measured by the inertial measurement unit in the first motion state information.

[0138] In this step, the remote control device extracts the gravity vector and the angular velocity measured by the inertial measurement unit in the first motion state information.

[0139] In this step, the remote control device extracts the gravity vector and the angular velocity measured by the inertial measurement unit in the first motion state information.

[0140] In this step, the remote control device extracts the gravity vector and the angular velocity measured by the inertial measurement unit in the first motion state information.

[0141] In this step, the remote control device extracts the gravity vector and the angular velocity measured by the inertial measurement unit in the first motion state information.

[0142] In this step, the remote control device extracts the gravity vector and the angular velocity measured by the inertial measurement unit in the first motion state information.

[0143] In this step, the remote control device extracts the gravity vector and the angular velocity measured by the inertial measurement unit in the first motion state information.

[0144] Step 106, according to the first position information, the second position information, the first attitude information and the camera calibration center coordinates of the shooting device, determining the control point of the laser point displayed in the display screen by the laser emitted by the remote control device to the display device.

[0145] In this step, the remote control device determines the target homography matrix according to the first position information, the second position information and the first attitude information, and determines the control point of the laser point displayed in the display screen by the laser emitted by the remote control device to the display device according to the target homography matrix and the camera calibration center coordinates of the shooting device; wherein the camera calibration center coordinates are the position information of the camera calibration center in the camera coordinate system, and the camera parameters are calibrated in advance.

[0146] Specifically, step 106 includes:

[0147] Step 1061, determining the target homography matrix according to the first position information, the second position information and the first attitude information.

[0148] In this step, the remote control device determines the target homography matrix according to the first position information, the second position information and the first attitude information. Specifically, the corresponding relationship between the first position information of the target feature region on the target image and the second position information of the calibration light source in the world coordinate system is determined, at this time, the remote control device can determine the target homography matrix according to the first position information, the second position information and the first attitude information, and the target homography matrix represents the conversion relationship between the camera coordinate system of the shooting device and the world coordinate system.

[0149] Step 1062, according to the target homography matrix and the camera calibration center coordinates of the shooting device, determining the reference control point of the remote control device in the world coordinate system.

[0150] In this step, after the remote control device determines the target homography matrix, the remote control device determines the reference control point of the remote control device in the world coordinate system according to the target homography matrix and the coordinates of the camera calibration center of the shooting device; as Figure 9 The black dot is the camera calibration center, and the irregular quadrilateral is the screen image of the coordinate system of the shooting device on the target image.

[0151] Step 1063, according to the coordinates of the reference control point and the conversion matrix between the coordinate system of the display screen and the world coordinate system, determining the control point of the laser point displayed in the display screen by the laser emitted by the remote control device to the display device.

[0152] In this step, the remote control device determines the reference control point in the world coordinate system, and obtains the coordinates of the reference control point. Since the ultimate goal is to determine the projection point coordinates of the remote control device on the display screen, at this time, the remote control device converts the coordinates of the reference control point into the coordinates of the control point of the remote control device on the display screen according to the conversion matrix of the coordinate system of the display screen and the world coordinate system, that is, the control point of the laser point displayed in the display screen by the laser emitted by the remote control device to the display device. The conversion matrix of the coordinate system of the display screen and the world coordinate system is set in advance.

[0153] Further, when the remote control device observes enough features again, that is, the number of target feature regions of the target image is greater than the preset number, the remote control device enters the process of determining the first pose information of the shooting device in step 103. Without calculating the rotation matrix and translation matrix of the shooting device relative to the world coordinate system, the difference between the center position of the homography matrix calculated by observation and the current remote control device position is recorded, and this error is divided into a small enough value, which does not easily cause sensory reaction. Gradually correct the projection point coordinates of the remote control device on the display screen determined when the number of target feature regions of the target image is less than the preset number, such as calculating the motion size of the laser point position under two motions. According to the amount of current motion, the error can be gradually corrected according to 1 / 100 of the proportion, until the recorded error is less than 1px, which can be considered as correction completion.

[0154] The remote control device of the embodiment can still determine the first pose information of the shooting device when the number of target feature regions in the target image of a continuous preset number of frames is less than the preset number, that is, the number of observed features is less. Then, according to the first position information, the second position information, the first pose information and the camera calibration center coordinates of the shooting device, the control point of the laser point displayed in the display screen by the laser emitted by the remote control device to the display device is determined, which can improve the robustness of the remote control device positioning, and avoid the influence of factors such as multipath effect, channel attenuation and electromagnetic interference when using UWB technology. Instead, the visual positioning method is used to improve the accuracy of the remote control device in determining the control point.

[0155] Reference Figure 10 The third embodiment of the present application is proposed, and the difference between the third embodiment of the present application and the first embodiment to the second embodiment is that before obtaining the target image collected by the shooting device on the display screen, the method comprises:

[0156] Step a: obtaining a calibration ruler image by shooting the calibration ruler pre-vertically placed on the ground from at least two preset directions by the shooting device, wherein the calibration ruler image comprises a feature region corresponding to a target light source on the calibration ruler.

[0157] In this step, first, the calibration ruler is placed on a plane perpendicular to the ground, and the remote control device captures an image of the plane on which the calibration ruler is placed from at least two preset orientations by the shooting device to obtain a calibration ruler image. Among them, the calibration ruler has a plurality of target light sources, and the calibration ruler image includes a feature area corresponding to the target light source.

[0158] Illustratively, the remote control device captures an image of the plane on which the calibration ruler is placed from five preset orientations by the shooting device to obtain five calibration ruler images. The five preset orientations can all observe all target light sources assembled on the calibration ruler.

[0159] Step b, obtaining first motion data of the inertial measurement unit when capturing the calibration ruler image at the first preset orientation, and determining initial attitude information of the inertial measurement unit at the first preset orientation according to the first motion data.

[0160] In this step, the remote control device obtains first motion data of the inertial measurement unit when capturing the calibration ruler image at the first preset orientation, and determines initial attitude information of the inertial measurement unit at the first preset orientation according to the first motion data. Specifically, the first motion data of the inertial measurement unit when capturing the calibration ruler image at the first preset orientation includes acceleration, angular velocity and corresponding timestamp data, and the remote control device determines the initial attitude information of the inertial measurement unit at the first preset orientation according to the acceleration, angular velocity and corresponding timestamp data,

[0161] Step c, obtaining third position information of the target light source in the world coordinate system, fourth position information of the feature area in the calibration ruler image, and angular velocity of the switching process between every two preset orientations.

[0162] Step d, determining the extrinsic parameter between the inertial measurement unit and the shooting device according to the third position information, the fourth position information, the angular velocity of the switching process between every two preset orientations and the initial attitude information.

[0163] In steps c to d, the remote control device obtains third position information of the target light source in the world coordinate system, fourth position information of the feature area in the calibration ruler image, and angular velocity of the switching process between every two preset orientations, and determines the extrinsic parameter between the inertial measurement unit and the shooting device according to the third position information, the fourth position information, the angular velocity of the switching process between every two preset orientations and the initial attitude information.

[0164] Specifically, the remote control device determines the attitude information of the inertial measurement unit at each preset orientation except the first preset orientation according to the angular velocity of the switching process between every two preset orientations and the initial attitude information, the attitude information being a rotation matrix of the inertial measurement unit at each preset orientation relative to the world coordinate system; the remote control device determines the rotation matrix of the shooting device relative to the world coordinate system when shooting the calibration ruler image at each preset orientation according to the third position information and the fourth position information by using a homography matrix solution; and the remote control device determines the extrinsic parameter between the inertial measurement unit and the shooting device according to the rotation matrix of the inertial measurement unit at any one preset orientation relative to the world coordinate system and the rotation matrix of the shooting device at any one preset orientation relative to the world coordinate system, and the specific formula is R ic wi_i T R wc_i , wherein R ic is the extrinsic parameter between the inertial measurement unit and the shooting device, R wi_i is the rotation matrix of the inertial measurement unit at any one preset orientation relative to the world coordinate system, and R wc_i is the rotation matrix of the shooting device at any one preset orientation relative to the world coordinate system.

[0165] Further, the known rotation matrix of the inertial measurement unit at any one preset orientation relative to the world coordinate system R is actually different from the real rotation matrix R wi_0 of the inertial measurement unit at any one preset orientation relative to the world coordinate system by an angle θ yaw . R(θ yaw ) and the conversion relationship are shown in the following formula Therefore, the specific formula R ic for calculating the extrinsic parameter between the inertial measurement unit and the shooting device is R wi_i T R wc_i , which can be transformed as follows:

[0166]

[0167] , wherein R i，i+1 imu =∫ i i+1 exp(gyr × )dt, R i，i+1 imu is determined according to the angular velocity gyr and represents a variable changing θ yaw_i .

[0168] ​At this point, based on the attitude information of the inertial measurement units and the shooting device at two adjacent preset orientations, the following hand-eye calibration format AX = XB can be obtained:

[0169]

[0170] in This is a typical hand-eye calibration problem in robotics, requiring at least two sets of A and B diagrams. Then, the initial rotation angle θ can be calculated. yaw Determine the initial rotation angle θ yaw Then, according to the formula R wi_i This allows us to determine the corresponding value; at this point, the remote control device can use any R. wi_i and R wc_i According to formula R ic =R wi_i T R wc_i Solve for R ic .

[0171] For example, this application can obtain 4 sets of AB values, namely (i, i+1), i∈(0, 1, 2, 3, 4), and then solve for the initial rotation angle θ. yaw Since we obtained more than one observation at this step, we can finally optimize the value using least squares, and then we can use any R. ic =R wi_i T R wc_i Solve R ic .

[0172] Furthermore, before determining the extrinsic parameters between the inertial measurement unit (IMU) and the imaging device, the remote control equipment first calibrates the IMU's error model, with the output being K. acc M acc bias acc K gyr M gyr bias gyr This is the correction matrix for both acceleration values ​​and angular velocity data, and the correction formula is as follows:

[0173]

[0174] in, This represents the scale error of the measured data for each axis. When there is no scale error in the measurement, s... i =1, i∈(a0, a1, a2, g0, g1, g2), where S i This refers to all variables representing the scaling error above, where Si includes S. a0 -Sg2 and S g0 -S g2 , represent the scale error value of each axis. Wherein, is the matrix of correcting the installation orthogonality error of the accelerometer and the angular velocity meter (any two axes should be perpendicular to each other), a yz , a zy , a zx is the orthogonality error of the acceleration between each two axes, r xz , r xy , r yz , r yx , r zy , r zx is the orthogonality error of the angular velocity between each two axes. Wherein, is the zero offset of the corresponding sensor, b a0 to b a2 represent the acceleration zero offset of each axis, b g0 to b g2 represent the angular velocity zero offset of each axis. acc and gyr are the measurement values of the accelerometer and the angular velocity meter at any time.

[0175] Before the inertial measurement unit measures the acceleration and angular velocity each time, it is corrected according to the above correction formula and then output.

[0176] The remote control device of the embodiment calibrates the external parameters between the inertial measurement unit and the shooting device before shooting the target image of the display screen by the shooting device, so as to facilitate subsequent determination of the attitude information of the shooting device relative to the world coordinate according to the external parameters, improve the accuracy of determining the attitude information of the shooting device relative to the world coordinate, and help to use the visual positioning mode, avoid the influence of factors such as multipath effect, channel attenuation, electromagnetic interference, etc. when using UWB technology, and improve the accuracy of the remote control device in determining the control point.

[0177] The embodiment also provides a remote control device positioning device, as shown in Figure 11 , the remote control device positioning device can comprise:

[0178] The acquisition module 1001 is used to acquire the target image collected by the shooting device on the display screen; the display screen contains a plurality of preset calibration light sources.

[0179] The extraction module 1002 is used to extract a plurality of target feature regions containing the calibration light sources from the target image.

[0180] The first determining module 1003 is configured to determine first attitude information of the photographing device according to first position information of the target feature region, second position information of the calibration light source, and first motion state information of an inertial measurement unit in the remote control device, when the number of the extracted target feature regions is greater than or equal to the number of the calibration light sources.

[0181] The second determining module 1004 is configured to determine a control point of a laser point displayed on the display screen by the laser emitted by the remote control device to the display device, based on the first attitude information of the photographing device and a plane in which the display screen is located in a world coordinate system determined based on the calibration light source.

[0182] In an optional example, the first determining module is further configured to:

[0183] extract second attitude information corresponding to the inertial measurement unit in the remote control device in the first motion state information, the second attitude information representing a conversion relationship between an inertial measurement unit coordinate system of the inertial measurement unit and the world coordinate system;

[0184] sort the plurality of target feature regions to obtain a target feature region sequence;

[0185] determine a target feature region subsequence set corresponding to the target feature region sequence according to a preset calibration light source sequence composed of the plurality of calibration light sources after a preset sorting and an arrangement order of the target feature region in the target feature region sequence, the target feature region subsequence set including a plurality of target feature region subsequences;

[0186] determine first reference attitude information of the photographing device according to the target feature region subsequence set, the preset calibration light source sequence, the first position information, and the second position information, the first reference attitude information representing a reference conversion relationship between a camera coordinate system of the photographing device and the world coordinate system;

[0187] determine first attitude information of the photographing device according to the first reference attitude information, the second attitude information, and an external parameter between the inertial measurement unit and the photographing device.

[0188] In an optional example, the first determining module is further configured to:

[0189] for each target feature region subsequence in the target feature region subsequence set, match the target feature region subsequence with the preset calibration light source sequence to obtain a target feature region matching sequence set corresponding to the target feature region subsequence;

[0190] For each of the target feature region subsequence, based on the target feature region matching sequence set, the first position information and the second position information, determine the first reference attitude information of the photographing device.

[0191] In an optional example, the first determining module is further configured to:

[0192] convert each of the first reference attitude information into second reference attitude information of the inertial measurement unit based on the extrinsic parameters between the inertial measurement unit and the photographing device;

[0193] respectively calculate errors between each of the second reference attitude information and the second attitude information to obtain an error set;

[0194] determine the first attitude information of the photographing device based on the error set.

[0195] In an optional example, the first determining module is further configured to:

[0196] if there is an error less than or equal to a first preset error threshold in the error set, determine the second reference attitude information corresponding to the minimum error in the error set as target second reference attitude information, and determine the first reference attitude information corresponding to the target second reference attitude information before coordinate conversion as the first attitude information of the photographing device;

[0197] if each error in the error set is greater than the first preset error threshold, determine the first attitude information of the photographing device based on the target images of a continuous preset number of frames.

[0198] In an optional example, the first determining module is further configured to:

[0199] obtain the target images of a continuous preset number of frames, and divide adjacent two frames of the target images into an image group;

[0200] for each of the image group, determine target feature regions in the target images of adjacent two frames in the image group according to the angular velocity measured by the inertial measurement unit corresponding to the image group and the extrinsic parameters between the inertial measurement unit and the photographing device;

[0201] for each of the image group, determine first inter-frame reference attitude information of the photographing device corresponding to the image group based on third position information of the target feature regions in the target images of adjacent two frames and the second position information of the calibration light source;

[0202] determine the first attitude information of the photographing device according to the first inter-frame reference attitude information and preset inter-frame reference attitude information of the inertial measurement unit.

[0203] The third position information is position information of the target feature region in a camera coordinate system corresponding to the target image; the first inter-frame reference attitude information is a reference conversion relationship between the camera coordinate system and a world coordinate system when the photographing device photographs adjacent two frames of the target image; and the preset inter-frame reference attitude information is a preset conversion relationship between the IMU coordinate system and the world coordinate system when the IMU photographs adjacent two frames of the target image.

[0204] In an optional example, the first determining module is further configured to:

[0205] For each of the image groups, convert each of the first inter-frame reference attitude information into second inter-frame reference attitude information of the IMU based on an external parameter between the IMU and the photographing device, the second inter-frame reference attitude information being a reference conversion relationship between the IMU coordinate system and the world coordinate system when the photographing device photographs adjacent two frames of the target image;

[0206] For each of the image groups, calculate a difference value between the second inter-frame reference attitude information and the corresponding preset inter-frame reference attitude information;

[0207] Obtain a target inter-frame reference attitude information, and determine first attitude information of the photographing device based on the target inter-frame reference attitude information, the target inter-frame reference attitude information being the first inter-frame reference attitude information corresponding to the second inter-frame reference attitude information with the smallest difference value and less than or equal to a second preset error threshold.

[0208] In an optional example, the second determining module is further configured to:

[0209] Obtain a vertical axis direction of a camera coordinate system of the photographing device and a rotation matrix of the photographing device relative to a world coordinate system in the first attitude information;

[0210] Rotate the vertical axis direction based on the rotation matrix to obtain a ray direction of the photographing device relative to the world coordinate system;

[0211] Determine a target ray according to the ray direction and position information of the photographing device;

[0212] Calculate an intersection point between the target ray and a plane in which the display screen is located in the world coordinate system based on the calibration light source to obtain a reference control point of the remote control device in the world coordinate system;

[0213] According to the coordinates of the reference control point and a conversion matrix between a coordinate system of the display screen and the world coordinate system, a control point of a laser point displayed in the display screen by the laser emitted by the remote control device to the display device is determined.

[0214] In an optional example, the control point determination apparatus further comprises a third determination module, which is configured to:

[0215] When the number of the extracted target feature regions is less than the number of the calibration light sources, first pose information of the photographing device is determined according to the first position information of the target feature regions, the second position information of the calibration light sources, camera parameters of the photographing device and first motion state information of an inertial measurement unit in the remote control device.

[0216] According to the first position information, the second position information, the first pose information and a camera calibration center coordinate of the photographing device, a control point of a laser point displayed in the display screen by the laser emitted by the remote control device to the display device is determined.

[0217] The camera calibration center coordinate is position information of a camera calibration center in the camera coordinate system, and the camera parameters are calibrated in advance.

[0218] In an optional example, the third determination module is further configured to:

[0219] The gravity vector and the angular velocity measured by the inertial measurement unit in the first motion state information are extracted.

[0220] According to the angular velocity and an extrinsic parameter between the inertial measurement unit and the photographing device, a rotation pose of the photographing device in a world coordinate system is determined.

[0221] According to the first position information of the target feature regions, the camera parameters of the photographing device, the rotation pose and the gravity vector, a rotation matrix of the photographing device is determined.

[0222] According to the first position information of the target feature regions, the second position information of the calibration light sources, the camera parameters of the photographing device and the rotation matrix, a translation matrix of the photographing device is determined.

[0223] According to the translation matrix and the rotation matrix, the first pose information of the photographing device is determined.

[0224] In an optional example, the third determination module is further configured to:

[0225] determine a target homography matrix according to the first position information, the second position information and the first attitude information;

[0226] determine a reference control point of the remote control device in the world coordinate system according to the target homography matrix and a camera calibration center coordinate of the photographing device;

[0227] determine a control point of a laser point displayed in the display screen according to a coordinate of the reference control point and a conversion matrix between a coordinate system of the display screen and the world coordinate system.

[0228] In an optional example, the extraction module is further configured to:

[0229] perform feature region detection on the target image to determine morphological information and size information of each feature region in the target image;

[0230] determine an aspect ratio of each feature region according to the morphological information of each feature region;

[0231] determine a pixel number corresponding to each feature region according to the size information of each feature region;

[0232] determine a target feature region containing the calibration light source in the target image as a feature region with an aspect ratio less than a preset ratio and a pixel number greater than a preset value;

[0233] extract a plurality of target feature regions containing the calibration light source from the target image.

[0234] The scheme of the embodiment is adopted, a target image collected by the photographing device on the display screen and a plurality of target feature regions containing the calibration light source in the target image are acquired in a visual positioning manner, first position information of the target feature regions, second position information of the calibration light source, and first motion state information of an inertial measurement unit in the remote control device are combined, a conversion relationship between a camera coordinate system of the photographing device and an inertial measurement unit coordinate system of the inertial measurement unit is combined, first attitude information of the photographing device is determined, the first attitude information represents a conversion relationship between the camera coordinate system of the photographing device and a world coordinate system, the control point of the remote control device in the display screen is determined based on the first attitude information of the photographing device and a plane in which the display screen is located in the world coordinate system determined based on the calibration light source, information in the target image collected by the photographing device on the display screen is converted to determine the control point of the remote control device in the display screen based on the conversion relationship between the camera coordinate system of the photographing device and the inertial measurement unit coordinate system of the inertial measurement unit and the conversion relationship between the camera coordinate system of the photographing device and the world coordinate system, avoiding that the control point is affected by factors such as multipath effect, channel attenuation, and electromagnetic interference when the UWB technology is used to determine the control point, and improving the accuracy of the remote control device in determining the control point.

[0235] Correspondingly, the embodiment of the application also provides a remote control device, as shown in Figure 12 Figure 12 The remote control device provided by the embodiment of the application is a structure schematic diagram. The remote control device 1100 client and / or server includes a processor 1101 having one or more processing cores, a memory 1102 having one or more computer readable storage media, and a computer program stored on the memory 1102 and executable on the processor. The processor 1101 is electrically connected to the memory 1102. Those skilled in the art can understand that the structure of the remote control device shown in the figure does not constitute a limitation on the remote control device, and can include more or fewer components than shown, or combine certain components, or different component arrangements.

[0236] The processor 1101 is the control center of the remote control device 1100, which connects all parts of the remote control device 1100 through various interfaces and lines, executes various functions of the remote control device 1100 and processes data by running or loading software programs and / or units stored in the memory 1102 and calling data stored in the memory 1102, thereby monitoring the remote control device 1100 as a whole. The processor 1101 can be a processor CPU, a graphics processor GPU, a network processor (NP), etc., and can realize or execute the methods, steps and logic block diagrams disclosed in the embodiments of the application.

[0237] ​In this embodiment of the application, the processor 1101 in the remote control device 1100 will load the computer program corresponding to the process of one or more application programs into the memory 1102 according to the following steps, and the processor 1101 will run the application program stored in the memory 1102 to execute the control position determination method. For the specific implementation of the method, please refer to the previous embodiments, which will not be repeated here.

[0238] Optional, such as Figure 12 As shown, the remote control device 1100 also includes: a touch screen display 1103, a radio frequency circuit 1104, an audio circuit 1105, an input unit 1106, and a power supply 1107. The processor 1101 is electrically connected to the touch screen display 1103, the radio frequency circuit 1104, the audio circuit 1105, the input unit 1106, and the power supply 1107. Those skilled in the art will understand that... Figure 12 The structure of the remote control device shown does not constitute a limitation on the remote control device. It may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0239] The touch display screen 1103 can be used to display a graphical user interface and receive operation instructions generated by user acting on the graphical user interface. The touch display screen 1103 can include a display panel and a touch panel. The display panel can be used to display information input by the user or information provided to the user and various graphical user interfaces of the remote control device, which can be composed of graphics, text, icons, videos and any combination thereof. Optionally, the display panel can be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. The touch panel can be used to collect touch operations (such as user operations on or near the touch panel using a finger, a stylus or any suitable object or accessory) of the user thereon or therearound, and generate corresponding operation instructions, and the operation instructions execute corresponding programs. Optionally, the touch panel can include two parts of a touch detection device and a touch controller. The touch detection device detects the touch position of the user and detects signals generated by the touch operation, and transmits the signals to the touch controller; the touch controller receives the touch information from the touch detection device, and converts it into touch coordinates, and then sends it to the processor 1101, and can receive commands from the processor 1101 and execute them. The touch panel can cover the display panel, and when the touch panel detects a touch operation thereon or therearound, it transmits to the processor 1101 to determine the type of touch event, and then the processor 1101 provides corresponding visual output on the display panel according to the type of touch event. In the embodiments of the present application, the touch panel and the display panel can be integrated into the touch display screen 1103 to realize the input and output functions. However, in some embodiments, the touch panel and the touch panel can realize the input and output functions as two independent components. That is, the touch display screen 1103 can also realize the input function as part of the input unit 1106.

[0240] The radio frequency circuit 1104 can be used to transceive radio frequency signals to establish wireless communication with network devices or other remote control devices, and transceive signals between network devices or other remote control devices.

[0241] The audio circuit 1105 can be used to provide an audio interface between a user and the remote control device through a speaker and a microphone. The audio circuit 1105 can convert received audio data into an electrical signal and transmit the electrical signal to the speaker for conversion into an audible signal. On the other hand, the microphone can collect a sound signal and convert the sound signal into an electrical signal, which is received by the audio circuit 1105 and converted into audio data. The audio data can be output to the processor 1101 for processing, transmitted to another remote control device via the radio frequency circuit 1104, or output to the memory 1102 for further processing. The audio circuit 1105 can also include a headphone jack to provide communication between an external device and the remote control device.

[0242] The input unit 1106 can be used to receive inputted digital, character information or user feature information (e.g., fingerprint, iris, face information, etc.), and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0243] The power supply 1107 is used to supply power to various components of the remote control device 1100. Optionally, the power supply 1107 can be logically connected to the processor 1101 via a power management device, so that the power management device can be used to manage charging, discharging, power consumption management, etc. The power supply 1107 can also include one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, etc.

[0244] Although Figure 12 The remote control device 1100 can also include a camera, a sensor, a wireless fidelity module, a Bluetooth module, etc., which are not shown in the embodiments.

[0245] In the above embodiments, the description of each embodiment focuses on different aspects, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0246] Those skilled in the art can understand that all or part of the steps of the various methods in the above embodiments can be completed by a computer program, or by a computer program controlling related hardware, which can be stored in a computer readable storage medium and loaded and executed by a processor.

[0247] To this end, an embodiment of the present application provides a computer readable storage medium, which stores a plurality of computer programs. The computer programs can be loaded by a processor to execute any of the control site determination methods provided by the embodiments of the present application. The computer programs can perform the specific implementation of the control site determination method, which can be referred to the previous embodiments and will not be described here.

[0248] The computer readable storage medium can include a read only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0249] Due to the computer program stored in the computer readable storage medium, any control site determination method provided by the embodiments of the present application can be executed, thus achieving the beneficial effects of any control site determination method provided by the embodiments of the present application. Details are described above, and thus will not be repeated here.

[0250] According to an aspect of the present application, a computer program product or computer program is also provided, which includes computer instructions stored in a computer readable storage medium. A processor of a remote control device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the remote control device to perform the method provided in various optional implementation manners in the above embodiments.

[0251] In the above remote control device positioning apparatus, computer readable storage medium, remote control device, and computer program product embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments. Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described remote control device positioning apparatus, computer readable storage medium, computer program product, remote control device, and corresponding units thereof, and the beneficial effects brought by the specific working process can be referred to the description of the control site determination method in the above embodiments, and thus will not be repeated here.

[0252] The above provides a detailed description of a control site determination method, device, apparatus, remote control device, computer readable storage medium, and computer program product provided by the embodiments of the present application. The principles and implementation manners of the present application are described by applying specific examples, and the above embodiment descriptions are only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, the specific implementation manners and application ranges will be changed according to the idea of the present application, and thus the content of the present description should not be understood as a limitation of the present application.

Claims

1. A control point determination method, characterized by, The method is applied to a remote control device for emitting laser to a display device to display a control point for indication on a display screen of the display device, the remote control device comprising a shooting device and an inertial measurement unit, a conversion relationship between a camera coordinate system of the shooting device and an inertial measurement unit coordinate system of the inertial measurement unit being calibrated in advance, and the method comprising: acquiring a target image collected by the shooting device on the display screen; the display screen comprising a plurality of preset calibration light sources; extracting a plurality of target feature regions comprising the calibration light sources from the target image; when the number of the extracted target feature regions is greater than or equal to the number of the calibration light sources, determining first attitude information of the shooting device according to first position information of the target feature regions, second position information of the calibration light sources and first motion state information of the inertial measurement unit in the remote control device; determining a control point of a laser point displayed in the display screen by the laser emitted by the remote control device to the display device based on the first attitude information of the shooting device and a plane where the display screen is located in a world coordinate system based on the calibration light sources; wherein the first position information is position information of the target feature regions in a camera coordinate system corresponding to the target image; the second position information is position information of the calibration light sources in the world coordinate system; the first motion state information comprises motion information and attitude information of the inertial measurement unit in an inertial measurement unit coordinate system; and the first attitude information represents a conversion relationship between the camera coordinate system of the shooting device and the world coordinate system.

2. The control site determination method of claim 1, wherein, The determination of the first attitude information of the shooting device according to the first position information of the target feature regions, the second position information of the calibration light sources and the first motion state information of the inertial measurement unit in the remote control device comprises: extracting second attitude information corresponding to the inertial measurement unit in the remote control device in the first motion state information, the second attitude information representing a conversion relationship between an inertial measurement unit coordinate system of the inertial measurement unit and the world coordinate system; sorting a plurality of the target feature regions to obtain a target feature region sequence; determining a target feature region subsequence set corresponding to the target feature region sequence according to a preset calibration light source sequence composed of a plurality of the calibration light sources after a preset sorting and an arrangement order of the target feature regions in the target feature region sequence, the target feature region subsequence set comprising a plurality of target feature region subsequences; determining first reference attitude information of the shooting device according to the target feature region subsequence set, the preset calibration light source sequence, the first position information and the second position information, the first reference attitude information representing a reference conversion relationship between the camera coordinate system of the shooting device and the world coordinate system; determining the first attitude information of the shooting device according to an external parameter between the inertial measurement unit and the shooting device, the first reference attitude information and the second attitude information.

3. The control site determination method of claim 2, wherein, The first reference attitude information of the photographing device is determined according to the target feature region subsequence set, the preset calibration light source sequence, the first position information and the second position information. The target feature region subsequence is matched with the preset calibration light source sequence for each target feature region subsequence in the target feature region subsequence set, so as to obtain a target feature region matching sequence set corresponding to the target feature region subsequence. The first reference attitude information of the photographing device is determined based on the target feature region matching sequence set, the first position information and the second position information for each target feature region subsequence.

4. The control site determination method of claim 2, wherein, The first attitude information of the photographing device is determined according to the extrinsic parameter between the inertial measurement unit and the photographing device, the first reference attitude information and the second attitude information. Each first reference attitude information is converted into second reference attitude information of the inertial measurement unit based on the extrinsic parameter between the inertial measurement unit and the photographing device. An error set is obtained by respectively calculating errors between each second reference attitude information and the second attitude information. The first attitude information of the photographing device is determined based on the error set.

5. The control site determination method of claim 4, wherein, The first attitude information of the photographing device is determined based on the error set. If there is an error less than or equal to a first preset error threshold in the error set, the second reference attitude information corresponding to the minimum error in the error set is determined as target second reference attitude information, and the first reference attitude information corresponding to the target second reference attitude information before coordinate conversion is determined as the first attitude information of the photographing device. If each error in the error set is greater than the first preset error threshold, the first attitude information of the photographing device is determined based on a continuous preset number of frames of the target image.

6. The control site determination method of claim 5, wherein, The first attitude information of the photographing device is determined based on a continuous preset number of frames of the target image. A continuous preset number of frames of the target image are acquired, and adjacent two frames of the target image are divided into an image group. For each image group, a target feature region in adjacent two frames of the target image in the image group is determined according to angular velocity measured by the inertial measurement unit corresponding to the image group and the extrinsic parameter between the inertial measurement unit and the photographing device. For each image group, first interframe reference attitude information of the photographing device corresponding to the image group is determined based on third position information of the target feature region in adjacent two frames of the target image and second position information of the calibration light source. The first attitude information of the photographing device is determined according to the first interframe reference attitude information and preset interframe reference attitude information of the inertial measurement unit. The third position information is position information of the target feature region in a camera coordinate system corresponding to the target image; the first inter-frame reference attitude information is a reference conversion relationship between the camera coordinate system and a world coordinate system when the photographing device photographs adjacent two frames of the target image; and the preset inter-frame reference attitude information is a preset conversion relationship between the IMU coordinate system and the world coordinate system when the IMU photographs adjacent two frames of the target image.

7. The control site determination method of claim 6, wherein, The first attitude information of the photographing device is determined according to the first inter-frame reference attitude information and the preset inter-frame reference attitude information of the IMU, and the method comprises the following steps: For each image group, the first inter-frame reference attitude information is converted into second inter-frame reference attitude information of the IMU based on the external parameter between the IMU and the photographing device, the second inter-frame reference attitude information being a reference conversion relationship between the IMU coordinate system and the world coordinate system when the photographing device photographs adjacent two frames of the target image; For each image group, a difference value between the second inter-frame reference attitude information and the corresponding preset inter-frame reference attitude information is calculated; Target inter-frame reference attitude information is obtained, and the first attitude information of the photographing device is determined according to the target inter-frame reference attitude information, the target inter-frame reference attitude information being the first inter-frame reference attitude information corresponding to the second inter-frame reference attitude information with the smallest difference value and less than or equal to a second preset error threshold.

8. The control site determination method of claim 1, wherein, The control point of the laser point displayed by the laser emitted by the remote control device on the display screen is determined based on the first attitude information of the photographing device and the plane in which the display screen is located in the world coordinate system determined based on the calibration light source, and the method comprises the following steps: The vertical axis direction of the camera coordinate system of the photographing device and the rotation matrix of the photographing device relative to the world coordinate system in the first attitude information are obtained; The vertical axis direction is rotated based on the rotation matrix to obtain the ray direction of the photographing device relative to the world coordinate system; The target ray is determined according to the ray direction and the position information of the photographing device; The intersection between the target ray and the plane in which the display screen is located in the world coordinate system determined based on the calibration light source is calculated to obtain the reference control point of the remote control device in the world coordinate system; The control point of the laser point displayed by the laser emitted by the remote control device on the display screen is determined according to the coordinates of the reference control point and the conversion matrix between the coordinate system of the display screen and the world coordinate system.

9. The control site determination method of claim 1, wherein, The method further comprises: When the number of the target feature regions is less than the number of the calibration light sources, first pose information of the photographing device is determined according to the first position information of the target feature regions, the second position information of the calibration light sources, camera parameters of the photographing device, and first motion state information of an inertial measurement unit in the remote control device; According to the first position information, the second position information, the first pose information, and camera calibration center coordinates of the photographing device, a control point of a laser point displayed on the display screen by laser emitted by the remote control device to the display device is determined; The camera calibration center coordinates are position information of a camera calibration center in the camera coordinate system, and the camera parameters are calibration parameters.

10. The control site determination method of claim 9, wherein, The first pose information of the photographing device is determined according to the first position information of the target feature regions, the second position information of the calibration light sources, camera parameters of the photographing device, and first motion state information of an inertial measurement unit in the remote control device, and includes: A gravity vector and an angular velocity measured by the inertial measurement unit in the first motion state information are extracted; According to the angular velocity and an external parameter between the inertial measurement unit and the photographing device, a rotation pose of the photographing device in a world coordinate system is determined; According to the first position information of the target feature regions, the camera parameters of the photographing device, the rotation pose, and the gravity vector, a rotation matrix of the photographing device is determined; According to the first position information of the target feature region 、 The second position information of the calibration light source, the camera parameters of the shooting device, and the rotation matrix determine a translation matrix of the shooting device. According to the translation matrix and the rotation matrix, the first pose information of the photographing device is determined.

11. The control site determination method of claim 9, wherein, The control point of the laser point displayed on the display screen by laser emitted by the remote control device to the display device is determined according to the first position information, the second position information, the first pose information, and the camera calibration center coordinates of the photographing device, and includes: According to the first position information, the second position information, and the first pose information, a target homography matrix is determined; According to the target homography matrix and the camera calibration center coordinates of the photographing device, a reference control point of the remote control device in the world coordinate system is determined; According to the coordinates of the reference control point and a conversion matrix between a coordinate system of the display screen and the world coordinate system, the control point of the laser point displayed on the display screen by laser emitted by the remote control device to the display device is determined.

12. The control site determination method of claim 1, wherein, The target feature regions containing the calibration light sources are extracted from the target image, and include: Feature region detection is performed on the target image to determine morphological information and size information of each feature region in the target image; According to the morphological information of each feature region, an aspect ratio of each feature region is determined; According to the size information of each feature region, a pixel number corresponding to each feature region is determined; A feature region with an aspect ratio less than a preset ratio and a pixel number greater than a preset value is determined as a target feature region containing the calibration light source in the target image; The target feature regions containing the calibration light sources are extracted from the target image.

13. A control point determination apparatus characterized by comprising: The control point determination apparatus comprises: an acquisition module, configured to acquire a target image collected by the photographing device on the display screen, wherein the display screen comprises a plurality of preset calibration light sources; an extraction module, configured to extract a plurality of target feature regions comprising the calibration light sources from the target image; a first determination module, configured to, when the number of the extracted target feature regions is greater than or equal to the number of the calibration light sources, determine first attitude information of the photographing device according to first position information of the target feature regions, second position information of the calibration light sources, and first motion state information of an inertial measurement unit in the remote control device; a second determination module, configured to determine a control point of a laser point displayed on the display screen by the laser emitted by the remote control device to the display device based on the first attitude information of the photographing device and a plane in which the display screen is located in a world coordinate system determined based on the calibration light sources.

14. An electronic device, comprising: The electronic device is configured with a processor and a memory, the memory stores a plurality of computer programs, and the processor loads the computer programs from the memory to execute the steps of the control point determination method according to any one of claims 1-12.

15. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a plurality of computer programs, and the computer programs are adapted to be loaded by the processor to execute the steps of the control point determination method according to any one of claims 1-12.

Images