Man-machine interaction method, device and equipment based on optical interaction equipment and storage medium
By partitioning and computing the unidirectional matrix of the projection plane in the optical interaction device, the complex positioning of laser spot feature points in the optical interaction device is solved, the response speed and accuracy of human-computer interaction are improved, and the requirements of real-time interaction are met.
Patent Information
- Application Number
- CN202411013047.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-11-19
AI Technical Summary
In real-time human-computer interaction applications, existing optical interaction devices cannot meet the real-time requirements of interaction due to the small area of laser spot feature points, and the precise positioning and calculation are complex and time-consuming, which affects the user experience.
By partitioning the projection plane, the homography matrix of each partition is calculated, and the feature point coordinates and homography matrix captured by the camera are used to convert feature points from the camera image coordinate system to the display screen coordinate system, reducing the detection and calibration calculation amount. Improve response speed and accuracy.
It realizes fast and accurate human-computer interaction response, improves the efficiency and accuracy of optical interaction devices in human-computer interaction, and meets the needs of real-time interaction.
Smart Images

Figure CN118981249A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of computer vision technology, and in particular to a human-computer interaction method, apparatus, device and storage medium based on an optical interaction device. Background Art
[0002] In the existing human-computer interaction technology, different interactive experiences are provided mainly by using graphical user interface, gesture recognition, virtual reality and other methods. In the scenario without fixed display device, optical interactive devices, such as laser projection equipment, can also be used to realize human-computer interaction.
[0003] The laser projection device emits a laser beam through a laser and forms a high-brightness light spot on the projection plane as a visible feature point. Since the laser spot on the projection plane is extremely small, the calculation process of accurately locating these small-area light spots is complex and time-consuming. However, in real-time human-computer interaction applications, the system needs to respond to user operations quickly and accurately. Such high computational complexity will slow down the detection and processing process, unable to meet the real-time requirements of interaction, thus affecting the user experience. Summary of the invention
[0004] Based on the above problems, the embodiments of the present application provide a human-computer interaction method, apparatus, device and storage medium based on an optical interaction device, with the aim of improving the response speed and accuracy of the optical interaction device in human-computer interaction.
[0005] In a first aspect, an embodiment of the present application provides a human-computer interaction method based on an optical interaction device, wherein the optical interaction device includes a camera, a projection module, and a host computer, and the method includes:
[0006] Acquire a first image of a projection plane captured by the camera, wherein the projection plane has a feature point to be detected;
[0007] Determine the coordinates of a first feature point in the first image in a camera image coordinate system;
[0008] Obtaining a homography matrix corresponding to each partition of the projection plane, and determining a target homography matrix corresponding to the partition where the feature point is located;
[0009] According to the first feature point coordinates and the target homography matrix, the second feature point coordinates of the feature point in the display screen coordinate system are determined, so that the host computer performs the interactive task according to the second feature point coordinates.
[0010] In one embodiment, determining the coordinates of the first feature point on the first image in the camera image coordinate system includes:
[0011] Binarizing the first image to obtain a mask image;
[0012] According to the mask image, first feature point coordinates of the feature point in a camera image coordinate system are determined.
[0013] In one embodiment, determining the first feature point coordinates of the feature point in the camera image coordinate system according to the mask image includes:
[0014] Counting the number of non-zero pixels at each position index in the horizontal direction and the vertical direction of the mask image respectively;
[0015] Determine the first feature point coordinates of the feature point in the camera image coordinate system according to the position index corresponding to the maximum value of the number of non-zero pixels in the horizontal direction and the vertical direction.
[0016] In one embodiment, determining the first feature point coordinates of the feature point in the camera image coordinate system according to the position index corresponding to the maximum value of the number of non-zero pixels in the horizontal direction and the vertical direction includes:
[0017] Determining the initial coordinates of the feature point in the camera image coordinate system according to the position index corresponding to the maximum value of the number of non-zero pixels in the horizontal direction and the vertical direction;
[0018] The intrinsic parameter matrix and the distortion coefficient of the camera are obtained, and the initial coordinates are corrected using the intrinsic parameter matrix and the distortion coefficient to obtain the first feature point coordinates of the feature point in the camera image coordinate system.
[0019] In one embodiment, before obtaining the homography matrix corresponding to each partition of the projection plane, the method further includes:
[0020] Projecting a preset calibration image onto the projection plane through the projection module;
[0021] The camera is controlled to shoot the projection plane to obtain a second image; wherein the preset calibration image and the second image both include a plurality of binary calibration codes and a checkerboard; the plurality of binary calibration codes of the preset calibration image divide the preset calibration image into a plurality of first partitions, and the plurality of binary calibration codes of the second image divide the projection plane in the second image into a plurality of second partitions; a checkerboard is provided in each of the first partitions and each of the second partitions; and each of the first partitions corresponds one-to-one to each of the second partitions.
[0022] In one embodiment, obtaining the homography matrix corresponding to each partition of the projection plane includes:
[0023] Obtaining the coordinates of the first corner points of each partition in the second image in the camera image coordinate system;
[0024] Obtain the coordinates of the second corner point of each corner point of each partition in the preset calibration image in the display screen coordinate system;
[0025] According to the first corner point coordinates and the second corner point coordinates of each corner point of each partition, a homography matrix corresponding to each partition is obtained.
[0026] In one embodiment, the method further comprises:
[0027] Determining a moving trajectory of the feature point according to a time series of coordinates of the second feature point;
[0028] Receive interaction information input by a user, and determine target presentation information according to the interaction information and the moving trajectory of the feature point.
[0029] In a second aspect, an embodiment of the present application further provides a human-computer interaction device based on an optical interaction device, wherein the optical interaction device includes a camera, a projection module, and a host computer, and the feature point calibration device based on the optical interaction device includes:
[0030] An image acquisition unit, used for acquiring a first image of a projection plane captured by the camera, wherein the projection plane has a feature point to be detected;
[0031] A feature point positioning unit, used to determine the first feature point coordinates of the feature point in the first image in the camera image coordinate system;
[0032] A processing unit, used to obtain a homography matrix corresponding to each partition of the projection plane, and determine a target homography matrix corresponding to the partition where the feature point is located;
[0033] The coordinate mapping unit is also used to determine the second feature point coordinates of the feature point in the display screen coordinate system according to the first feature point coordinates and the target homography matrix, so that the host computer performs the interactive task according to the second feature point coordinates.
[0034] In a third aspect, an embodiment of the present application further provides an optical interaction device, the optical interaction device comprising a laser projection device, a camera, a projection module and a host computer;
[0035] The laser projection device is used to emit a laser beam to the projection plane; the laser beam forms a characteristic point on the projection plane;
[0036] The camera is used to image the projection plane to obtain image data;
[0037] The projection module is used to project a preset calibration image onto the projection plane;
[0038] The host computer is configured to execute any one of the above-mentioned human-computer interaction methods based on an optical interaction device;
[0039] The laser projection device includes a photoelectric button, which includes a touch panel, a beam emitting element and a beam sensing element; the beam emitting element is used to emit a laser beam to the touch panel; the beam sensing element is used to receive the laser beam reflected by the touch panel; the beam sensing element is configured to generate a button activation signal and trigger a custom event if the beam sensing element receives the laser beam reflected from the touch panel, so that the host computer responds to and executes the custom event.
[0040] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is run on a computer, the computer executes any one of the above-mentioned human-computer interaction methods based on an optical interaction device.
[0041] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:
[0042] The embodiment of the present application determines the coordinates of the first feature points in the first image in the camera image coordinate system by shooting the first image of the projection plane; obtains the homography matrix corresponding to each partition of the projection plane, and determines the target homography matrix corresponding to the partition where the feature points are located; uses the target homography matrix to transform the feature points from the camera image coordinate system to the display screen coordinate system, so that the host computer performs the corresponding interactive tasks according to the transformed coordinates. In this way, the coordinates of the feature points are first detected on the first image, and then the coordinates of the first feature points are transformed from the camera image coordinate system to the display screen coordinate system using the homography matrix to obtain the transformed coordinates of the second feature points, which can reduce the calculation amount and processing time of detecting and calibrating feature points, and improve the response speed of human-computer interaction. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0044] Figure 1 A schematic diagram of a human-computer interaction method based on an optical interaction device provided in an embodiment of the present application;
[0045] Figure 2 A schematic diagram of feature point calibration on a projection plane provided in an embodiment of the present application;
[0046] Figure 3 A schematic diagram of a preset calibration image provided in an embodiment of the present application;
[0047] Figure 4 A schematic diagram of the structure of a human-computer interaction device based on an optical interaction device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0048] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0049] In the existing human-computer interaction technology, different interactive experiences are provided mainly by using graphical user interface, gesture recognition, virtual reality and other methods. In the scenario without fixed display device, optical interactive devices, such as laser projection equipment, can also be used to realize human-computer interaction.
[0050] The laser projection device emits a laser beam through a laser and forms a high-brightness light spot on the projection plane as a visible feature point. The sensor (such as a camera or photodetector) in the optical interaction device captures the position changes of the feature points on the projection plane in real time, and feeds back the captured position changes of the feature points to the controller. The controller performs interactive tasks (such as clicking, dragging, etc.) according to the preset algorithm program. Since the area of the laser spot feature points on the projection plane is extremely small, the calculation process of accurately locating these small-area spots is complex and time-consuming. However, in real-time human-computer interaction applications, the system needs to respond to user operations quickly and accurately. Such high computational complexity will cause the detection and processing process to slow down, and cannot meet the real-time requirements of the interaction, thus affecting the user experience.
[0051] Based on this, in each embodiment of the present application, the projection plane is partitioned, and the homography matrix is calculated for each partition; by obtaining the coordinates of the feature points in the image of the projection plane taken by the camera, and the homography matrix of the partition where the feature points are located, the coordinates of the feature points in the display screen coordinate system are obtained using the homography matrix, so that the host computer performs the corresponding interactive tasks according to the converted coordinates.
[0052] The various embodiments of the present application are described in further detail below in conjunction with the accompanying drawings.
[0053] The embodiment of the present application provides a human-computer interaction method based on an optical interaction device, wherein the optical interaction device includes a camera, a projection module and a host computer, such as Figure 1 As shown, the method includes steps S101-S104.
[0054] S101: Acquire a first image of a projection plane captured by a camera, wherein the projection plane has feature points to be detected;
[0055] The projection plane is photographed using a camera, and the photographed image is obtained as a first image, wherein the first image includes one or more feature points on the projection plane that need to be detected and located. Feature points refer to target points on the projection plane for interaction, and feature points need to be detected and located during the optical interaction process to perform interactive tasks. Feature points include high-brightness light spots formed on the projection plane by a laser projection device. The projection plane is the surface on which the laser projection device projects a light beam, and specifically can be any surface that can display a laser light spot, such as a wall, a whiteboard, a screen, a desktop, or any surface suitable for projection. The camera that photographs the projection plane can be an independent camera device or a camera integrated in the optical interaction device. There is an active filter in front of the camera lens that can filter out interference from non-infrared light. The filter is removed when the homography matrix of each partition of the projection plane is obtained or calculated, and is loaded when the feature point (such as a light spot) is detected. Exemplarily, when an infrared laser emits an infrared laser and forms a light spot on the projection plane, the projection plane is photographed by a camera to obtain a first image to be detected.
[0056] S102: Determine the coordinates of a first feature point in a first image in a camera image coordinate system;
[0057] The camera image coordinate system is a two-dimensional coordinate system that describes and locates the position of each pixel in the image captured by the camera. The coordinate system in the image captured by the camera is used to represent the pixel position in the image. The origin of the camera image coordinate system is usually in the upper left corner of the image, that is, the x-axis is from left to right in the horizontal direction, and the y-axis is from top to bottom in the vertical direction. To determine the coordinates of the feature points in the image in the camera image coordinate system, the edges of the feature points can be detected by an edge detection algorithm, and then the center coordinates of the feature points can be determined. Alternatively, the Hough transform method can be used to detect feature points of a specific shape (such as a circle) and determine their position coordinates. The edge detection algorithm and the Hough transform method are prior art and will not be described in detail here.
[0058] S103: Obtain the homography matrix corresponding to each partition of the projection plane, and determine the target homography matrix corresponding to the partition where the feature point is located;
[0059] The homography matrix can be understood as the coordinate mapping relationship between the display screen coordinate system and the camera image coordinate system. The coordinates in the camera image coordinate system can be converted to the coordinates in the display screen coordinate system using the homography matrix. In traditional methods, the homography matrix is usually calculated using the four corner points of the projection plane or by projecting a single chessboard on the projection plane. However, the homography matrix obtained in this way is usually only applicable to small-sized projection planes or scenes where the relative position of the camera and the projection plane is fixed and does not change much. In actual application scenarios, if the size of the projection plane is large, the perspective distortion in different areas may be significantly different. In addition, if the projection plane is a complex or irregular surface (such as a curved surface or a plane with local deformation), the homography matrix calculated using only the four corner points of the projection plane or a single chessboard may not be able to adapt to these changes.
[0060] In order to solve this problem and to ensure that the image captured by the camera is aligned with the display content on the host screen, the embodiment of the present application divides the projection plane into multiple partitions and independently calculates the homography matrix for each partition. In this way, the deformation and perspective distortion of each local area can be accurately processed, ensuring that the image displayed on the screen is consistent with the geometric relationship of the actual scene, thereby improving the accuracy of image correction.
[0061] Determining the partition where the feature point is located may specifically include the following methods:
[0062] (1) Divide the projection plane into several regular grids, and set one or more grids to correspond to a partition. By calculating the coordinates of the feature point and determining the grid in which the feature point falls, the partition in which the feature point is located can be determined.
[0063] (2) Pre-set a number of marking points on the projection plane, and determine the partition where the feature point is located by calculating the relative position between the feature point and these marking points.
[0064] The partition where the feature point is located can be determined based on actual needs and scene characteristics, and the embodiments of the present application do not limit this.
[0065] S104: Determine the coordinates of the second feature points of the feature points in the display screen coordinate system according to the coordinates of the first feature points and the target homography matrix, so that the host computer performs the interactive task according to the coordinates of the second feature points.
[0066] The display screen coordinate system is a coordinate system used to locate and render images or graphic elements on a display device (such as a computer monitor, TV screen, or mobile phone screen). The second feature point coordinates are the feature point coordinates transformed by the target homography matrix. The mapping relationship between the coordinates in the camera image coordinate system and the coordinates in the display screen coordinate system is as follows:
[0067]
[0068] in, is the homogeneous coordinate of the point in the camera image coordinate system. x and y are the horizontal and vertical coordinates of the feature point in the camera image coordinate system, respectively. 1 is the conventional representation in homogeneous coordinates, which is used to support the calculation of perspective transformation. i is the homography matrix corresponding to the i-th partition, H is the homogeneous coordinate of the feature point in the display screen coordinate system. u and v are the horizontal and vertical coordinates of the feature point in the display screen coordinate system, respectively. The last 1 is also the conventional representation in homogeneous coordinates. Specifically, H i It is a 3*3 matrix.
[0069] The host computer (or "master computer") performs corresponding interactive tasks according to the coordinates of the second feature point. These interactive tasks may include interactive operations such as clicking, handwriting, and annotation. The specific tasks depend on the application scenario and the corresponding preset algorithm, and the embodiment of the present application does not limit this.
[0070] Compared with the prior art, the whole image needs to be transformed in perspective first, that is, each pixel in the whole image needs to be transformed to a new position according to the homography matrix, and then the coordinates of the feature points are found from the image after the perspective transformation. This processing method is not only time-consuming, but also requires a lot of computing resources, especially when the image resolution is high, the processing time will increase significantly. The embodiment of the present application obtains the first image of the projection plane taken by the camera, determines the first feature point coordinates of the feature points in the first image in the camera image coordinate system; obtains the homography matrix corresponding to each partition of the projection plane, and determines the target homography matrix corresponding to the partition where the feature points are located; uses the homography matrix to convert the feature points from the camera coordinate system to the display screen coordinate system, so that the host computer performs the corresponding interactive tasks according to the converted coordinates. The fast spot detection algorithm designed in this way can reduce the computational complexity and improve the efficiency of detecting and calibrating feature points. At the same time, by dividing the projection plane into multiple partitions, the homography matrix of each area can be calculated more accurately, realizing accurate mapping and efficient interactive operation from the physical projection plane to the display screen.
[0071] To make the image processing steps simpler and more efficient, in one embodiment, determining the first feature point coordinates of a feature point on a first image in a camera image coordinate system includes: binarizing the first image to obtain a mask image; and determining the first feature point coordinates of the feature point in the camera image coordinate system based on the mask image.
[0072] The binarization process converts the first image into an image with only black and white colors (i.e., a binary image, also called a mask image). Under the binarization process, each pixel in the image is divided into black and white (black usually represents the background, and white usually represents the foreground or the target object of interest). Through a pre-set threshold, all pixels above the preset threshold are set to white, and pixels below the preset threshold are set to black, so that the feature points can be separated from the background that is not of concern.
[0073] In the mask image, image processing algorithms (such as morphological operations, contour detection, etc.) can be further used to identify white areas, which represent feature points. In a feasible embodiment, the first feature point coordinates of the feature points in the camera image coordinate system can be obtained by calculating the center points or centroids of these areas.
[0074] Binarization processing can filter out a lot of noise information, effectively separate the feature points in the image from the background, and improve the accuracy of feature point recognition. It can also reduce the complexity of data processing and improve the overall computing efficiency.
[0075] In order to quickly and accurately determine the position of the feature point in the image, in one embodiment, the first feature point coordinates of the feature point in the camera image coordinate system are determined based on the mask image, including: counting the number of non-zero pixels of each position index in the horizontal and vertical directions of the mask image respectively; and determining the first feature point coordinates of the feature point in the camera image coordinate system based on the position index corresponding to the maximum value of the number of non-zero pixels in the horizontal and vertical directions.
[0076] For scenes where a laser beam forms a feature point, the mask image is usually an image with a value at the infrared spot and zero at other locations. Figure 2 , Figure 2 It is a calibration diagram on the projection plane, which contains the feature points to be detected. It can be seen that Figure 2 There is an obvious white dot in the middle upper right corner. The method to determine the coordinates of the white dot position can be: for each row in the horizontal direction of the mask image (i.e. along the X-axis direction), count the number of all non-zero (i.e. white) pixels in each row, find the row with the largest number of non-zero pixels in the horizontal direction, and record the row index (X coordinate) of the row; and for each column in the vertical direction of the mask image (i.e. along the Y-axis direction), count the number of all non-zero (i.e. white) pixels in each column, find the column with the largest number of non-zero pixels in the vertical direction, and record the column index (Y coordinate) of the column. According to the row index and column index obtained in the above steps, determine the coordinates of the first feature point of the feature point in the camera image coordinate system.
[0077] By counting the number of non-zero pixels, the location of feature points in the image can be quickly and accurately determined, and because noise usually does not form high-density pixel clusters, some random noise can also be effectively filtered out.
[0078] In order to eliminate the image distortion caused by the camera lens and make the obtained feature point coordinates more accurate, in one embodiment, the first feature point coordinates of the feature point in the camera image coordinate system are determined according to the position index corresponding to the maximum value of the number of non-zero pixels in the horizontal and vertical directions, including: determining the initial coordinates of the feature point in the camera image coordinate system according to the position index corresponding to the maximum value of the number of non-zero pixels in the horizontal and vertical directions; obtaining the camera's intrinsic parameter matrix and distortion coefficient, and using the intrinsic parameter matrix and distortion coefficient to correct the initial coordinates to obtain the first feature point coordinates of the feature point in the camera image coordinate system.
[0079] According to the above embodiment, the method for confirming the position coordinates of the feature point may include determining the initial coordinates (x, y) of the feature point in the camera image coordinate system by using the position index corresponding to the maximum number of non-zero pixels in the horizontal direction and the position index corresponding to the maximum number of non-zero pixels in the vertical direction. Furthermore, the initial coordinates (x, y) may be corrected using the camera's intrinsic parameter matrix and distortion coefficient.
[0080] Specifically, the intrinsic parameter matrix contains the internal parameters of the camera, such as focal length, optical center, etc. These parameters describe the geometric relationship in the camera imaging process and are used to project points in three-dimensional space onto the two-dimensional image plane. The distortion coefficient describes the distortion effect of the lens, including radial distortion and tangential distortion. Distortion causes straight lines in the image to become curves, especially in the edge areas of the image. Before applying distortion correction, the initial image coordinates must first be normalized to the camera image coordinate system. Assuming (x, y) is the initial coordinate, the normalization process is:
[0081]
[0082] Among them, x norm and norm is the coordinate of the feature point after normalization to the camera image coordinate system, f x and f y is the focal length parameter in the intrinsic parameter matrix, c x and c y is the optical center coordinate in the camera intrinsic matrix.
[0083] Then radial and tangential distortion correction is performed using the following formulas:
[0084] 1) Radial distortion correction:
[0085] r 2 =x 2 norm +y 2 norm
[0086] x dist1 =x norm (1+k1r 2 +k2r 4 +k3r 6 )
[0087] y dist1 =y norm (1+k1r 2 +k2r 4 +k3r 6 )
[0088] 2) Tangential distortion correction:
[0089] x dist2 =x dist1 +2p1x norm y norm +p2(r 2 +2x 2 norm )
[0090] y dist2 =y dist1 +p1(r 2 +2y 2 norm )+2p2x norm y norm
[0091] The coordinates after distortion correction need to be remapped back to the image coordinate system:
[0092] x final =f x *x dist2 +c x
[0093] y final =f y *y dist2 +c y
[0094] Where r is the distance from the optical center of the image (c x , c y ) to a given pixel point, x dist1 and dist1 is the coordinate after applying the radial distortion coefficient, k1, k2, k3 are the radial distortion coefficients, x dist2 anddist2 are the coordinates after applying the tangential distortion coefficients, p1 and p2 are the tangential distortion coefficients, x final and final It is the final coordinate remapped to the camera image coordinate system after distortion correction, that is, the coordinate of the first feature point.
[0095] By applying the above camera intrinsic parameter matrix and distortion coefficient to the feature point coordinates, the accuracy of the feature point coordinates can be significantly improved, especially when the feature point is located at the edge of the image or a wide-angle lens is used, the image distortion caused by the lens can be eliminated to obtain more accurate first feature point coordinates.
[0096] In order to reduce the amount of calculation and processing time during human-computer interaction, in one embodiment, before obtaining the homography matrix corresponding to each partition of the projection plane, it also includes: projecting a preset calibration image to the projection plane through a projection module; controlling the camera to shoot the projection plane to obtain a second image; wherein the preset calibration image and the second image both include multiple binary calibration codes and a checkerboard; the multiple binary calibration codes of the preset calibration image divide the preset calibration image into multiple first partitions, and the multiple binary calibration codes of the second image divide the projection plane in the second image into multiple second partitions; each first partition and each second partition are provided with a checkerboard; each first partition corresponds one-to-one to each second partition.
[0097] For example, please refer to Figure 3 , Figure 3 is a schematic diagram of a preset calibration image. Figure 3 There are 4 binary calibration codes in the image, which divide the entire chessboard image into 7 partitions. Each partition has a chessboard. The homography matrix of each partition can be H i (i=1,2,3,4,5,6,7). In practical applications, the chessboard can cover the entire partition, or Figure 3 As shown, there is a certain blank area, through which different checkerboard patterns are separated to increase the contrast between different checkerboard areas, thereby improving the accuracy of feature point detection and matching; at the same time, it can also reduce the number of feature points that need to be processed in the image, thereby improving the efficiency of image processing.
[0098] The binary calibration code may be an ArUco code, a Quick Response Code (QR Code), an origin grid, or the like. Figure 3The binary calibration code in is Aruco code, which is a grid diagram based on Hamming code. ArUco code uses Hamming distance to design identifiers. Each ArUco marker consists of a black border and an internal binary matrix. The black border is used to improve the accuracy of detection, while the internal binary matrix encodes a unique identifier (ID, Identity), so each ArUco code has an ID corresponding to it. Figure 3 For example, from left to right, from top to bottom, the image ArUco code ID numbers are 0, 1, 3, and 2, respectively. It should be noted that the numbering order can be set in the research and development design stage, and the specific numbering order can be set as needed, which is not limited in the embodiments of the present application. Furthermore, when determining the range occupied by each partition, taking the local chessboard in the upper left corner of the preset calibration image as an example, the partition occupied by the local chessboard in the upper left corner includes the area enclosed by the corner point coordinates of the upper left corner of the preset calibration image and the corner point coordinates of the lower right corner of the ArUco code No. 0.
[0099] In the present application embodiment, checkerboard and ArUco code are used as detection points for determining the homography matrix, with high contrast and clear position in the image, which can be accurately detected and identified, thereby ensuring that the calculated homography matrix has high precision. In practical applications, the drawDetectedMarkers function provided in the OpenCV library can be used to detect the ID of the ArUco code and the coordinates of the four vertices of the ArUco code. It will be appreciated by those skilled in the art that the steps of detecting the ID of the ArUco code and the coordinates of the four vertices of the ArUco code can also be realized when selecting other image processing methods.
[0100] In addition, it can be understood that the second image is an image captured by the camera on the projection plane, and what is displayed on the projection plane is a preset calibration image projected by the projection module. Since the preset calibration image itself contains multiple binary calibration codes and checkerboards, the second image captured by the camera will also contain multiple binary calibration codes and checkerboards with the same layout as the preset calibration image. Therefore, these binary calibration codes and checkerboards also divide the second image into multiple partitions. From a practical application perspective, it can also be understood that these binary calibration codes and checkerboards divide the projection plane into multiple partitions, so as to calculate the local homography matrix for different partitions of the projection plane.
[0101] In the preset calibration image, the arrangement of the binary calibration code and the chessboard defines the position of each first partition. In the second image, by identifying the position of the calibration code and the chessboard, the position of each second partition can be determined. For example, computer vision technology (such as OpenCV's findChessboardCorners function) can be used to identify the binary calibration code and the chessboard. By identifying the binary calibration code ID and position of the preset calibration image and the second image, the first partition in the preset calibration image can be matched and mapped with the second partition in the second image.
[0102] By using the binary calibration codes and checkerboards in the preset calibration image and identifying these binary calibration codes and checkerboards in the second image, the partitions in the projection plane and the image can be effectively divided and matched. This not only simplifies the partition matching process, but also improves the accuracy of calibration and correction of the coordinates of the first feature point, thereby improving the accuracy of feature point calibration, as well as the real-time performance and user experience during the interaction process.
[0103] It should be noted that the selection Figure 3 The preset calibration image shown is only an example. In specific implementation, other binary calibration codes can be selected to divide the checkerboard image into different types, and the number of selected binary calibration codes is not limited. Any local mapping of multiple checkerboard partitions (that is, using different numbers or different layouts of checkerboards and calibration codes to achieve partitioned local mapping of the projection plane or image) should be included in the protection scope of the present invention.
[0104] In order to avoid errors caused by using a globally unified homography matrix, in one embodiment, the homography matrix corresponding to each partition of the projection plane is obtained, including: obtaining the first corner point coordinates of each corner point of each partition in the second image in the camera image coordinate system; obtaining the second corner point coordinates of each corner point of each partition in the preset calibration image in the display screen coordinate system; and obtaining the homography matrix corresponding to each partition according to the first corner point coordinates and the second corner point coordinates of each corner point of each partition.
[0105] In the preset calibration image, the corner point position is usually the corner point coordinates of each partition preset in the research and development design stage. Of course, the preset calibration image can also be used to identify the corner point coordinates in the display screen coordinate system. In the second image, the checkerboard and ArUco code are used to identify and locate the corner points of each partition from the second image. Then, the first corner point coordinates in the camera image coordinate system and the second corner point coordinates in the display screen coordinate system are used to calculate the homography matrix of each partition. The homography matrix is a 3*3 matrix that describes the perspective transformation between two planes, which can be calculated by the findHomography function in OpenCV. It is easy for those skilled in the art to understand that when other specific methods for calculating the homography matrix are selected, the homography matrix of each partition can also be calculated.
[0106] The traditional overall image calibration method may not fully consider the local characteristics of the projection surface, resulting in a large overall error, especially in non-planar or large-scale projection scenes. Therefore, the traditional method requires unified processing of the entire projection plane, which has high computational complexity and low efficiency, and is difficult to meet the needs of real-time interaction. However, the embodiment of the present application calculates the homography matrix of each partition separately through partition processing, which can effectively reduce the overall error caused by factors such as uneven projection surface or translucency distortion, thereby improving the accuracy of image correction.
[0107] In order to achieve more intelligent and flexible interactive functions, in one embodiment, it also includes: determining the movement trajectory of the feature point based on the time series of the second feature point coordinates; receiving the interactive information input by the user, and determining the target presentation information based on the interactive information and the movement trajectory of the feature point.
[0108] Here, the time series of the coordinates of the second feature points represents the continuous data records of these second feature point coordinates at different time points. According to the time series of the coordinates of the second feature points, the moving trajectory of the feature points on the projection plane can be determined. It can be understood that the moving trajectory refers to the path of the feature points moving on the projection plane within a period of time. Specifically, if the interactive information input by the user is handwriting input, the target presentation information may be to project the corresponding lines in the projection plane according to the moving path of the feature points to simulate the handwriting input. If the interactive information input by the user is a annotation, and the user uses a laser to draw a preset pattern (such as a circle) on the projection plane for the text part projected on the projection plane, the target presentation information may be to highlight the text part enclosed by the circle, or to display an annotation panel in the area surrounding the drawn circle.
[0109] In some other possible embodiments, the user's interaction intention can also be identified based on the moving trajectory of the feature point. For example, in a specified area of the projection plane, the moving trajectory of the feature point moves a certain distance from bottom to top in the vertical direction, and when the moving speed exceeds a certain threshold, the host computer identifies the user's interaction intention as scrolling down based on the characteristics of this moving trajectory. At this time, the projected screen can be scrolled down. The specific method of judging the interactive task based on the moving trajectory can be determined according to the actual needs of the business, and the embodiments of the present application are not limited to this. In this way, a variety of interactive tasks can be implemented based on the light spot trajectory projected by a laser pen, laser mouse or other laser device, thereby improving the user experience and the intelligence of the system.
[0110] Furthermore, the laser projection device provided in the embodiment of the present application may also have a touch button. When the user clicks or presses the touch button, a custom event will be triggered, such as clicking to confirm a selection, turning pages, and other operations. The host computer will respond to the user's touch operation in real time and execute the corresponding custom event, thereby realizing rapid processing and feedback of user instructions. Through this design, the laser projection device not only has higher interactivity and intelligence, but also can meet the user's demand for rapid response of optoelectronic interactive devices, thereby improving user experience and operating efficiency.
[0111] The design of touch buttons can be flexible and diverse. For example, touch buttons can be physical buttons or virtual buttons. Physical buttons can be mechanical buttons, capacitive buttons, or photoelectric buttons, while virtual buttons can be implemented through touch screens or projection interfaces. The location and function of touch buttons can also be customized according to actual application requirements to adapt to different usage scenarios. In addition, touch buttons can correspond to different custom events based on the frequency and duration of the click / press action. For example, a short press on a touch button can trigger a "confirm selection" event, while a long press on a touch button can trigger a "page turn" event. In this way, users can achieve rich functions through different touch operations, improving the flexibility and ease of use of the device.
[0112] Furthermore, based on the homography matrix corresponding to each partition of the projection plane calculated in the above embodiment, the optical interaction system in the optical interaction device saves the homography matrix corresponding to each partition as a conversion parameter. When the subsequent projection module projects other images, the conversion parameters can be directly used to perform trapezoidal correction on the image to restore the projected image to a regular rectangle.
[0113] In order to implement the human-computer interaction method based on the optical interaction device of the embodiment of the present application, the embodiment of the present application also provides a human-computer interaction device based on the optical interaction device, wherein the optical interaction device includes a camera, a projection module and a host computer, such as Figure 4 As shown, the device comprises:
[0114] An image acquisition unit 401 is used to acquire a first image of a projection plane captured by the camera, wherein the projection plane has feature points to be detected;
[0115] A feature point positioning unit 402, used to determine first feature point coordinates of a feature point in the first image in a camera image coordinate system;
[0116] The processing unit 403 is used to obtain a homography matrix corresponding to each partition of the projection plane, and determine a target homography matrix corresponding to the partition where the feature point is located;
[0117] The coordinate mapping unit 404 is further used to determine the second feature point coordinates of the feature point in the display screen coordinate system according to the first feature point coordinates and the target homography matrix, so that the host computer performs the interactive task according to the second feature point coordinates.
[0118] In one embodiment, the processing unit 403 is further configured to perform binarization processing on the first image to obtain a mask image;
[0119] The feature point positioning unit 402 is further configured to determine the first feature point coordinates of the feature point in the camera image coordinate system according to the mask image.
[0120] In one embodiment, the pixel counting unit is used to count the number of non-zero pixels of each position index in the horizontal direction and the vertical direction of the mask image respectively;
[0121] The feature point positioning unit 402 is used to determine the first feature point coordinates of the feature point in the camera image coordinate system according to the position index corresponding to the maximum value of the number of non-zero pixels in the horizontal direction and the vertical direction.
[0122] In one embodiment, the initial coordinate determining unit is used to determine the initial coordinates of the feature point in the camera image coordinate system according to the position index corresponding to the maximum value of the number of non-zero pixels in the horizontal direction and the vertical direction;
[0123] The coordinate correction unit is used to obtain the intrinsic parameter matrix and the distortion coefficient of the camera, and use the intrinsic parameter matrix and the distortion coefficient to correct the initial coordinates to obtain the first feature point coordinates of the feature point in the camera image coordinate system.
[0124] In one embodiment, an image projection unit is used to project a preset calibration image onto the projection plane through the projection module;
[0125] The image acquisition unit 401 is also used to control the camera to shoot the projection plane to obtain a second image; wherein the preset calibration image and the second image both include multiple binary calibration codes and a checkerboard; the multiple binary calibration codes of the preset calibration image divide the preset calibration image into multiple first partitions, and the multiple binary calibration codes of the second image divide the projection plane in the second image into multiple second partitions; each of the first partitions and each of the second partitions is provided with a checkerboard; each of the first partitions corresponds one-to-one to each of the second partitions.
[0126] In one embodiment, the corner point detection unit is used to obtain the first corner point coordinates of each corner point of each partition in the second image in the camera image coordinate system;
[0127] The corner point detection unit is further used to obtain the second corner point coordinates of each corner point of each partition in the preset calibration image in the display screen coordinate system;
[0128] The processing unit 403 is further configured to obtain a homography matrix corresponding to each partition according to the first corner point coordinates and the second corner point coordinates of each corner point of each partition.
[0129] In one embodiment, the trajectory tracking unit is used to determine the moving trajectory of the feature point according to the time series of the coordinates of the second feature point;
[0130] The processing unit 403 is further configured to receive interaction information input by the user, and determine target presentation information according to the interaction information and the moving trajectory of the feature point.
[0131] In actual application, the processing unit 403 can be realized by a processor in the optical interaction device combined with a communication interface, and the image acquisition unit 401, the feature point positioning unit 402, the coordinate mapping unit 404, the pixel statistics unit, the initial coordinate determination unit, the coordinate correction unit, the image projection unit, the corner point detection unit and the trajectory tracking unit can be realized by the communication interface in the human-computer interaction device based on the optical interaction device.
[0132] It should be noted that: the above embodiment provides a human-computer interaction device based on an optical interactive device, and only uses the division of the above program modules as an example when performing human-computer interaction. In actual applications, the above processing units can be assigned to different program modules as needed, that is, the internal structure of the device is divided into different program modules to complete all or part of the processing described above. In addition, the human-computer interaction device based on an optical interactive device provided in the above embodiment and the human-computer interaction method embodiment based on an optical interactive device belong to the same concept. The specific implementation process is detailed in the method embodiment, which will not be repeated here.
[0133] Based on the hardware implementation of the above program module, and in order to implement a human-computer interaction method based on an optical interaction device provided in an embodiment of the present application, an optical interaction device is further provided in an embodiment of the present application, and the optical interaction device includes a laser projection device, a camera, a projection module and a host computer;
[0134] The laser projection device is used to emit a laser beam to the projection plane; the laser beam forms a characteristic point on the projection plane;
[0135] The camera is used to image the projection plane to obtain image data;
[0136] The projection module is used to project a preset calibration image onto the projection plane;
[0137] The host computer is configured to execute any one of the above-mentioned human-computer interaction methods based on an optical interaction device;
[0138] The laser projection device includes a photoelectric button, which includes a touch panel, a beam emitting element and a beam sensing element; the beam emitting element is used to emit a laser beam to the touch panel; the beam sensing element is used to receive the laser beam reflected by the touch panel; the beam sensing element is configured to generate a button activation signal and trigger a custom event if the beam sensing element receives the laser beam reflected from the touch panel, so that the host computer responds to and executes the custom event.
[0139] In the above-mentioned embodiment, it is mentioned that the laser projection device can have a physical touch button, such as a mechanical button or a photoelectric button. Mechanical buttons can provide clear physical feedback and enhance the user's operating experience, but physical pressure is required when the user presses the button, which easily causes the position of the laser beam projected by the laser projection device to the projection plane to shake. Therefore, in a feasible embodiment, a photoelectric button can be used as a touch button. Specifically, the photoelectric button includes a touch panel, a beam emitting element and a beam sensing element, wherein the beam sensing element can be a photodiode. In actual applications, the touch panel may be made of a light-transmitting material, and the laser beam projected by the beam emitting element will pass through the touch panel, so the user's finger will cause the laser beam to be reflected when touching or approaching the touch panel. The beam sensing element receives the reflected laser beam, outputs a key activation signal and triggers a custom event, so that the host computer responds and executes the corresponding custom event. Custom events can be defined and configured according to specific application scenarios and user needs, for example, it can be to emit a laser beam to the projection plane, to confirm a certain operation, or to turn pages forward or backward, etc.
[0140] By adopting photoelectric buttons and eliminating the physical pressure required by mechanical buttons, the position jitter of the feature points to be detected can be avoided during the process of laser projection onto the projection plane using a laser projection device, thereby reducing the time and computing resources required for detecting and calibrating the feature points to be detected. It can also execute custom events more quickly and efficiently, and achieve fast response, real-time and efficient human-computer interaction.
[0141] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is run on a computer, any one of the above-mentioned human-computer interaction methods based on an optical interaction device is executed.
[0142] An embodiment of the present application further provides a computer program product having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, any one of the above-mentioned human-computer interaction methods based on an optical interaction device is executed.
[0143] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0144] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
[0145] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0146] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0147] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.
Claims
1. A human-computer interaction method based on an optical interaction device, characterized in that: The optical interaction device includes a camera, a projection module and a host computer, and the method includes: Acquire a first image of a projection plane captured by the camera, wherein the projection plane has a feature point to be detected; Determine the coordinates of a first feature point in the first image in a camera image coordinate system; Obtaining a homography matrix corresponding to each partition of the projection plane, and determining a target homography matrix corresponding to the partition where the feature point is located; According to the first feature point coordinates and the target homography matrix, the second feature point coordinates of the feature point in the display screen coordinate system are determined, so that the host computer performs the interactive task according to the second feature point coordinates.
2. The human-computer interaction method based on an optical interaction device according to claim 1, characterized in that: The determining the first feature point coordinates of the feature point on the first image in the camera image coordinate system includes: Binarizing the first image to obtain a mask image; According to the mask image, first feature point coordinates of the feature point in a camera image coordinate system are determined.
3. The human-computer interaction method based on an optical interaction device according to claim 2, characterized in that: The determining, according to the mask image, the first feature point coordinates of the feature point in the camera image coordinate system comprises: Counting the number of non-zero pixels at each position index in the horizontal direction and the vertical direction of the mask image respectively; Determine the first feature point coordinates of the feature point in the camera image coordinate system according to the position index corresponding to the maximum value of the number of non-zero pixels in the horizontal direction and the vertical direction.
4. The human-computer interaction method based on an optical interaction device according to claim 3, characterized in that: The determining, according to the position index corresponding to the maximum value of the number of non-zero pixels in the horizontal direction and the vertical direction, the first feature point coordinate of the feature point in the camera image coordinate system comprises: Determining the initial coordinates of the feature point in the camera image coordinate system according to the position index corresponding to the maximum value of the number of non-zero pixels in the horizontal direction and the vertical direction; The intrinsic parameter matrix and the distortion coefficient of the camera are obtained, and the initial coordinates are corrected using the intrinsic parameter matrix and the distortion coefficient to obtain the first feature point coordinates of the feature point in the camera image coordinate system.
5. The human-computer interaction method based on an optical interaction device according to claim 1, characterized in that: Before obtaining the homography matrix corresponding to each partition of the projection plane, the method further includes: Projecting a preset calibration image onto the projection plane through the projection module; The camera is controlled to shoot the projection plane to obtain a second image; wherein the preset calibration image and the second image both include a plurality of binary calibration codes and a checkerboard; the plurality of binary calibration codes of the preset calibration image divide the preset calibration image into a plurality of first partitions, and the plurality of binary calibration codes of the second image divide the projection plane in the second image into a plurality of second partitions; a checkerboard is provided in each of the first partitions and each of the second partitions; and each of the first partitions corresponds one-to-one to each of the second partitions.
6. The human-computer interaction method based on an optical interaction device according to claim 5, characterized in that: The obtaining of the homography matrix corresponding to each partition of the projection plane includes: Obtaining the coordinates of the first corner points of each partition in the second image in the camera image coordinate system; Obtain the coordinates of the second corner point of each corner point of each partition in the preset calibration image in the display screen coordinate system; According to the first corner point coordinates and the second corner point coordinates of each corner point of each partition, a homography matrix corresponding to each partition is obtained.
7. The human-computer interaction method based on an optical interaction device according to claim 1, characterized in that: The method further comprises: Determining a moving trajectory of the feature point according to a time series of coordinates of the second feature point; Receive interaction information input by a user, and determine target presentation information according to the interaction information and the moving trajectory of the feature point.
8. A human-computer interaction device based on an optical interaction device, characterized in that: The optical interaction device includes a camera, a projection module and a host computer, and the feature point calibration device based on the optical interaction device includes: An image acquisition unit, used for acquiring a first image of a projection plane captured by the camera, wherein the projection plane has a feature point to be detected; A feature point positioning unit, used to determine the first feature point coordinates of the feature point in the first image in the camera image coordinate system; A processing unit, used to obtain a homography matrix corresponding to each partition of the projection plane, and determine a target homography matrix corresponding to the partition where the feature point is located; The coordinate mapping unit is also used to determine the second feature point coordinates of the feature point in the display screen coordinate system according to the first feature point coordinates and the target homography matrix, so that the host computer performs the interactive task according to the second feature point coordinates.
9. An optical interactive device, characterized in that: The optical interaction device includes a laser projection device, a camera, a projection module and a host computer; The laser projection device is used to emit a laser beam to the projection plane; the laser beam forms a characteristic point on the projection plane; The camera is used to image the projection plane to obtain image data; The projection module is used to project a preset calibration image onto the projection plane; The host computer is configured to execute the human-computer interaction method based on the optical interaction device according to any one of claims 1 to 7; The laser projection device includes a photoelectric button, which includes a touch panel, a beam emitting element and a beam sensing element; the beam emitting element is used to emit a laser beam to the touch panel; the beam sensing element is used to receive the laser beam reflected by the touch panel; the beam sensing element is configured to generate a button activation signal and trigger a custom event if the beam sensing element receives the laser beam reflected from the touch panel, so that the host computer responds to and executes the custom event.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is run on a computer, the human-computer interaction method based on an optical interaction device as claimed in any one of claims 1 to 7 is executed.
Citation Information
Cited By
Display screen brightness compensation method, device and system and storage medium
CN120356444A
