Desktop projection touch control method and system based on deep infrared fusion

Abstract

The invention relates to a depth infrared fusion desktop projection touch control method and system, and the method comprises the following steps: synchronously collecting RGB, infrared and depth image data of a projection desktop area through a depth camera; rGB image data is processed, a hand region is segmented, recognized and extracted, and a coordinate set and finger posture parameters are obtained; edge enhancement is carried out based on the infrared image data, and an enhanced finger contour feature map is obtained; combining the depth image data and the feature map to construct an enhanced three-dimensional hand model, and analyzing the spatial distance between the enhanced three-dimensional hand model and a projection desktop to obtain a fingertip distance change curve; when the curve meets a preset threshold value, a touch event judgment result is generated, and the technical problem that when a model constructed in the prior art judges whether a touch event is generated or not according to a fingertip distance change curve, misjudgment or delayed response often occurs is solved.

Description

Technical Field

[0001] The present invention relates to the field of projection touch technology, and in particular to a deep infrared fusion desktop projection touch method and system. Background Art

[0002] In today's digital interactive era, desktop projection touch technology has been widely adopted in many fields. However, existing technologies still have significant limitations. Regarding image acquisition, traditional methods struggle to accurately and simultaneously acquire high-quality RGB, infrared, and depth image data in complex environments. Factors such as ambient light interference and limited shooting angles often result in noise, blur, or missing information in the captured images, severely impacting subsequent accurate recognition and analysis of the hand area.

[0003] When it comes to hand region segmentation, recognition, and feature extraction, existing technologies are susceptible to interference from complex situations such as hand occlusion and multi-person interaction when processing hand region coordinate sets and finger gesture parameters. When multiple hands are simultaneously present in the projected desktop area, the algorithm may misjudge, failing to accurately segment each hand's individual region and accurately extract subtle changes in finger gesture parameters, which in turn affects the system's understanding of user intent.

[0004] During the touch detection phase, models built based on existing technologies often misjudge or delay responses when determining whether a touch event has occurred based on the fingertip distance curve. Due to the lack of effective fusion and weighted processing of depth image data and finger contour feature maps, the generated three-dimensional hand model is inaccurate and unable to accurately reflect the user's hand's actual position and movement in space. This makes it difficult for the system to meet the real-time and accuracy requirements for fast and complex gesture operations, greatly reducing the user's interactive experience. Summary of the Invention

[0005] The main purpose of the present invention is to provide a desktop projection touch method and system with deep infrared fusion, which solves the technical problem that the model constructed by the existing technology often makes misjudgments or delayed responses when judging whether a touch event is generated based on the fingertip distance change curve.

[0006] To achieve the above object, the present invention provides a desktop projection touch control method with deep infrared fusion, comprising the following steps: The projected desktop area is synchronously imaged using a depth camera to obtain RGB image data, infrared image data, and depth image data; Performing hand region segmentation, recognition, and extraction on the RGB image data to obtain a hand region coordinate set and finger posture parameters; Performing edge enhancement processing on the hand region coordinate set and the finger posture parameters based on the infrared image data to obtain an enhanced finger contour feature map; Constructing an enhanced three-dimensional hand model based on the depth image data and the enhanced finger contour feature map by weighting, and performing spatial distance analysis on the projected desktop area based on the enhanced three-dimensional hand model to obtain a fingertip distance change curve; When the fingertip distance change curve meets the preset touch threshold, a touch event determination result is generated.

[0007] Furthermore, the hand region segmentation, recognition and extraction of the RGB image data to obtain a hand region coordinate set and finger posture parameters includes: Performing YCbCr color space conversion on the RGB image data to obtain a skin color candidate area map, and performing Sobel operator calculation on the skin color candidate area map to obtain an edge gradient map; Reconstructing the contour of the skin color candidate area map by an ellipse fitting operation to obtain a hand area mask, and performing convex hull analysis based on the edge gradient map to obtain a region boundary point set; Performing watershed segmentation based on the hand region mask to obtain a finger region sub-atlas, and iteratively refining the region boundary point set to obtain a skeleton feature map; The extreme points of the finger region sub-atlas are extracted by distance transformation to obtain a hand region coordinate set, and angle projection is performed based on the skeleton feature map to obtain finger posture parameters.

[0008] Furthermore, the edge enhancement processing is performed on the hand region coordinate set and the finger posture parameters based on the infrared image data to obtain an enhanced finger contour feature map, including: Performing heat distribution mapping on the infrared image data to obtain a heat zone intensity map, and performing anisotropic diffusion on the heat zone intensity map to obtain a temperature field distribution map; Performing regional projection on the temperature field distribution map based on the hand region coordinate set to obtain an infrared edge contour map, and performing boundary mapping according to the thermal zone intensity map to obtain a thermal zone overlap map; Performing multi-scale edge operations on the infrared edge contour map to obtain an enhanced edge map, and performing wavelet decomposition on the hot zone overlap map to obtain a contour mutation point set; Performing region adaptive threshold segmentation on the enhanced edge map based on the finger posture parameters to obtain a fingertip hot zone feature map, and performing curve fitting on the contour mutation point set to obtain a hot zone contour descriptor; Perform bilateral filtering on the fingertip hot zone feature map and the hot zone contour descriptor to obtain an enhanced finger contour feature map.

[0009] Furthermore, performing regional projection on the temperature field distribution map based on the hand region coordinate set to obtain an infrared edge contour map includes: Performing polar coordinate transformation on the hand region coordinate set to obtain a hand region polar coordinate mapping diagram, and performing Gaussian curvature calculation on the temperature field distribution diagram to obtain a temperature field curvature distribution diagram; performing a regional mask operation on the temperature field curvature distribution map based on the hand region polar coordinate mapping map to obtain a hand thermal zone gradient map, and performing a Laplace transform on the hand thermal zone gradient map to obtain a thermal zone boundary enhancement map; Performing arc detection on the hot zone boundary enhancement image through Hough circle transform to obtain a fingertip arc feature set, and performing region growing segmentation based on the fingertip arc feature set to obtain a fingertip hot zone contour image; The fingertip hot zone contour map is subjected to morphological refinement processing to obtain a fingertip edge skeleton map, and a B-spline curve fitting is performed based on the fingertip edge skeleton map to obtain an infrared edge contour map.

[0010] Furthermore, performing polar coordinate transformation on the hand region coordinate set to obtain a hand region polar coordinate mapping diagram includes: Performing centroid positioning calculation on the hand region coordinate set to obtain a hand region centroid coordinate point set, and performing radial distance calculation on the hand region centroid coordinate point set to obtain a hand region radial distribution map; performing angle quantization segmentation on the hand region coordinate set based on the hand region radial distribution map to obtain a hand region angle distribution sequence, and performing polar coordinate grid division on the hand region angle distribution sequence to obtain a hand region grid mapping matrix; performing radial normalization processing on the hand region grid mapping matrix to obtain a normalized polar coordinate distribution map, and performing polar coordinate interpolation operation based on the normalized polar coordinate distribution map to obtain a hand region polar coordinate density map; Performing polar coordinate gradient calculation based on the polar coordinate density map of the hand region to obtain a polar coordinate gradient field of the hand region, and performing polar coordinate boundary extraction on the polar coordinate gradient field of the hand region to obtain a polar coordinate boundary set of the hand region; Polar coordinate smoothing is performed on the hand region polar coordinate boundary set to obtain a hand region polar coordinate contour map, and polar coordinate resampling is performed based on the hand region polar coordinate contour map to obtain a hand region polar coordinate mapping map.

[0011] Furthermore, the polar coordinate gradient calculation is performed based on the polar coordinate density map of the hand area to obtain the polar coordinate gradient field of the hand area, including: Performing radial difference operation on the polar coordinate density map of the hand region to obtain a radial gradient component map, and performing angular projection transformation on the radial gradient component map to obtain a polar coordinate directional gradient matrix; Performing polar coordinate spatial filtering on the polar coordinate density map of the hand region based on the polar coordinate directional gradient matrix to obtain a polar coordinate gradient intensity distribution map, and performing polar coordinate tensor decomposition on the polar coordinate gradient intensity distribution map to obtain a polar coordinate principal direction field; Performing polar coordinate flow field analysis on the polar coordinate main direction field to obtain a polar coordinate streamline feature set, and performing polar coordinate curvature calculation based on the polar coordinate streamline feature set to obtain a polar coordinate curvature distribution diagram; The polar coordinate vector field of the polar coordinate streamline feature set is reconstructed based on the polar coordinate curvature distribution map to obtain a polar coordinate gradient field of the hand region.

[0012] Furthermore, the spatial distance analysis of the projection desktop area is performed based on the enhanced three-dimensional hand model to obtain a fingertip distance change curve, including: Performing spatial coordinate transformation on the enhanced three-dimensional hand model to obtain a hand depth map in a projection plane coordinate system, and performing Gaussian curvature calculation on the hand depth map to obtain a local surface feature set; Performing region segmentation on the hand depth map based on the local surface feature set to obtain a fingertip candidate region map, and performing principal curvature analysis on the fingertip candidate region map to obtain a spatial curvature descriptor; Performing radial scanning on the fingertip candidate area map through polar coordinate transformation to obtain a depth gradient sequence, and performing depth threshold stratification based on the spatial curvature descriptor to obtain a fingertip trajectory point set; Performing a temporal correlation analysis on the fingertip trajectory point set to obtain a motion trajectory vector field, and performing a spatial interpolation operation based on the depth gradient sequence to obtain a continuous depth change map; The continuous depth change map is subjected to spatiotemporal filtering based on the motion trajectory vector field to obtain a fingertip distance change curve, which includes the fingertip spatial position, motion speed and acceleration parameters.

[0013] The present invention also provides a deep infrared fusion desktop projection touch system, comprising: The acquisition module is used to synchronously acquire images of the projected desktop area through a depth camera to obtain RGB image data, infrared image data, and depth image data; An extraction module is used to segment, identify and extract the hand region from the RGB image data to obtain a hand region coordinate set and finger posture parameters; an enhancement module, configured to perform edge enhancement processing on the hand region coordinate set and the finger posture parameters based on the infrared image data to obtain an enhanced finger contour feature map; an analysis module, configured to construct an enhanced three-dimensional hand model based on the depth image data and the enhanced finger contour feature map by weight, and perform spatial distance analysis on the projected desktop area based on the enhanced three-dimensional hand model to obtain a fingertip distance change curve; The judgment module is configured to generate a touch event judgment result when the fingertip distance change curve meets a preset touch threshold.

[0014] The present invention also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the above methods when executing the computer program.

[0015] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0016] The present invention provides a desktop projection touch method with deep infrared fusion, comprising the following steps: synchronously capturing images of a projected desktop area using a depth camera to obtain RGB image data, infrared image data, and depth image data; performing hand region segmentation, identification, and extraction on the RGB image data to obtain a hand region coordinate set and finger posture parameters; performing edge enhancement processing on the hand region coordinate set and the finger posture parameters based on the infrared image data to obtain an enhanced finger contour feature map; weightedly constructing an enhanced three-dimensional hand model based on the depth image data and the enhanced finger contour feature map, and performing spatial distance analysis on the projected desktop area based on the enhanced three-dimensional hand model to obtain a fingertip distance change curve; generating a touch event determination result when the fingertip distance change curve meets a preset touch threshold, thereby solving the technical problem of models constructed in the prior art that often suffer from misjudgment or delayed response when determining whether to generate a touch event based on the fingertip distance change curve. The method implements spatial distance analysis of the projected desktop area based on the enhanced three-dimensional hand model to obtain a fingertip distance change curve, and generates a touch event determination result based on the preset touch threshold. This method effectively solves the problems of misjudgment or delayed response in traditional methods, and can quickly and accurately capture the user's touch intention, especially when facing fast and complex gesture operations. It can meet real-time and accuracy requirements and significantly improve the technical effect of the user's interactive experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a schematic diagram of the steps of a desktop projection touch control method using deep infrared fusion according to an embodiment of the present invention; Figure 2 This is a structural block diagram of a desktop projection touch control system with deep infrared fusion according to an embodiment of the present invention; Figure 3 It is a schematic block diagram of the structure of a computer device according to an embodiment of the present invention.

[0018] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0020] like Figure 1 As shown, Figure 1 This is a schematic diagram of the steps of a desktop projection touch control method using deep infrared fusion in one embodiment of the present invention; An embodiment of the present invention provides a desktop projection touch method with deep infrared fusion, comprising the following steps: Step S1: synchronously capture images of the projected desktop area through a depth camera to obtain RGB image data, infrared image data, and depth image data.

[0021] Specifically, in the deep infrared fusion desktop projection touch control method, the crucial step of "using a depth camera to simultaneously capture images of the projected desktop area to obtain RGB image data, infrared image data, and depth image data" is fundamental and forms the foundation for subsequent operations. The depth camera possesses a unique capability, enabling simultaneous image capture of the projected desktop area. The RGB image data reflects the true color information of the projected desktop area, much like a color photograph taken with a standard camera, clearly capturing details such as the color and texture of objects within the desktop area. The infrared image data is based on infrared light emitted by objects. Due to differences in infrared reflection and radiation between the hand and the desktop, the infrared image data can highlight the features of the hand. The depth image data records the distance between objects within the projected desktop area and the camera, allowing the objects' positions in three-dimensional space to be determined. For example, in a teaching scenario, teachers use desktop projection to present and explain courseware. The depth camera is appropriately positioned to simultaneously capture images of the projected desktop area. During this process, RGB image data can clearly record the color and text content of the courseware projected on the desktop; infrared image data can keenly capture the position and movement of the teacher's hand, accurately identifying the hand even in low-light environments; and depth image data can accurately measure the distance between the teacher's hand and the desktop, providing a basis for subsequent judgment of the teacher's operating intentions. Through the simultaneous collection of these three types of data, various information about the projected desktop area can be comprehensively and accurately obtained, laying a solid foundation for subsequent operations such as hand area segmentation and recognition and edge enhancement processing.

[0022] Step S2: segmenting, identifying, and extracting the hand region of the RGB image data to obtain a hand region coordinate set and finger posture parameters.

[0023] Specifically, in the deep infrared fusion desktop projection touch control method, segmenting, identifying, and extracting the hand region from the RGB image data to obtain a set of hand region coordinates and finger posture parameters is a key step. This step occurs after the depth camera completes synchronous image acquisition of the projected desktop area and acquires the RGB image data. The RGB image data contains color information of the projected desktop area, including images of the hand. First, an image processing algorithm analyzes the RGB image data, segmenting the hand region from the overall image based on the differences in color and texture between the hand and the background. This is like precisely circling a specific pattern in a painting. Next, the segmented hand region is identified and its specific location in the image is determined, resulting in a set of hand region coordinates. This set clearly defines the hand's position within the projected desktop area. Further analysis of the hand's morphology is then performed. By detecting and calculating features such as the hand's contour and joints, finger posture parameters are extracted. These parameters reflect the degree of finger flexion and extension. Taking the teaching scenario as an example, when the teacher uses a desktop projection to explain, the depth camera collects RGB image data containing the teacher's hand. After the above processing, the system can accurately segment the teacher's hand area, obtain the coordinate set of the hand on the projection desktop, and also obtain the finger posture parameters. For example, when the teacher points his finger at a certain knowledge point on the courseware, the system can recognize this action through the finger posture parameters, and then provide a basis for subsequent interactive operations, such as zooming in on the knowledge point content, making the teaching process more convenient and efficient.

[0024] Step S3: performing edge enhancement processing on the hand region coordinate set and the finger posture parameters based on the infrared image data to obtain an enhanced finger contour feature map.

[0025] Specifically, in a desktop projection touch method using deep infrared fusion, edge enhancement processing of the hand region coordinate set and finger posture parameters based on infrared image data to generate an enhanced finger contour feature map is a key step in improving the accuracy of subsequent touch detection. After completing hand region segmentation, recognition, and extraction from the RGB image data, generating the hand region coordinate set and finger posture parameters, this step is combined with the infrared image data. Infrared image data can highlight the differences between the hand and its surroundings because the infrared radiation characteristics of the hand differ from those of the desktop and other background objects. Leveraging this characteristic, the infrared image data is combined with the hand region coordinate set and finger posture parameters. First, the hand region is located in the infrared image based on the hand region coordinate set. The finger posture within this region is then analyzed. A specific edge enhancement algorithm is used to enhance the edge information of the finger portion in the infrared image. This algorithm enhances edge contrast, making the finger contour clearer and more distinct, and reducing edge blurring caused by image noise or lighting. After processing, an enhanced finger contour feature map is generated, which more accurately reflects the true contour and details of the finger. Taking the teaching scenario as an example, when the teacher operates on the projection desktop, the depth camera collects infrared image data. Based on the previously obtained hand area coordinate set, the system accurately finds the position of the teacher's hand in the infrared image. For the posture of the teacher's fingers, such as bending or stretching, the finger contours are clearly presented through edge enhancement processing. For example, when the teacher writes with his fingers on the projection desktop, the enhanced finger contour feature map can accurately capture the trajectory and shape changes of the fingers, providing a more accurate basis for the subsequent construction of an enhanced three-dimensional hand model, so that the system can more accurately identify the teacher's operating intentions and achieve smoother teaching interaction.

[0026] Step S4: constructing an enhanced three-dimensional hand model based on the depth image data and the enhanced finger contour feature map by weight, and performing spatial distance analysis on the projected desktop area based on the enhanced three-dimensional hand model to obtain a fingertip distance change curve.

[0027] Specifically, in the deep infrared fusion desktop projection touch method, an enhanced 3D hand model is constructed based on weighted depth image data and an enhanced finger contour feature map. Spatial distance analysis of the projected desktop area is then performed based on this model to generate a fingertip distance curve, a key step in achieving accurate touch detection. Once the infrared image data is processed to generate the enhanced finger contour feature map, subsequent steps are combined with the depth image data. The depth image data contains distance information between objects within the projected desktop area and the camera, reflecting the hand's position in 3D space; the enhanced finger contour feature map accurately depicts the finger's 2D contour details. By weighting these two types of data—that is, assigning different weights based on their importance to constructing an accurate hand model—the advantages of both are combined. The weighting of the depth image data ensures the model's spatial accuracy, while the weighting of the enhanced finger contour feature map ensures that the model reflects the finger's true shape. This results in an enhanced 3D hand model that more realistically reproduces the hand's 3D form within the projected desktop area. This enhanced 3D hand model is then used to perform spatial distance analysis of the projected desktop area. Specifically, the system continuously monitors the changes in distance between the fingertips in the model and various locations within the projected desktop area. As the hand operates on the projected desktop, the distance between the fingertips and the desktop continuously changes. These distance changes are recorded chronologically to generate a fingertip distance change curve. For example, when a teacher uses a desktop projection to give a lecture, the depth camera captures depth image data and combines it with the previously generated enhanced finger contour feature map to construct an enhanced 3D hand model. As the teacher taps, swipes, and other operations on the projected desktop, the system analyzes the changes in the distance between the fingertips and the desktop in real time based on this model. For example, when the teacher taps a location on the projected desktop, the distance between the fingertip and the desktop gradually decreases, reaching a minimum at the moment of tapping, and then gradually increases. These changes are reflected in the fingertip distance change curve. By analyzing this curve, the system can more accurately determine the teacher's operational intentions, providing a reliable basis for subsequently generating touch event determination results.

[0028] Step S5: When the fingertip distance change curve meets a preset touch threshold, a touch event determination result is generated.

[0029] Specifically, in the deep infrared fusion desktop projection touch control method, generating a touch event determination result when the fingertip distance change curve meets the preset touch threshold is a key decision-making step in the entire process. This determination is made after constructing an enhanced 3D hand model based on depth image data and enhanced finger contour feature maps, and performing spatial distance analysis on the projected desktop area to generate the fingertip distance change curve. The preset touch threshold is a standard value set based on actual application requirements and extensive experimental data, and it defines the conditions for triggering a touch event. As the system continuously monitors the fingertip distance change curve, it compares the distance change between the fingertip and the projected desktop, as reflected by the curve, with the preset touch threshold. If one or more data points in the fingertip distance change curve meet the preset touch threshold conditions, such as the minimum distance between the fingertip and the desktop reaching the set threshold, or the rate or amplitude of the distance change falling within a specific threshold range, the system determines that a touch event has occurred. For example, when a teacher uses desktop projection for teaching demonstrations, the system generates a real-time fingertip distance change curve based on hand movements on the projected desktop. Assume the preset touch threshold is set to a click touch event when the fingertip is less than 1 cm from the desktop. When the teacher attempts to click a courseware icon on the projected desktop, the fingertip distance curve shows a decreasing trend as the finger approaches the desktop. When the curve shows that the fingertip is less than 1 cm from the desktop, the preset touch threshold is met, and the system generates a click touch event result and performs the corresponding action, such as opening the content corresponding to the courseware icon. This enables interaction between the teacher and the projected desktop, improving the convenience and efficiency of teaching.

[0030] In a specific embodiment, the segmentation, recognition, and extraction of the hand region on the RGB image data to obtain a hand region coordinate set and finger posture parameters includes: Performing YCbCr color space conversion on the RGB image data to obtain a skin color candidate area map, and performing Sobel operator calculation on the skin color candidate area map to obtain an edge gradient map; Reconstructing the contour of the skin color candidate area map by an ellipse fitting operation to obtain a hand area mask, and performing convex hull analysis based on the edge gradient map to obtain a region boundary point set; Performing watershed segmentation based on the hand region mask to obtain a finger region sub-atlas, and iteratively refining the region boundary point set to obtain a skeleton feature map; The extreme points of the finger region sub-atlas are extracted by distance transformation to obtain a hand region coordinate set, and angle projection is performed based on the skeleton feature map to obtain finger posture parameters.

[0031] Specifically, in the deep infrared fusion desktop projection touch method, hand region segmentation, identification, and extraction from RGB image data to obtain a set of hand region coordinates and finger posture parameters are crucial for achieving precise touch interaction. This process involves multiple, closely linked steps, using a series of image processing and analysis algorithms to gradually extract key information from the RGB image data. First, the RGB image data is converted to the YCbCr color space to generate a skin color candidate region map. While the RGB color space intuitively represents image color, it has certain limitations when it comes to skin color detection. The YCbCr color space, on the other hand, separates luminance information (Y) from chrominance information (Cb and Cr), making it more conducive to skin color recognition. In the converted YCbCr color space, skin colors are typically concentrated within a specific chromaticity range. By setting an appropriate chromaticity threshold, regions likely to contain skin color can be screened out to form a skin color candidate region map. For example, in an RGB image of a teacher's hand operating a projected desktop, approximately 30% of the image area is marked as skin color candidate regions after the YCbCr color space conversion. Next, the Sobel operator is applied to the skin color candidate region map to obtain an edge gradient map. The Sobel operator is a commonly used edge detection operator that highlights edge information in an image by calculating the horizontal and vertical gradient values ​​for each pixel in the image. In the skin color candidate region map, the edges of the hand typically have large gradient values. The Sobel operator can clearly extract these edges, forming an edge gradient map. This helps to more accurately determine the hand contour. Then, the contour of the skin color candidate region map is reconstructed using an ellipse fitting operation to obtain a hand region mask. Ellipse fitting is a commonly used contour fitting method that approximates the hand contour using an ellipse. After obtaining the ellipse, the area inside the ellipse is marked as the hand region, thereby obtaining a hand region mask. This mask helps to more accurately determine the hand's position in the image. Convex hull analysis is also performed on the edge gradient map to obtain a set of region boundary points. Convex hull analysis is an algorithm used to find the smallest convex polygon for a set of points. In the edge gradient map, the edge points of the hand can be considered a set of points. Through convex hull analysis, the convex hull of these points can be found, which is the smallest circumscribed polygon of the hand region. The vertices of this polygon are the set of region boundary points, which provide important boundary information for subsequent finger segmentation and posture analysis. Next, watershed segmentation is performed based on the hand region mask to obtain a sub-atlas of the finger region. Watershed segmentation is a segmentation method based on image grayscale values. It treats the image as a terrain surface, with areas with high grayscale values ​​as peaks and areas with low grayscale values ​​as valleys. By simulating the flow of water, the image is segmented into different regions.Based on the hand region mask, the watershed segmentation algorithm can be used to further segment the hand region into multiple finger regions, resulting in a set of finger region sub-atlases. For example, in an image of a teacher manipulating a projected table with five fingers spread out, watershed segmentation successfully segments the hand region into five sub-atlases, corresponding to each of the five fingers. Simultaneously, the region boundary point set is iteratively refined to obtain a skeleton feature map. Iterative refinement is an algorithm that gradually refines lines in an image. Through multiple iterations, the region boundary point set is refined into a single centerline, the skeleton feature map. This skeleton feature map more clearly represents the shape and structure of the fingers. Finally, the extreme points of the finger region sub-atlas are extracted using a distance transform to obtain a set of hand region coordinates. A distance transform is an algorithm that calculates the distance from each pixel in the image to its nearest background pixel. In the finger region sub-atlas, the center of each finger typically has the largest distance value. By extracting these extreme points, the center coordinates of each finger can be obtained, thereby obtaining a set of hand region coordinates. For example, in the aforementioned finger region sub-image set consisting of five sub-images, distance transformation successfully extracted five extreme points, corresponding to the center coordinates of the five fingers. Furthermore, angle projection was performed based on the skeleton feature map to obtain finger posture parameters. Angle projection is a method that projects lines from the skeleton feature map onto different angles. By analyzing the projection results, posture parameters such as the finger's bending angle and extension state can be obtained. For example, angle projection analysis revealed that the teacher's index finger was bent at a 30-degree angle, indicating that the teacher may be performing a click operation. For example, in a teaching scenario, when a teacher uses a desktop projection to teach, the depth camera captures RGB image data containing the teacher's hand movements. Through the above series of steps, the system can accurately segment the teacher's hand region from the image, obtaining a set of hand region coordinates and finger posture parameters. This information can be used for subsequent edge enhancement, 3D hand model construction, and touch event detection, enabling precise interaction between the teacher and the projected desktop. For example, when a teacher wants to click on a courseware icon on the projection desktop, the system can accurately judge the teacher's intention by analyzing the finger posture parameters and hand area coordinate set, and trigger the corresponding operation in time, thereby improving the efficiency and convenience of teaching.

[0032] In a specific embodiment, the edge enhancement processing is performed on the hand region coordinate set and the finger posture parameters based on the infrared image data to obtain an enhanced finger contour feature map, including: Performing heat distribution mapping on the infrared image data to obtain a heat zone intensity map, and performing anisotropic diffusion on the heat zone intensity map to obtain a temperature field distribution map; Performing regional projection on the temperature field distribution map based on the hand region coordinate set to obtain an infrared edge contour map, and performing boundary mapping according to the thermal zone intensity map to obtain a thermal zone overlap map; Performing multi-scale edge operations on the infrared edge contour map to obtain an enhanced edge map, and performing wavelet decomposition on the hot zone overlap map to obtain a contour mutation point set; Performing region adaptive threshold segmentation on the enhanced edge map based on the finger posture parameters to obtain a fingertip hot zone feature map, and performing curve fitting on the contour mutation point set to obtain a hot zone contour descriptor; Perform bilateral filtering on the fingertip hot zone feature map and the hot zone contour descriptor to obtain an enhanced finger contour feature map.

[0033] Specifically, in the deep infrared fusion desktop projection touch method, edge enhancement processing is performed on the hand region coordinate set and finger posture parameters based on infrared image data to generate an enhanced finger contour feature map. This is a key step in improving finger contour recognition accuracy and providing an accurate basis for subsequent 3D hand model construction. The implementation process of this step is explained in detail below. First, heat distribution mapping is performed on the infrared image data to obtain a thermal intensity map. Infrared images record infrared radiation information from the surface of an object, and different temperatures correspond to different infrared radiation intensities. Through heat distribution mapping, the radiation intensity in the infrared image can be converted into an intuitive thermal intensity map. The value of each pixel in the map represents the relative heat intensity of that area. For example, in an infrared image of a teacher's hand operating a projected desktop, the hand generates heat due to blood flow and metabolism. In the thermal intensity map, the value of the hand area may be between 80 and 100, while other areas such as the desktop may have values ​​between 20 and 40. Next, anisotropic diffusion is performed on the thermal intensity map to obtain a temperature field distribution map. Anisotropic diffusion is an algorithm used to smooth images while preserving edge information. It adjusts the diffusion coefficient based on the gradient information of different image regions. This process effectively removes noise from the thermal intensity map while preserving the edges of temperature variations, resulting in a smoother temperature field distribution map that reflects the regularity of temperature distribution. Next, the temperature field distribution map is regionally projected based on the hand region coordinate set to produce an infrared edge contour map. The hand region coordinate set clearly defines the hand's position in the image. By projecting the temperature field distribution map onto these regions, the edge information of the hand region is highlighted, as temperature variations at edges are typically more pronounced. Simultaneously, boundary mapping is performed based on the thermal intensity map to produce a thermal overlap map. Boundary mapping extracts boundary information from the thermal intensity map and analyzes the overlap between different thermal regions. For example, there may be some overlap between fingers. The thermal overlap map quantifies this overlap, providing data support for subsequent analysis. Next, multi-scale edge operations are performed on the infrared edge contour map to produce an enhanced edge map. Multi-scale edge operations involve edge detection at different scales, capturing edge features of varying sizes. In this way, the clarity and continuity of the finger edges can be further enhanced, making the edges more prominent. At the same time, the heat zone overlap map is subjected to wavelet decomposition to obtain a set of contour mutation points. Wavelet decomposition is a method of decomposing a signal into different frequency components. In the heat zone overlap map, contour mutation points usually correspond to key parts such as finger joints and fingertips. Wavelet decomposition can accurately locate these mutation points, providing important information for subsequent finger contour analysis. Afterwards, the enhanced edge map is subjected to regional adaptive threshold segmentation based on the finger posture parameters to obtain the fingertip heat zone feature map.Finger posture parameters describe the state of the finger, such as bending and extension. Based on these parameters, the threshold can be adaptively adjusted to accurately segment the fingertip region in the enhanced edge map. For example, if the finger is bent, the characteristics of the fingertip hotspot may differ from those in an extended state. Adaptive threshold segmentation can better adapt to this variation. Simultaneously, curve fitting is performed on the set of contour mutation points to obtain a hotspot contour descriptor. Curve fitting approximates the set of contour mutation points with a curve, which more concisely describes the finger's contour shape. Finally, bilateral filtering is performed on the fingertip hotspot feature map and the hotspot contour descriptor to obtain an enhanced finger contour feature map. Bilateral filtering is a filtering method that smooths images while preserving edge information. It combines information from both the spatial and numerical domains. Bilateral filtering further removes noise from the fingertip hotspot feature map and the hotspot contour descriptor while preserving finger contour details, making the final enhanced finger contour feature map clearer and more accurate. For example, in a teaching scenario, a teacher uses a desktop projection to teach while a depth camera captures infrared image data. Through these processing steps, the system can accurately extract an enhanced finger contour feature map from the infrared image. For example, when analyzing the teacher's finger movement when clicking an icon on the projected desktop, the enhanced finger contour feature map can clearly show the position and shape of the fingertip, providing key information for subsequently building a more accurate 3D hand model. This enables the system to more accurately determine the teacher's operating intentions and achieve smoother teaching interaction.

[0034] In a specific embodiment, performing regional projection on the temperature field distribution map based on the hand region coordinate set to obtain an infrared edge contour map includes: Performing polar coordinate transformation on the hand region coordinate set to obtain a hand region polar coordinate mapping diagram, and performing Gaussian curvature calculation on the temperature field distribution diagram to obtain a temperature field curvature distribution diagram; performing a regional mask operation on the temperature field curvature distribution map based on the hand region polar coordinate mapping map to obtain a hand thermal zone gradient map, and performing a Laplace transform on the hand thermal zone gradient map to obtain a thermal zone boundary enhancement map; Performing arc detection on the hot zone boundary enhancement image through Hough circle transform to obtain a fingertip arc feature set, and performing region growing segmentation based on the fingertip arc feature set to obtain a fingertip hot zone contour image; The fingertip hot zone contour map is subjected to morphological refinement processing to obtain a fingertip edge skeleton map, and a B-spline curve fitting is performed based on the fingertip edge skeleton map to obtain an infrared edge contour map.

[0035] Specifically, in the deep infrared fusion desktop projection touch method, regional projection of the temperature field distribution map based on the hand region coordinate set to obtain an infrared edge contour map is a gradually refined and refined process, which plays a key role in accurately identifying the edge contour of the hand in the infrared image. Below, we explain the implementation of this process in detail, using a teaching scenario. First, a polar coordinate transformation is performed on the acquired hand region coordinate set to obtain a polar coordinate mapping of the hand region. In a rectangular coordinate system, the hand region coordinate set records the position information of each hand point on a plane, but this representation is not intuitive when dealing with certain specific geometric features. Polar coordinate transformation converts these coordinates into polar coordinate form, describing the position of points using distance and angle. This makes it easier to analyze the shape and structure of the hand region. For example, in a teaching scenario, a teacher is operating on a projected table. After the polar coordinate transformation, the distance and angle relationships of various parts of the hand relative to a central point can be clearly seen, which facilitates more accurate regional analysis. Simultaneously, Gaussian curvature is calculated on the temperature field distribution map to obtain a temperature field curvature distribution map. Gaussian curvature reflects the degree of curvature in the temperature field distribution. Temperature variations between the hand and the surrounding environment typically result in changes in curvature. By calculating Gaussian curvature, we can highlight the characteristics of the hand region in the temperature field. For example, at the edge of the hand, where temperature variations are greater, the Gaussian curvature value is also relatively large. Next, we perform a region masking operation on the temperature field curvature distribution map based on the polar coordinate map of the hand region to obtain a hand thermal gradient map. The region masking operation acts like a "filter," extracting hand-related information from the temperature field curvature distribution map within the range defined by the polar coordinate map of the hand region and removing irrelevant background information. This operation yields a gradient map containing only the hand thermal region, which more clearly demonstrates the temperature variation trend of the hand thermal region. We then perform a Laplace transform on the hand thermal gradient map to obtain a thermal boundary enhancement map. The Laplace transform is a commonly used edge detection method that emphasizes edge information by calculating the second-order derivative of each pixel in an image. In the hand heat gradient map, the Laplace transform can more clearly display the boundaries of the hand heat zone, making subsequent analysis more accurate. For example, after the Laplace transform, the originally blurred hand heat zone boundaries become sharper, facilitating further processing. The Hough circle transform is then used to perform arc detection on the enhanced heat zone boundary map to obtain a set of fingertip arc features. In infrared images of the hand, the fingertips typically exhibit approximately circular features, and the Hough circle transform can effectively detect these circular areas. By setting appropriate parameters, we can accurately locate the fingertips and extract their arc features.For example, in a hotspot boundary enhancement image containing the teacher's finger movements, the Hough circle transform successfully detected five fingertip arc features. These features provide an important basis for subsequent region segmentation. Region growing segmentation is then performed based on the fingertip arc feature set to obtain a fingertip hotspot contour map. Region growing segmentation is a segmentation method based on pixel similarity. It starts from points in the fingertip arc feature set and gradually expands to surrounding pixels with similar temperature characteristics, forming a complete fingertip hotspot contour. This allows us to clearly see the hotspot range of each fingertip. Finally, morphological thinning is performed on the fingertip hotspot contour map to obtain a fingertip edge skeleton map. Morphological thinning gradually refines the lines in the fingertip hotspot contour map, retaining only the central skeleton portion. This provides a more concise representation of the fingertip edge shape. For example, after thinning, the originally thick fingertip contour line becomes a thin skeleton line, which is more convenient for subsequent curve fitting. Next, B-spline curve fitting is performed on the fingertip edge skeleton map to obtain the infrared edge contour map. B-spline curves are a commonly used curve fitting method. They can be used to fit a smooth curve based on the points in the fingertip edge skeleton image. This curve is the resulting infrared edge profile, which accurately describes the edge shape of the hand in the infrared image. In a teaching scenario, through the above series of steps, we can accurately extract the infrared edge profile of the teacher's hand from the infrared image. This profile is important for subsequently constructing an enhanced 3D hand model, performing spatial distance analysis, and generating touch event detection results. For example, when a teacher taps a courseware icon on the projected desktop, the system can accurately determine the finger's position and movement based on the infrared edge profile, enabling precise interactive operations and improving teaching efficiency and effectiveness.

[0036] In a specific embodiment, performing polar coordinate transformation on the hand region coordinate set to obtain a hand region polar coordinate mapping diagram includes: Performing centroid positioning calculation on the hand region coordinate set to obtain a hand region centroid coordinate point set, and performing radial distance calculation on the hand region centroid coordinate point set to obtain a hand region radial distribution map; performing angle quantization segmentation on the hand region coordinate set based on the hand region radial distribution map to obtain a hand region angle distribution sequence, and performing polar coordinate grid division on the hand region angle distribution sequence to obtain a hand region grid mapping matrix; performing radial normalization processing on the hand region grid mapping matrix to obtain a normalized polar coordinate distribution map, and performing polar coordinate interpolation operation based on the normalized polar coordinate distribution map to obtain a hand region polar coordinate density map; Performing polar coordinate gradient calculation based on the polar coordinate density map of the hand region to obtain a polar coordinate gradient field of the hand region, and performing polar coordinate boundary extraction on the polar coordinate gradient field of the hand region to obtain a polar coordinate boundary set of the hand region; Polar coordinate smoothing is performed on the hand region polar coordinate boundary set to obtain a hand region polar coordinate contour map, and polar coordinate resampling is performed based on the hand region polar coordinate contour map to obtain a hand region polar coordinate mapping map.

[0037] Specifically, in the deep infrared fusion desktop projection touch method, performing a polar coordinate transformation on the acquired hand area coordinate set to obtain a polar coordinate map of the hand area is a crucial step in subsequent accurate hand feature analysis. The following describes the implementation of this step in detail, using a teaching scenario. In this teaching scenario, the teacher operates on the projected desktop, and the depth camera captures the relevant images to obtain a hand area coordinate set. First, the center of mass is calculated for this set to obtain the hand area center of gravity coordinate point set. Center of mass calculation is like finding the equilibrium point of an object. For the hand area coordinate set, its center of mass is determined using a specific algorithm. For example, if the hand area coordinate set contains 100 coordinate points, the center of mass calculation yields a single coordinate point representing the hand's center of mass. Next, radial distance calculation is performed on the hand area center of mass coordinate point set to obtain a radial distribution map of the hand area. Radial distance refers to the distance from each coordinate point to the center of mass. Calculating these distances allows for a visual representation of the distribution of various hand parts relative to the center of mass. For example, the radial distance from the coordinate points of the fingertips to the center of gravity may be larger, while the radial distance from the center of the palm is relatively smaller. Based on the hand region radial distribution map, the hand region coordinate set is segmented by angle quantization to obtain a hand region angle distribution sequence. Angle quantization segmentation involves dividing the hand region by angle, for example, dividing 360 degrees into several intervals and counting the number or distribution of coordinate points within each interval. This provides a more detailed understanding of the hand's characteristics at different angles. The hand region angle distribution sequence is then meshed using polar coordinates to obtain a hand region mesh mapping matrix. Polar coordinate meshing is like drawing a grid in a polar coordinate system and filling these grids with hand region information to form a matrix. This matrix provides a more regular representation of the hand region's distribution in polar coordinates. The hand region mesh mapping matrix is ​​then radially normalized to obtain a normalized polar coordinate distribution map. Radial normalization is performed to eliminate the effects of differences in hand size or position and to standardize radial distances within a standard range. For example, all radial distances are mapped to between 0 and 1, so that the features of different hands can be compared more fairly in subsequent analysis. Polar coordinate interpolation is performed based on the normalized polar coordinate distribution map to obtain a polar coordinate density map of the hand area. Polar coordinate interpolation can make estimates between known coordinate points, supplement some missing information, and make the polar coordinate representation of the hand area more continuous and accurate. The polar coordinate density map can reflect the density distribution of the hand area in polar coordinates. Areas with high density may represent the main parts of the hand, such as the palm or fingers. Polar coordinate gradient calculation is then performed based on the polar coordinate density map of the hand area to obtain the polar coordinate gradient field of the hand area. Polar coordinate gradient calculation can analyze the speed and direction of density changes in the polar coordinate density map, just like analyzing the slope of the terrain on a map.The gradient field helps us locate the boundaries and areas of sharp change in the hand region. Next, we perform polar coordinate boundary extraction on the polar coordinate gradient field of the hand region, obtaining a polar coordinate boundary set for the hand region. By setting an appropriate threshold or employing a specific algorithm, we extract the boundary coordinate points of the hand region from the polar coordinate gradient field. These points constitute the polar coordinate boundary set. Finally, we perform polar coordinate smoothing on the polar coordinate boundary set of the hand region to obtain a polar coordinate contour map of the hand region. Polar coordinate smoothing can remove noise and irregular fluctuations at the boundary, making the contour smoother. For example, using a moving average or other smoothing algorithm, we can adjust the coordinates of boundary points to make the contour appear more natural. Polar coordinate resampling is performed based on the polar coordinate contour map of the hand region to obtain a polar coordinate map of the hand region. Polar coordinate resampling involves reselecting coordinate points according to specific rules to ensure that the resolution and accuracy of the polar coordinate map meet the requirements of subsequent analysis. Through these steps, we can obtain an accurate polar coordinate map of the hand region from the hand region coordinate set in the teaching scenario. This mapping provides an important foundation for subsequent edge enhancement processing based on infrared image data, three-dimensional hand model construction, etc., which helps the system to more accurately identify the teacher's hand movements and positions, and achieve smoother teaching interactions. For example, it can accurately judge the teacher's action of clicking on the projected desktop courseware icon, thereby improving teaching efficiency.

[0038] In a specific embodiment, performing polar coordinate gradient calculation based on the hand region polar coordinate density map to obtain the hand region polar coordinate gradient field includes: Performing radial difference operation on the polar coordinate density map of the hand region to obtain a radial gradient component map, and performing angular projection transformation on the radial gradient component map to obtain a polar coordinate directional gradient matrix; Performing polar coordinate spatial filtering on the polar coordinate density map of the hand region based on the polar coordinate directional gradient matrix to obtain a polar coordinate gradient intensity distribution map, and performing polar coordinate tensor decomposition on the polar coordinate gradient intensity distribution map to obtain a polar coordinate principal direction field; Performing polar coordinate flow field analysis on the polar coordinate main direction field to obtain a polar coordinate streamline feature set, and performing polar coordinate curvature calculation based on the polar coordinate streamline feature set to obtain a polar coordinate curvature distribution diagram; The polar coordinate vector field of the polar coordinate streamline feature set is reconstructed based on the polar coordinate curvature distribution map to obtain a polar coordinate gradient field of the hand region.

[0039] Specifically, in the deep infrared fusion desktop projection touch method, polar coordinate gradient calculation based on the hand area polar coordinate density map to obtain the hand area polar coordinate gradient field is the core step in deeply mining the hand's features in polar coordinates, providing key data support for the subsequent accurate extraction of hand boundaries and contours. Taking the teaching scenario of a teacher operating a courseware on a projected desktop as an example, the implementation process is described in detail below. First, a radial difference operation is performed on the obtained hand area polar coordinate density map to obtain a radial gradient component map. The radial difference operation is used to calculate the change in density in the radial direction for each point in the polar coordinate density map, similar to measuring the height difference between two points on a map. For example, in the hand area polar coordinate density map, if the radial distance of a point is 10 units and the radial distance of its adjacent point is 12 units, the radial difference operation can be used to obtain the change in density in the radial direction at that point. Subsequently, the radial gradient component map is angularly projected to obtain the polar coordinate direction gradient matrix. The angular projection transformation rearranges and integrates the information in the radial gradient component map according to the angular direction, fusing the radial and angular information to form a matrix that reflects the gradients in different angular directions, allowing us to observe the density variation trends of the hand from the angular dimension. Next, polar spatial filtering is performed on the polar density map of the hand region based on the polar directional gradient matrix to obtain a polar gradient intensity distribution map. Polar spatial filtering acts like a "filter" on the image. Based on the information provided by the polar directional gradient matrix, it processes each point in the polar density map, enhancing areas with significant density variations and suppressing noise and unnecessary interference. After filtering, the resulting polar gradient intensity distribution map more clearly displays the gradient intensity distribution of the hand region in polar coordinates. For example, areas with large density variations, such as finger edges, will show higher gradient intensity values ​​in the map. The polar gradient intensity distribution map is then subjected to polar tensor decomposition to obtain the polar principal direction field. Tensor decomposition is a complex but effective mathematical technique that extracts the principal direction information (i.e., the direction with the most dramatic density variation) for each point from the polar gradient intensity distribution map. Through tensor decomposition, we can determine the dominant direction of change in each area of ​​the hand, which is crucial for understanding the shape and structure of the hand. Afterwards, a polar coordinate flow field analysis is performed on the polar coordinate principal direction field to obtain a polar coordinate streamline feature set. Polar coordinate flow field analysis is a concept of simulating fluid flow, which regards the polar coordinate principal direction field as a "flow field" in which the principal direction is the direction of fluid flow. By analyzing this "flow field", we can obtain a series of streamlines that reflect the trend and direction of density changes in the hand area, forming a polar coordinate streamline feature set. For example, when a teacher bends his fingers to operate the projection desktop, the polar coordinate streamline feature set can clearly show the path of density changes in the bent part of the finger. Next, the polar coordinate curvature is calculated based on the polar coordinate streamline feature set to obtain a polar coordinate curvature distribution map.Curvature calculation is used to measure the degree of curvature of streamlines. In the hand region, the curvature of parts such as finger joints varies significantly. By calculating curvature, we can quantify this curvature information and form a distribution map, visually demonstrating the curvature of various parts of the hand. Finally, based on the polar curvature distribution map, the polar streamline feature set is reconstructed into a polar coordinate vector field to obtain the polar gradient field of the hand region. Polar coordinate vector field reconstruction integrates the information from the polar curvature distribution map and the polar streamline feature set, assigning each point a vector containing its direction and intensity information, thereby forming a polar gradient field of the hand region. In teaching scenarios, when teachers use their fingers to slide and click on the projected desktop, this polar gradient field accurately reflects the changing characteristics of hand movements in polar coordinates. This provides a detailed and accurate data foundation for subsequent edge enhancement processing based on infrared image data and the construction of a three-dimensional hand model. This helps the system more accurately identify the teacher's operational intentions and achieve efficient teaching interaction.

[0040] In a specific embodiment, performing spatial distance analysis on the projected desktop area based on the enhanced three-dimensional hand model to obtain a fingertip distance change curve includes: Performing spatial coordinate transformation on the enhanced three-dimensional hand model to obtain a hand depth map in a projection plane coordinate system, and performing Gaussian curvature calculation on the hand depth map to obtain a local surface feature set; Performing region segmentation on the hand depth map based on the local surface feature set to obtain a fingertip candidate region map, and performing principal curvature analysis on the fingertip candidate region map to obtain a spatial curvature descriptor; Performing radial scanning on the fingertip candidate area map through polar coordinate transformation to obtain a depth gradient sequence, and performing depth threshold stratification based on the spatial curvature descriptor to obtain a fingertip trajectory point set; Performing a temporal correlation analysis on the fingertip trajectory point set to obtain a motion trajectory vector field, and performing a spatial interpolation operation based on the depth gradient sequence to obtain a continuous depth change map; The continuous depth change map is subjected to spatiotemporal filtering based on the motion trajectory vector field to obtain a fingertip distance change curve.

[0041] Specifically, in the deep infrared fusion desktop projection touch method, spatial distance analysis of the projected desktop area based on an enhanced 3D hand model to generate a fingertip distance change curve is a key step in achieving accurate touch detection. Taking a teaching scenario where a teacher operates a courseware on a projected desktop as an example, the implementation process of this step is detailed below. First, a spatial coordinate transformation is performed on the enhanced 3D hand model to obtain a hand depth map in the projection plane coordinate system. Since the enhanced 3D hand model is constructed in 3D space, while the projected desktop is a 2D plane, the spatial coordinate transformation can convert the 3D model to a 2D coordinate system corresponding to the projected desktop, placing the hand model and the projected desktop in the same reference system. For example, by converting the coordinates of each hand point in the 3D model from the world coordinate system to the projection plane coordinate system, the hand depth map now intuitively displays the projection shape of the hand on the projected desktop and the depth information of each part. Next, Gaussian curvature is calculated on the hand depth map to obtain a set of local surface features. Gaussian curvature describes the degree of curvature of a surface. In a hand depth map, different parts exhibit varying degrees of curvature, such as finger joints, which exhibit greater curvature. By calculating Gaussian curvature, these curvature features can be quantified, forming a local surface feature set that provides a basis for subsequent region segmentation. The hand depth map is then segmented based on the local surface feature set to produce a fingertip candidate region map. Based on the curvature information of each point in the local surface feature set, regions with significant curvature variation that meet the characteristics of a fingertip are identified to form a fingertip candidate region map. For example, in a hand depth map of a teacher operating on a projection table, analyzing the local surface feature set allows identification of areas with significant curvature, such as finger joints, thereby screening out regions that may be fingertips. Principal curvature analysis is then performed on the fingertip candidate region map to produce a spatial curvature descriptor. Principal curvature analysis further determines the curvature of each point in the fingertip candidate region in different directions, thereby more accurately describing the spatial shape characteristics of the fingertip and forming a spatial curvature descriptor. Next, the fingertip candidate region map is radially scanned using a polar coordinate transformation to produce a depth gradient sequence. Polar coordinate transformation enables the scanning process to radiate from the center to the surrounding areas, and collect all-round depth information of the fingertip candidate area map. During the scanning process, the depth changes at each position are recorded to form a depth gradient sequence. For example, when performing radial scanning from the center of the fingertip to the edge, the depth information will change as the distance increases, and these change data constitute the depth gradient sequence. Depth threshold stratification is performed based on the spatial curvature descriptor to obtain a fingertip trajectory point set. According to the fingertip shape characteristics reflected by the spatial curvature descriptor, different depth thresholds are set, and the data in the depth gradient sequence is layered to filter out points that meet the fingertip trajectory to form a fingertip trajectory point set. Afterwards, the fingertip trajectory point set is subjected to time series correlation analysis to obtain the motion trajectory vector field.Since hand manipulation on the projected tabletop is a dynamic process, temporal correlation analysis can correlate fingertip trajectory points at different moments in time, determining the direction and velocity of each point, thereby forming a motion trajectory vector field. Simultaneously, spatial interpolation is performed based on the depth gradient sequence to produce a continuous depth change map. Spatial interpolation allows for reasonable estimation between known depth gradient sequence data points, supplementing missing depth information and making depth changes more continuous and smooth, thus forming a continuous depth change map. Finally, spatiotemporal filtering is performed on the continuous depth change map based on the motion trajectory vector field to produce a fingertip distance change curve. Spatiotemporal filtering combines information from both temporal and spatial dimensions. It processes the data in the continuous depth change map based on the fingertip motion information in the motion trajectory vector field, removing noise and unnecessary fluctuations, highlighting the actual changes in the distance between the fingertip and the projected tabletop over time, and ultimately producing the fingertip distance change curve. For example, when a teacher clicks on a courseware icon on the projected desktop, the fingertip distance change curve can clearly show the distance change of the fingertip from being away from the desktop to touching the desktop and then leaving the desktop, providing accurate data support for subsequent judgment of whether the touch threshold is met and generating touch event judgment results, thereby achieving precise teaching interactive operations.

[0042] The above describes the desktop projection touch method of deep infrared fusion in the embodiment of the present invention. The following describes the desktop projection touch system of deep infrared fusion in the embodiment of the present invention. Figure 2 In one embodiment of the present invention, a desktop projection touch control system with deep infrared fusion includes: The acquisition module 21 is used to synchronously acquire images of the projection desktop area through a depth camera to obtain RGB image data, infrared image data, and depth image data; An extraction module 22 is used to perform hand region segmentation, recognition, and extraction on the RGB image data to obtain a hand region coordinate set and finger posture parameters; An enhancement module 23 is configured to perform edge enhancement processing on the hand region coordinate set and the finger posture parameters based on the infrared image data to obtain an enhanced finger contour feature map; an analysis module 24 for constructing an enhanced three-dimensional hand model based on the depth image data and the enhanced finger contour feature map by weight, and performing spatial distance analysis on the projected desktop area based on the enhanced three-dimensional hand model to obtain a fingertip distance change curve; The judgment module 25 is configured to generate a touch event judgment result when the fingertip distance variation curve meets a preset touch threshold.

[0043] In this embodiment, for the specific implementation of each unit in the above system embodiment, please refer to the above method embodiment, which will not be repeated here.

[0044] Reference Figure 3 In an embodiment of the present invention, a computer device is also provided, wherein the internal structure of the computer device can be as follows: Figure 3 As shown. The computer device includes a processor, memory, display screen, input device, network interface and database connected via a system bus. The processor of the computer design is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store the corresponding data in this embodiment. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the above method is implemented.

[0045] Those skilled in the art will understand that Figure 3 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention and does not constitute a limitation on the computer device to which the solution of the present invention is applied.

[0046] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which implements the above-described method when executed by a processor. It is understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0047] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware using a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the above-described method embodiments. Any reference to memory, storage, database, or other media provided herein and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-speed SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM.

[0048] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, apparatus, article, or method comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, apparatus, article, or method. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, apparatus, article, or method comprising the element.

[0049] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A desktop projection touch method with deep infrared fusion, characterized in that: The following steps are involved: The projected desktop area is synchronously imaged using a depth camera to obtain RGB image data, infrared image data, and depth image data; Performing hand region segmentation, recognition, and extraction on the RGB image data to obtain a hand region coordinate set and finger posture parameters; Performing edge enhancement processing on the hand region coordinate set and the finger posture parameters based on the infrared image data to obtain an enhanced finger contour feature map; Constructing an enhanced three-dimensional hand model based on the depth image data and the enhanced finger contour feature map by weighting, and performing spatial distance analysis on the projected desktop area based on the enhanced three-dimensional hand model to obtain a fingertip distance change curve; When the fingertip distance change curve meets the preset touch threshold, a touch event determination result is generated.

2. The desktop projection touch control method of deep infrared fusion according to claim 1, characterized in that: The hand region segmentation, recognition and extraction of the RGB image data to obtain a hand region coordinate set and finger posture parameters includes: Performing YCbCr color space conversion on the RGB image data to obtain a skin color candidate area map, and performing Sobel operator calculation on the skin color candidate area map to obtain an edge gradient map; Reconstructing the contour of the skin color candidate area map by an ellipse fitting operation to obtain a hand area mask, and performing convex hull analysis based on the edge gradient map to obtain a region boundary point set; Performing watershed segmentation based on the hand region mask to obtain a finger region sub-atlas, and iteratively refining the region boundary point set to obtain a skeleton feature map; The extreme points of the finger region sub-atlas are extracted by distance transformation to obtain a hand region coordinate set, and angle projection is performed based on the skeleton feature map to obtain finger posture parameters.

3. The desktop projection touch control method of deep infrared fusion according to claim 1, characterized in that: The step of performing edge enhancement processing on the hand region coordinate set and the finger posture parameters based on the infrared image data to obtain an enhanced finger contour feature map includes: Performing heat distribution mapping on the infrared image data to obtain a heat zone intensity map, and performing anisotropic diffusion on the heat zone intensity map to obtain a temperature field distribution map; Performing regional projection on the temperature field distribution map based on the hand region coordinate set to obtain an infrared edge contour map, and performing boundary mapping according to the thermal zone intensity map to obtain a thermal zone overlap map; Performing multi-scale edge operations on the infrared edge contour map to obtain an enhanced edge map, and performing wavelet decomposition on the hot zone overlap map to obtain a contour mutation point set; Performing region adaptive threshold segmentation on the enhanced edge map based on the finger posture parameters to obtain a fingertip hot zone feature map, and performing curve fitting on the contour mutation point set to obtain a hot zone contour descriptor; Perform bilateral filtering on the fingertip hot zone feature map and the hot zone contour descriptor to obtain an enhanced finger contour feature map.

4. The desktop projection touch control method of deep infrared fusion according to claim 3, characterized in that: The performing regional projection on the temperature field distribution map based on the hand region coordinate set to obtain an infrared edge contour map includes: Performing polar coordinate transformation on the hand region coordinate set to obtain a hand region polar coordinate mapping diagram, and performing Gaussian curvature calculation on the temperature field distribution diagram to obtain a temperature field curvature distribution diagram; performing a regional mask operation on the temperature field curvature distribution map based on the hand region polar coordinate mapping map to obtain a hand thermal zone gradient map, and performing a Laplace transform on the hand thermal zone gradient map to obtain a thermal zone boundary enhancement map; Performing arc detection on the hot zone boundary enhancement image through Hough circle transform to obtain a fingertip arc feature set, and performing region growing segmentation based on the fingertip arc feature set to obtain a fingertip hot zone contour image; The fingertip hot zone contour map is subjected to morphological refinement processing to obtain a fingertip edge skeleton map, and a B-spline curve fitting is performed based on the fingertip edge skeleton map to obtain an infrared edge contour map.

5. The desktop projection touch control method of deep infrared fusion according to claim 1, characterized in that: The step of performing polar coordinate transformation on the hand region coordinate set to obtain a hand region polar coordinate mapping diagram includes: Performing centroid positioning calculation on the hand region coordinate set to obtain a hand region centroid coordinate point set, and performing radial distance calculation on the hand region centroid coordinate point set to obtain a hand region radial distribution map; performing angle quantization segmentation on the hand region coordinate set based on the hand region radial distribution map to obtain a hand region angle distribution sequence, and performing polar coordinate grid division on the hand region angle distribution sequence to obtain a hand region grid mapping matrix; performing radial normalization processing on the hand region grid mapping matrix to obtain a normalized polar coordinate distribution map, and performing polar coordinate interpolation operation based on the normalized polar coordinate distribution map to obtain a hand region polar coordinate density map; Performing polar coordinate gradient calculation based on the polar coordinate density map of the hand region to obtain a polar coordinate gradient field of the hand region, and performing polar coordinate boundary extraction on the polar coordinate gradient field of the hand region to obtain a polar coordinate boundary set of the hand region; Polar coordinate smoothing is performed on the hand region polar coordinate boundary set to obtain a hand region polar coordinate contour map, and polar coordinate resampling is performed based on the hand region polar coordinate contour map to obtain a hand region polar coordinate mapping map.

6. The desktop projection touch control method of deep infrared fusion according to claim 5, characterized in that: The polar coordinate gradient calculation is performed based on the polar coordinate density map of the hand area to obtain the polar coordinate gradient field of the hand area, including: Performing radial difference operation on the polar coordinate density map of the hand region to obtain a radial gradient component map, and performing angular projection transformation on the radial gradient component map to obtain a polar coordinate directional gradient matrix; Performing polar coordinate spatial filtering on the polar coordinate density map of the hand region based on the polar coordinate directional gradient matrix to obtain a polar coordinate gradient intensity distribution map, and performing polar coordinate tensor decomposition on the polar coordinate gradient intensity distribution map to obtain a polar coordinate principal direction field; Performing polar coordinate flow field analysis on the polar coordinate main direction field to obtain a polar coordinate streamline feature set, and performing polar coordinate curvature calculation based on the polar coordinate streamline feature set to obtain a polar coordinate curvature distribution diagram; The polar coordinate vector field of the polar coordinate streamline feature set is reconstructed based on the polar coordinate curvature distribution map to obtain a polar coordinate gradient field of the hand region.

7. The desktop projection touch control method of deep infrared fusion according to claim 1, characterized in that: The performing of spatial distance analysis on the projection desktop area based on the enhanced three-dimensional hand model to obtain a fingertip distance change curve includes: Performing spatial coordinate transformation on the enhanced three-dimensional hand model to obtain a hand depth map in a projection plane coordinate system, and performing Gaussian curvature calculation on the hand depth map to obtain a local surface feature set; Performing region segmentation on the hand depth map based on the local surface feature set to obtain a fingertip candidate region map, and performing principal curvature analysis on the fingertip candidate region map to obtain a spatial curvature descriptor; Performing radial scanning on the fingertip candidate area map through polar coordinate transformation to obtain a depth gradient sequence, and performing depth threshold stratification based on the spatial curvature descriptor to obtain a fingertip trajectory point set; Performing a temporal correlation analysis on the fingertip trajectory point set to obtain a motion trajectory vector field, and performing a spatial interpolation operation based on the depth gradient sequence to obtain a continuous depth change map; The continuous depth change map is subjected to spatiotemporal filtering based on the motion trajectory vector field to obtain a fingertip distance change curve, which includes the fingertip spatial position, motion speed and acceleration parameters.

8. A deep infrared fusion desktop projection touch system, characterized in that: include: The acquisition module is used to synchronously acquire images of the projected desktop area through a depth camera to obtain RGB image data, infrared image data, and depth image data; An extraction module is used to perform hand region segmentation, recognition, and extraction on the RGB image data to obtain a hand region coordinate set and finger posture parameters; an enhancement module, configured to perform edge enhancement processing on the hand region coordinate set and the finger posture parameters based on the infrared image data to obtain an enhanced finger contour feature map; an analysis module, configured to construct an enhanced three-dimensional hand model based on the depth image data and the enhanced finger contour feature map by weight, and perform spatial distance analysis on the projected desktop area based on the enhanced three-dimensional hand model to obtain a fingertip distance change curve; The judgment module is configured to generate a touch event judgment result when the fingertip distance change curve meets a preset touch threshold.

9. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images