A planar multi-finger touch detection system, method, apparatus and medium

Abstract

The application provides a planar multi-finger touch detection system, method, device and medium. Light emitted by a light source module capable of being fixed on a user's wrist is irradiated on the user's fingers. In the case that the user's fingers touch a physical plane, a camera on a mixed reality device can collect a finger touch image sequence when the user touches the physical plane, so as to obtain a projection area of a gap between a finger tip in each finger touch image in the finger touch image sequence and the physical plane on the physical plane, and detect the case that the user's fingers touch the physical plane by using the projection area of the gap between the finger tip in each finger touch image and the physical plane on the physical plane. Since the projection area of the gap between the finger tip and the physical plane on the physical plane is larger and clearer than an image of the gap between the finger tip and the physical plane directly collected by the camera, the accuracy of detecting the case that the user's fingers touch the physical plane can be increased.

Description

Technical Field

[0001] This application relates to the field of virtual interaction technology, and more specifically, to a planar multi-finger touch detection system, method, apparatus, and medium. Background Technology

[0002] Currently, in mixed reality hand interaction spaces, users need to touch physical surfaces. After the mixed reality device captures the user's finger touch image, it identifies the user's finger touch pattern in the image and executes instructions matching the identified touch pattern within the mixed reality environment to complete the hand interaction. The identification of user finger touch patterns primarily relies on computer vision methods. This method places high demands on the computing power of the mixed reality device and suffers from insufficient accuracy. Summary of the Invention

[0003] To address the aforementioned issues, the purpose of this application is to provide a planar multi-finger touch detection system, method, apparatus, and medium.

[0004] In a first aspect, embodiments of this application provide a planar multi-finger touch detection system, including: a light source module and a mixed reality device;

[0005] The light source module is fixed to the user's wrist, with the light-emitting side of the light source module facing the user's fingers after fixing; a camera is provided on the mixed reality device;

[0006] When a user needs to interact with the mixed reality device, the light source module is turned on, and the light emitted by the light source module shines on the user's finger.

[0007] When a user touches a physical plane with their finger, the camera can capture a sequence of finger touch images and feed the captured finger touch image sequence back to the mixed reality device; wherein, the finger touch image sequence includes: multiple consecutive finger touch images;

[0008] The mixed reality device is used to process the multiple finger touch images and detect when the user's finger touches a physical plane.

[0009] Secondly, embodiments of this application also provide a planar multi-finger touch detection method for performing the functions implemented by the mixed reality device in the above-described planar multi-finger touch detection system, the method comprising:

[0010] Extract the fingertip region from each of the aforementioned finger touch images;

[0011] Based on the fingertip regions extracted from each of the finger touch images, each of the finger touch images is cropped to obtain a cropped image sequence of each fingertip; wherein, each cropped image of each fingertip in the cropped image sequence of each fingertip includes: the fingertip region of each finger, the projection region of each fingertip on the solid plane, and the projection region of the gap between each fingertip and the solid plane on the solid plane.

[0012] The system detects when a user's fingers touch a physical plane by using a sequence of cropped images of each fingertip.

[0013] Thirdly, embodiments of this application also provide a planar multi-finger touch detection device, comprising:

[0014] The extraction module is used to extract the fingertip area from each of the finger touch images;

[0015] The cropping module is used to crop each finger touch image based on the fingertip region extracted from each finger touch image to obtain a cropped image sequence of each fingertip; wherein, the cropped image of each fingertip in the cropped image sequence of each fingertip includes: the fingertip region of each finger, the projection region of each fingertip on the solid plane, and the projection region of the gap between each fingertip and the solid plane on the solid plane;

[0016] The detection module is used to detect when the user's fingers touch a physical plane by using a sequence of cropped images of the fingertips of each finger.

[0017] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the method described in any one of claims 6-8.

[0018] In the solutions provided by the first to fourth aspects of this application, a light source module that can be fixed on the user's wrist is set in the planar multi-finger touch detection system. When the user needs to interact with the mixed reality device, the light emitted by the light source module will illuminate the user's fingers. When the user's fingers touch the solid plane, the camera on the mixed reality device can capture a sequence of finger touch images when the user touches the solid plane. This allows the system to obtain the projection area of ​​the gap between the fingertip and the solid plane in each finger touch image in the sequence, and to utilize the gap between the fingertip and the solid plane in each finger touch image. The gap between the fingertip and the solid plane is projected onto the solid plane to detect the user's finger touching the solid plane. Compared with related technologies that rely solely on computer vision methods to detect finger touching the solid plane, the projection area of ​​the gap between the fingertip and the solid plane is larger and clearer than the image of the gap between the fingertip and the solid plane directly captured by the camera. Therefore, using the projection area of ​​the gap between the fingertip and the solid plane can better detect the user's finger touching the solid plane without increasing the computing power of the mixed reality device, thus increasing the accuracy of the detection.

[0019] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A schematic diagram of the planar multi-finger touch detection system provided in Embodiment 1 of this application is shown;

[0022] Figure 2 A schematic diagram of the clamping member provided in Embodiment 1 of this application is shown;

[0023] Figure 3 A flowchart of a planar multi-finger touch detection method provided in Embodiment 2 of this application is shown;

[0024] Figure 4 This paper shows a schematic diagram of the structure of a planar multi-finger touch detection device provided in Embodiment 3 of this application;

[0025] Figure 5 A schematic diagram of the structure of an electronic device provided in Embodiment 4 of this application is shown. Detailed Implementation

[0026] In the description of this application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0027] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0028] In this application, unless otherwise expressly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0029] Currently, in mixed reality hand interaction spaces, users need to touch physical surfaces. After the mixed reality device captures the user's finger touch image, it identifies the user's finger touch pattern in the image and executes instructions matching the identified touch pattern within the mixed reality environment to complete the hand interaction. The identification of user finger touch patterns primarily relies on computer vision methods. This method places high demands on the computing power of the mixed reality device and suffers from insufficient accuracy.

[0030] Currently, in mixed reality hand interaction spaces, users need to touch physical surfaces. After the mixed reality device captures the user's finger touch image, it identifies the user's finger touch pattern in the image and executes instructions matching the identified touch pattern within the mixed reality environment to complete the hand interaction. The identification of user finger touch patterns primarily relies on computer vision methods. This method places high demands on the computing power of the mixed reality device and suffers from insufficient accuracy.

[0031] Based on this, this embodiment proposes a planar multi-finger touch detection system, method, device, and medium. By incorporating a light source module that can be fixed to the user's wrist within the planar multi-finger touch detection system, when the user needs to interact with the mixed reality device, the light emitted by the light source module illuminates the user's fingers. When the user's fingers touch the solid plane, the camera on the mixed reality device can capture a sequence of finger touch images. This allows for the determination of the gap between the fingertip and the solid plane in each of the multiple finger touch images within the sequence. The system projects the user's finger onto the physical plane using the projection area of ​​the gap between the fingertip and the physical plane in each touch image. Since the projection area of ​​the gap between the fingertip and the physical plane is larger and clearer than the image of the gap between the fingertip and the physical plane directly captured by the camera, it is possible to better detect the user's finger onto the physical plane without increasing the computing power of the mixed reality device, thus increasing the accuracy of the detection.

[0032] Before introducing the planar multi-finger touch detection system, method, apparatus, and medium proposed in this embodiment, the principle of the light source module will be explained first. (See [link to relevant documentation]). Figure 1 The diagram shows the structure of a planar multi-finger touch detection system. When a finger touches a solid plane, the light source module projects light onto the finger. The light source module is fixed to the user's wrist and emits light to illuminate the finger and its surrounding area. When the fingertip touches the solid plane, some of the light projected onto the plane is blocked by the finger, creating a shadow in front of the finger on the solid plane. This shadow represents the projection area of ​​the gap between the fingertip and the solid plane onto the solid plane. Assume the distance between the light source module and the solid plane is h. L The distance between the fingertip and the solid plane is h. F The resulting gap between the fingertip and the solid plane is the distance d between the end of the fingertip closest to the fingertip and the projection position of the fingertip on the solid plane within the projection area of ​​the solid plane. S It can be calculated using the following formulas 1 and 2:

[0033]

[0034]

[0035] Among them, h L h is a known quantity. F and d S d is an unknown quantity; H This d represents the projection distance between the light source module and the fingertip onto the solid plane. H It is related to the size of the hand and the posture of the hand when touching the solid surface, d H Typically, it is a known quantity within a fixed range (e.g., 10 cm to 20 cm); r mag This represents the projection magnification, a value obtained from experiments, and is a known quantity.

[0036] h can be calculated using formulas 1 and 2 above. F and d S This cleverly establishes h, which is difficult to observe in the vertical direction. F A relatively large distance d that is easily observable by the camera S The connection is that the image of the gap between the fingertip and the solid plane, captured by the camera of the mixed reality device, is projected onto the solid plane to detect the finger touching the solid plane. This reduces the computing power used by the mixed reality device to detect the finger touching the solid plane. Moreover, it can improve the spatial resolution of the contact state judgment of an independent finger near the solid plane, increasing the spatial resolution of the judgment from 10 mm to 2 mm, which is close to the limit resolution controlled by the user, and greatly improves the detection accuracy of the finger touching the solid plane.

[0037] This physical plane should be aligned with the virtual interactive interface projected by the VR / AR system, thus endowing any non-intelligent spatial surface with intelligent touch interaction capabilities. This allows users to operate and interact with the mixed reality device by performing actions (such as trajectory operations and click operations) on the virtual interactive interface with their fingers.

[0038] This physical surface can be any non-smart physical surface, including but not limited to: desktops, touchpads, photo albums, drawing boards, and text editors. This transforms any non-smart physical surface into a "touchscreen".

[0039] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and embodiments.

[0040] Example 1

[0041] like Figure 1 As shown, this embodiment 1 proposes a planar multi-finger touch detection system, including: a light source module 100 and a mixed reality device 102. The light source module 100 is fixed on the user's wrist, with the light-emitting side of the fixed light source module 100 facing the user's fingers; the mixed reality device 102 is equipped with a camera 1020.

[0042] Here, the mixed reality device can be replaced with a VR device or an AR device. This embodiment does not impose any limitations.

[0043] The mixed reality device projects a virtual interactive interface onto a physical plane. When a user needs to interact with the mixed reality device 102, the light source module 100 is turned on, and the light emitted by the light source module 100 shines on the user's fingers.

[0044] The switching control of the light source module 100 can be achieved through a mixed reality device or by setting a separate switch; this embodiment does not impose any limitations.

[0045] When a user touches a physical plane with their finger, the camera 1020 can capture a sequence of finger touch images when the user touches the physical plane, and feed the captured finger touch image sequence back to the mixed reality device 102; wherein, the finger touch image sequence includes: multiple consecutive finger touch images.

[0046] Specifically, the light source module 100 includes: a flexible rubber ring, an electroluminescent light source, and a clamping component.

[0047] The flexible rubber ring is fitted onto the user's wrist, the clamping member is fixed to the flexible rubber ring, and the electroluminescent light source is fixed to the clamping member.

[0048] Here, the flexible rubber ring can be a smart bracelet, or any flexible rubber ring that can be worn on the hand; we will not go into detail about them here.

[0049] In one embodiment, the electroluminescent light source includes, but is not limited to, light-emitting diodes and organic light-emitting diodes.

[0050] In one implementation, see Figure 2 The diagram shows the structure of the clamping component, which is a "Y" shaped structure.

[0051] The mixed reality device 102 is used to process the multiple finger touch images and detect when the user's finger touches a physical plane.

[0052] Specifically, the mixed reality device is used to process the multiple finger touch images and detect the user's finger touching a physical plane, including the following steps (1) to (3):

[0053] (1) Extract the fingertip area from each of the finger touch images;

[0054] (2) Based on the fingertip region in each of the extracted finger touch images, each of the finger touch images is cropped to obtain a cropped image sequence of each fingertip; wherein, the cropped image of each fingertip in the cropped image sequence of each fingertip includes: the fingertip region of each finger, the projection region of each fingertip on the solid plane, and the projection region of the gap between each fingertip and the solid plane on the solid plane;

[0055] (3) The user’s finger touches the physical plane by using the cropped image sequence of each fingertip.

[0056] In step (1) above, the mixed reality device is used to extract the fingertip region from each of the finger touch images, including:

[0057] The finger touch images are processed using a hand tracking model to extract the fingertip region from each finger touch image.

[0058] This hand tracking model includes, but is not limited to, the MediaPipe model and the InterHand model.

[0059] In step (2) above, the fingertips include: the fingertip of the thumb, the fingertip of the index finger, the fingertip of the middle finger, the fingertip of the ring finger, and the fingertip of the little finger.

[0060] The specific process of cropping each finger touch image based on the fingertip area extracted from each finger touch image is existing technology and will not be described in detail here.

[0061] The cropped image sequence of each fingertip includes: the fingertip of each finger in each finger touch image and the area around the fingertip.

[0062] In step (3) above, each cropped image sequence of fingertips carries a unique hot code for that fingertip. This unique hot code is used to represent the finger identifier of the finger to which the cropped image sequence of each fingertip belongs.

[0063] The aforementioned finger markings include: thumb, index finger, middle finger, ring finger, and little finger.

[0064] Specifically, the mixed reality device is used to detect when the user's finger touches a physical plane using a sequence of cropped images of the fingertips, including the following specific steps:

[0065] (1) Input the cropped image sequence of each fingertip into the trained convolutional neural network to obtain the image features of the cropped image sequence of each fingertip; the image features of the cropped image sequence of each fingertip include: the feature vector of the projection area of ​​the gap between each fingertip and the solid plane on the solid plane.

[0066] (2) Input the image features and the unique hot code of the cropped image sequence of each fingertip into the multilayer sensor to detect the situation of each fingertip touching the physical plane.

[0067] In step (1) above, the specific process of inputting the cropped image sequence of each fingertip into the trained convolutional neural network to obtain the image features of the cropped image sequence of each fingertip is existing technology and will not be described in detail here.

[0068] In step (2) above, the specific process of inputting the image features and the unique hot code of the cropped image sequence of each fingertip into the multilayer perceptron to detect the situation of each fingertip touching the physical plane is existing technology and will not be described in detail here.

[0069] Since the image features of the cropped image sequence of each fingertip include the feature vector of the projection area of ​​the gap between each fingertip and the solid plane on the solid plane, a multilayer perceptron can be used to process the feature vector of the projection area of ​​the gap between each fingertip and the solid plane on the solid plane. This process can replace the processing of the feature vector of the gap between each fingertip and the solid plane in related technologies. Because the projection area of ​​the gap between the fingertip and the solid plane on the solid plane is larger and clearer than the image of the gap between the fingertip and the solid plane directly captured by the camera, it is simpler and easier to operate to detect the user's touch on the solid plane using the projection area of ​​the gap between the fingertip and the solid plane. This achieves the goal of better detecting the user's finger touch on the solid plane without increasing the computing power of the mixed reality device, thus increasing the accuracy of the detection.

[0070] The planar multi-finger touch detection system proposed in this embodiment can accurately detect single-finger touch and multi-finger touch, thereby achieving accurate and sensitive multi-finger planar interaction, enriching the way users interact with mixed reality devices, and improving user experience.

[0071] In summary, this embodiment proposes a planar multi-finger touch detection system. By incorporating a light source module that can be fixed to the user's wrist within the system, when the user needs to interact with the mixed reality device, the light emitted by the light source module illuminates the user's fingers. When the user's fingers touch the solid plane, the camera on the mixed reality device can capture a sequence of finger touch images. This allows for the determination of the projection area of ​​the gap between the fingertip and the solid plane in each of the multiple finger touch images within the sequence. Furthermore, the system utilizes the gap between the fingertip and the solid plane in each touch image to determine the optimal system for detecting finger touches. The gap between the fingertip and the solid plane is projected onto the solid plane to detect the user's finger touching the solid plane. Compared with related technologies that rely solely on computer vision methods to detect finger touching the solid plane, the projection area of ​​the gap between the fingertip and the solid plane is larger and clearer than the image of the gap between the fingertip and the solid plane directly captured by the camera. Therefore, using the projection area of ​​the gap between the fingertip and the solid plane can better detect the user's finger touching the solid plane without increasing the computing power of the mixed reality device, thus increasing the accuracy of the detection.

[0072] Example 2

[0073] See Figure 3 The illustrated planar multi-finger touch detection method is used to perform the functions implemented by the mixed reality device in the planar multi-finger touch detection system described in Embodiment 1 above. The method includes the following specific steps:

[0074] Step 200: Extract the fingertip area from each of the finger touch images.

[0075] In step 200 above, extracting the fingertip region from each of the finger touch images includes:

[0076] The finger touch images are processed using a hand tracking model to extract the fingertip region from each finger touch image.

[0077] Step 202: Based on the fingertip regions extracted from each of the finger touch images, crop each of the finger touch images to obtain a cropped image sequence of each fingertip; wherein, the cropped image of each fingertip in the cropped image sequence of each fingertip includes: the fingertip region of each finger, the projection region of each fingertip on the solid plane, and the projection region of the gap between each fingertip and the solid plane on the solid plane.

[0078] Step 204: Detect the user's finger touching a physical plane using a sequence of cropped images of each fingertip.

[0079] Here, each cropped image sequence of fingertips carries a unique hot code for the cropped image sequence of each fingertips; the unique hot code is used to represent the finger identifier of the finger to which the cropped image sequence of each fingertips belongs.

[0080] In step 204 above, the step of detecting the user's finger touching a physical plane using a cropped image sequence of each fingertip includes:

[0081] The cropped image sequence of each fingertip is input into a trained convolutional neural network to obtain the image features of the cropped image sequence of each fingertip; the image features of the cropped image sequence of each fingertip include: the feature vector of the projection area of ​​the gap between each fingertip and the solid plane on the solid plane.

[0082] The image features and the unique hot code of the cropped image sequence of each fingertip are respectively input into a multilayer perceptron to detect the situation of each fingertip touching a solid plane.

[0083] In summary, this embodiment proposes a planar multi-finger touch detection method. It detects user finger touches on a physical plane by projecting the gap between the fingertip and the physical plane onto the physical plane in the touch image. Compared to related technologies that rely solely on computer vision methods, this method utilizes the larger and clearer projection area of ​​the gap between the fingertip and the physical plane compared to images directly captured by a camera. Therefore, without increasing the computing power of the mixed reality device, it can better detect user finger touches on a physical plane, increasing the accuracy of the detection.

[0084] Example 3

[0085] This embodiment proposes a planar multi-finger touch detection device for performing the planar multi-finger touch detection method proposed in Embodiment 2 above.

[0086] See Figure 4 The diagram shows a planar multi-finger touch detection device. This embodiment proposes a planar multi-finger touch detection device, comprising:

[0087] The extraction module 300 is used to extract the fingertip region from each of the finger touch images;

[0088] The cropping module 302 is used to crop each finger touch image based on the fingertip region extracted from each finger touch image to obtain a cropped image sequence of each fingertip; wherein, the cropped image of each fingertip in the cropped image sequence of each fingertip includes: the fingertip region of each finger, the projection region of each fingertip on the solid plane, and the projection region of the gap between each fingertip and the solid plane on the solid plane;

[0089] The detection module 304 is used to detect the user's finger touching a physical plane by using a sequence of cropped images of the fingertips of each finger.

[0090] In summary, this embodiment proposes a planar multi-finger touch detection device. It detects user finger touches on a physical plane by projecting the gap between the fingertip and the physical plane onto the physical plane in the touch image. Compared to related technologies that rely solely on computer vision methods, this method utilizes the larger and clearer projection area of ​​the gap between the fingertip and the physical plane compared to images directly captured by a camera. Therefore, without increasing the computing power of the mixed reality device, it can better detect user finger touches on a physical plane, increasing the accuracy of the detection.

[0091] Example 4

[0092] This embodiment proposes a computer-readable storage medium storing a computer program. When the computer program is run by a processor, it executes the steps of the planar multi-finger touch detection method described in Embodiment 2 above. For specific implementation details, please refer to Method Embodiment 1, which will not be repeated here.

[0093] In addition, see Figure 5 The diagram shows the structure of an electronic device. This embodiment also proposes an electronic device, which includes a bus 51, a processor 52, a transceiver 53, a bus interface 54, a memory 55, and a user interface 56. The electronic device includes a memory 55.

[0094] In this embodiment, the electronic device further includes: one or more programs stored in the memory 55 and executable on the processor 52, configured to be executed by the processor to perform the one or more programs for the following steps (1) to (3):

[0095] (1) Extract the fingertip area from each of the finger touch images;

[0096] (2) Based on the fingertip region in each of the extracted finger touch images, each of the finger touch images is cropped to obtain a cropped image sequence of each fingertip; wherein, the cropped image of each fingertip in the cropped image sequence of each fingertip includes: the fingertip region of each finger, the projection region of each fingertip on the solid plane, and the projection region of the gap between each fingertip and the solid plane on the solid plane;

[0097] (3) The user’s finger touches the physical plane by using the cropped image sequence of each fingertip.

[0098] Transceiver 53 is used to receive and send data under the control of processor 52.

[0099] The bus architecture (represented by bus 51) can include any number of interconnected buses and bridges, linking various circuits including one or more processors represented by processor 52 and memory represented by memory 55. Bus 51 can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be further described in this embodiment. Bus interface 54 provides an interface between bus 51 and transceiver 53. Transceiver 53 can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. For example, transceiver 53 receives external data from other devices. Transceiver 53 is used to transmit data processed by processor 52 to other devices. Depending on the nature of the computing system, a user interface 56 may also be provided, such as a keypad, display, speaker, microphone, or joystick.

[0100] Processor 52 is responsible for managing bus 51 and general processing, such as running general-purpose operating system 551 as described above. Memory 55 can be used to store data used by processor 52 during operation.

[0101] Optionally, the processor 52 may be, but is not limited to, a central processing unit, a microcontroller, a microprocessor, or a programmable logic device.

[0102] It is understood that the memory 55 in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 55 of the systems and methods described in this embodiment is intended to include, but is not limited to, these and any other suitable types of memory.

[0103] In some implementations, memory 55 stores elements such as executable modules or data structures, or subsets thereof, or extended sets thereof: operating system 551 and application programs 552.

[0104] The operating system 551 includes various system programs, such as the framework layer, core library layer, and driver layer, used to implement various basic business functions and handle hardware-based tasks. The application program 552 includes various applications, such as a media player and a browser, used to implement various application functions. Programs implementing the methods of the embodiments of this application can be included in the application program 552.

[0105] In summary, this embodiment proposes a computer-readable storage medium and electronic device that detects user finger touches on a physical plane by projecting the gap between the fingertip and the physical plane in the physical plane image onto the physical plane. Compared with related technologies that rely solely on computer vision methods to detect finger touches on a physical plane, the projection area of ​​the gap between the fingertip and the physical plane on the physical plane is larger and clearer than the image of the gap directly captured by the camera. Therefore, utilizing the projection area of ​​the gap between the fingertip and the physical plane on the physical plane can better detect user finger touches on a physical plane without increasing the computing power of the mixed reality device, thus increasing the accuracy of the detection.

[0106] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A planar multi-finger touch detection system, characterized in that, include: Light source modules and mixed reality devices; The light source module is fixed to the user's wrist, with the light-emitting side of the light source module facing the user's fingers after fixing; a camera is provided on the mixed reality device; The mixed reality device projects a virtual interactive interface onto a physical plane. When a user needs to interact with the mixed reality device, the light source module is turned on, and the light emitted by the light source module shines on the user's fingers. When a user touches the physical plane with their finger, the camera can capture a sequence of finger touch images and feed the captured finger touch image sequence back to the mixed reality device; wherein, the finger touch image sequence includes: multiple consecutive finger touch images; The mixed reality device is used to process the multiple finger touch images and detect when the user's finger touches a physical plane; The mixed reality device is used to process the multiple finger touch images and detect when the user's finger touches a physical plane, including: Extract the fingertip region from each of the aforementioned finger touch images; Based on the fingertip regions extracted from each of the finger touch images, each of the finger touch images is cropped to obtain a cropped image sequence of each fingertip; wherein, each cropped image of each fingertip in the cropped image sequence of each fingertip includes: the fingertip region of each finger, the projection region of each fingertip on the solid plane, and the projection region of the gap between each fingertip and the solid plane on the solid plane. The system detects when a user's fingers touch a physical plane by using a sequence of cropped images of each fingertip. Each cropped image sequence of fingertips carries a unique hot code for the cropped image sequence of each fingertips; the unique hot code is used to represent the finger identifier of the finger to which the cropped image sequence of each fingertips belongs. The mixed reality device is used to detect when the user's fingers touch a physical plane using a sequence of cropped images of the fingertips, including: The cropped image sequence of each fingertip is input into a trained convolutional neural network to obtain the image features of the cropped image sequence of each fingertip; the image features of the cropped image sequence of each fingertip include: the feature vector of the projection area of ​​the gap between each fingertip and the solid plane on the solid plane. The image features and the unique hot code of the cropped image sequence of each fingertip are respectively input into a multilayer perceptron to detect the situation of each fingertip touching a solid plane.

2. The system according to claim 1, characterized in that, The mixed reality device is used to extract the fingertip region from each of the finger touch images, including: The finger touch images are processed using a hand tracking model to extract the fingertip region from each finger touch image.

3. The system according to claim 1, characterized in that, The light source module includes: a flexible rubber ring, an electroluminescent light source, and a clamping component; The flexible rubber ring is fitted onto the user's wrist, the clamping member is fixed to the flexible rubber ring, and the electroluminescent light source is fixed to the clamping member.

4. A planar multi-finger touch detection method, used to perform the functions implemented by the mixed reality device in the planar multi-finger touch detection system according to any one of claims 1-3, characterized in that, The method includes: Extract the fingertip region from each of the aforementioned finger touch images; Based on the fingertip regions extracted from each of the finger touch images, each of the finger touch images is cropped to obtain a cropped image sequence of each fingertip; wherein, each cropped image of each fingertip in the cropped image sequence of each fingertip includes: the fingertip region of each finger, the projection region of each fingertip on the solid plane, and the projection region of the gap between each fingertip and the solid plane on the solid plane. The system detects when a user's fingers touch a physical plane by using a sequence of cropped images of each fingertip.

5. The method according to claim 4, characterized in that, The step of extracting the fingertip region from each of the finger touch images includes: The finger touch images are processed using a hand tracking model to extract the fingertip region from each finger touch image.

6. The method according to claim 4, characterized in that, Each cropped image sequence of fingertips carries a unique hot code for the cropped image sequence of each fingertips; the unique hot code is used to represent the finger identifier of the finger to which the cropped image sequence of each fingertips belongs. The process of detecting user finger touches on a physical plane using a cropped image sequence of each fingertip includes: The cropped image sequence of each fingertip is input into a trained convolutional neural network to obtain the image features of the cropped image sequence of each fingertip; the image features of the cropped image sequence of each fingertip include: the feature vector of the projection area of ​​the gap between each fingertip and the solid plane on the solid plane. The image features and the unique hot code of the cropped image sequence of each fingertip are respectively input into a multilayer perceptron to detect the situation of each fingertip touching a solid plane.

7. A planar multi-finger touch detection device, used to perform the functions implemented by the mixed reality device in the planar multi-finger touch detection system according to any one of claims 1-3, characterized in that, include: The extraction module is used to extract the fingertip area from each finger touch image; The cropping module is used to crop each finger touch image based on the fingertip region extracted from each finger touch image to obtain a cropped image sequence of each fingertip; wherein, the cropped image of each fingertip in the cropped image sequence of each fingertip includes: the fingertip region of each finger, the projection region of each fingertip on the solid plane, and the projection region of the gap between each fingertip and the solid plane on the solid plane; The detection module is used to detect when the user's fingers touch a physical plane by using a sequence of cropped images of the fingertips of each finger.

8. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is run by the processor, it performs the steps of the method described in any one of claims 4-6.

Images