Low-power control method applied to tactile sensor, tactile sensor and dexterous hand

Abstract

This application discloses a low-power control method for a visual touch sensor, a visual touch sensor, and a dexterous hand. The low-power control method for the visual touch sensor includes: acquiring distance information of an object through the sensing module; acquiring a preset threshold; comparing the acquired object distance with the preset threshold and obtaining the comparison result; controlling the sensing module to operate according to the comparison result; wherein, when the acquired object distance is greater than the preset threshold, the sensing module is controlled to operate in depth detection mode to acquire the object distance information; when the acquired object distance is not greater than the preset threshold, the sensing module is controlled to operate in image capture mode and acquire image information. This application can solve the problem that traditional visual touch sensors need to work continuously after power-on, even when there is no contact with an object, the computing unit continues to calculate and output visual data, resulting in high system power consumption.

Description

Technical Field

[0001] This application relates to the field of optical measurement technology, and in particular to a low-power control method for visual touch sensors, a visual touch sensor, and a dexterous hand. Background Technology

[0002] A dexterous hand is a type of robotic component that simulates a human hand. A sensor, called a vision-touch sensor, is typically mounted at the fingertips of a dexterous hand. Tactile feedback has always been a crucial requirement in human-computer interaction and advanced robotics, and tactile sensors play a key role in robot operation. Vision-based tactile sensors, in particular, can convert contact characteristics into images, offering advantages such as high spatial resolution, low-cost manufacturing, and compatibility with computer vision and machine learning algorithms, thus attracting widespread interest in the robotics field. Currently, existing vision-touch sensors typically employ CMOS miniature cameras, using CMOS to capture amplitude-mask images at the front end. These images are then used to further infer the sensor's geometry and mechanical characteristics, providing real-time 3D visualization data. However, the CMOS sensor and the image-based computing unit are constantly operating, simultaneously transmitting the acquired image data to the internal processor for computation, resulting in wasted power consumption and computing resources. Summary of the Invention

[0003] The main objective of this application is to propose a low-power control method for visual touch sensors, aiming to solve the problem of high power consumption in existing visual touch sensor systems.

[0004] To achieve the above objectives, this application proposes a low-power control method for a visual touch sensor. The visual touch sensor includes a sensing module, comprising: acquiring distance information of an object through the sensing module; acquiring a preset threshold; comparing the acquired object distance with the preset threshold and acquiring the comparison result; controlling the sensing module to operate according to the comparison result; wherein, when the acquired object distance is greater than the preset threshold, the sensing module is controlled to operate in a depth detection mode to acquire the object distance information; when the acquired object distance is not greater than the preset threshold, the sensing module is controlled to operate in an image capture mode and acquire image information.

[0005] Optionally, the step of obtaining the distance information of an object through the sensing module includes: obtaining the distance information of the object multiple times through the sensing module within a preset time period; obtaining a preset threshold, comparing the obtained object distance with the preset threshold, and obtaining the comparison result includes: obtaining the preset threshold, comparing the multiple obtained object distances with the preset threshold respectively, and obtaining multiple comparison results.

[0006] Optionally, the step of controlling the operation of the perception module based on the comparison results includes: controlling the operation of the perception module based on multiple comparison results; wherein, when the object distances acquired multiple times are all greater than a preset threshold, the perception module is controlled to operate in depth detection mode to acquire distance information; when the object distances acquired multiple times are not greater than the preset threshold, the perception module is controlled to operate in image capture mode to acquire image information.

[0007] Optionally, when the acquired object distance is not greater than a preset threshold, the step of controlling the perception module to operate in image capture mode and acquire image information includes: when the acquired object distance is not greater than the preset threshold, controlling the perception module to turn off depth detection mode and controlling the perception module to operate in image capture mode to acquire image information; determining the geometric feature change amount of the image based on the image information, and controlling the perception module to operate based on the geometric feature change amount; wherein, when the geometric feature change amount is greater than a preset change amount, controlling the perception module to continuously acquire image information and output it; when the geometric feature change amount is not greater than the preset change amount, controlling the perception module to enter depth detection mode to acquire object distance information.

[0008] Optionally, the step of determining the geometric feature changes of the image based on the image information includes: Multiple frames of images captured at different times are obtained from the image information, and the geometric and mechanical features of each frame are calculated. The geometric and mechanical features of multiple frames are compared to output the amount of geometric feature change between multiple frames.

[0009] Optionally, the sensing module includes a structured light camera and a fill light. The steps of controlling the sensing module to work in the depth detection mode include: turning off the fill light and controlling the structured light camera to project speckle and perform depth detection. The steps of controlling the sensing module to work in the image capture mode and acquire image information include: turning on the fill light and controlling the structured light camera to stop projecting speckle and perform image capture.

[0010] Optionally, the sensing module includes a proximity sensor and a CMOS camera. The steps of controlling the sensing module to operate in depth detection mode include: controlling the CMOS camera to stop working and controlling the proximity sensor to work for depth detection; the steps of controlling the sensing module to operate in image capture mode and acquire image information include: controlling the proximity sensor to stop working and controlling the CMOS camera to work for image capture.

[0011] Optionally, the step of obtaining the preset threshold includes: obtaining a depth monitoring strategy and determining the preset threshold according to the depth monitoring strategy; wherein the depth monitoring strategy is one of single-point monitoring, multi-point monitoring and area array monitoring. This application also proposes a visual touch sensor, which includes: a controller for executing the low-power control method applied to the visual touch sensor described above; a sensing module connected to the controller, the sensing module for monitoring the depth of an object and outputting the acquired distance information to the controller; the sensing module is also used to capture image information and output the captured image information to the controller, so that the controller generates corresponding geometric shapes and mechanical data.

[0012] Optionally, the sensing module is an indirect time-of-flight camera, which includes an image sensor and a filtering module. The filtering module is disposed on the image acquisition side of the image sensor. The image sensor includes a pixel array, which includes at least a first pixel, a second pixel, a third pixel, and a fourth pixel. The filtering module includes a filter array, which includes at least a first filter, a second filter, a third filter, and a fourth filter. The first filter, the second filter, the third filter, and the fourth filter filter each filter a different wavelength band. The first filter is configured to correspond to the first pixel, the second filter is configured to correspond to the second pixel, the third filter is configured to correspond to the third pixel, and the fourth filter is configured to correspond to the fourth pixel.

[0013] Optionally, the indirect time-of-flight camera further includes a processing unit electrically connected to the controller. The processing unit has a depth detection mode and an image capture mode. When the controller controls the processing unit to operate in the depth detection mode, the processing unit only acquires the light data collected by the first pixel and outputs the corresponding distance information based on the light data collected by the first pixel. When the controller controls the processing unit to operate in the image capture mode, the processing unit acquires the light data collected by all pixels and outputs the corresponding image information based on the light data collected by all pixels.

[0014] Optionally, the first filter is used to filter out light other than infrared light; the second filter is used to filter out light other than red light and infrared light; the third filter is used to filter out light other than green light and infrared light; and the fourth filter is used to filter out light other than blue light and infrared light.

[0015] Optionally, the pixel array has a central region and an outer peripheral region; wherein, the first pixel is disposed in the outer peripheral region of the pixel array; and the second, third, and fourth pixels are disposed in the central region of the pixel array.

[0016] Optionally, the sensing module is one of a time-of-flight camera, a structured light camera, a binocular camera, or a combined camera; wherein the combined camera includes a proximity sensor and a CMOS camera. This application also proposes a dexterous hand, comprising: the aforementioned visual-touch sensor; a touch-sensing module, the touch-sensing module being configured corresponding to the sensing module of the visual-touch sensor, the touch-sensing module having a contact area and a light-transmitting area, the contact area being configured corresponding to the central area of ​​the pixel array of the sensing module, and the light-transmitting area being arranged around the periphery of the contact area; wherein, a touch-sensing pattern is provided on the side of the contact area closer to the sensing module; the contact area is composed of silicone material, and the light-transmitting area is composed of a transparent conductive film.

[0017] Optionally, the dexterous hand also includes: a dimming module, which is disposed on the side of the touch module close to the sensing module corresponding to the touch pattern; the dimming module has a transparent state when powered on and a non-transparent state when powered off. In this application's technical solution, by replacing the traditional 2D CMOS camera with a sensing module, some functions of the sensing module can be turned off when there is no contact with an object. This solves the problem that traditional visual touch sensors need to work continuously after power-on, and the computing unit continues to calculate and output visualization data even when there is no contact with an object. Furthermore, by utilizing the depth detection function of the sensing module, the image transmission and back-end computing module of the sensing module are turned on or off when the touch sensor is a certain distance away from the object being touched, thereby reducing system power consumption and saving system computing resources to a certain extent. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0019] Figure 1 This is a flowchart of an embodiment of the low-power control method applied to a visual touch sensor according to this application; Figure 2 This is a flowchart of an embodiment of the low-power control method applied to a visual touch sensor according to this application; Figure 3 This is a flowchart of an embodiment of the low-power control method applied to a visual touch sensor according to this application; Figure 4 This is a flowchart of an embodiment of the low-power control method applied to a visual touch sensor according to this application; Figure 5 This is a schematic diagram of an embodiment of the visual touch sensor of this application; Figure 6 This is a schematic diagram of an embodiment of the sensing module of the visual-touch sensor of this application; Figure 7 This is a schematic diagram of an embodiment of the sensor module of the dexterous hand of this application; Figure 8 This is a schematic diagram of an embodiment of the dexterous hand of this application.

[0020] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0022] It should be noted that if the embodiments of this application involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0023] Furthermore, if the embodiments of this application involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application. A dexterous hand is a type of robotic component that simulates a human hand. Sensors, called visual-touch sensors, are typically mounted on the fingertips of a dexterous hand. Tactile feedback has always been a crucial requirement in human-computer interaction and advanced robotics, and visual-touch sensors play a key role in robot operation. Vision-based visual-touch sensors, in particular, can convert contact characteristics into images, offering advantages such as high spatial resolution, low-cost manufacturing, and compatibility with computer vision and machine learning algorithms, thus attracting widespread interest in the robotics field. Currently, existing visual-touch sensors typically employ CMOS miniature cameras, embedding CMOS cameras and LED light sources within a hollow structural base. The contact sensing area of ​​the sensor is often made of silicone, which usually has a specially designed internal pattern. When the sensor is activated, the CMOS miniature camera captures the pattern image within the silicone in real time. When the silicone contact point adheres to a certain surface terrain, the CMOS captures an amplitude-mask image at the front end, and the image is used to further calculate its geometric shape and mechanical characteristics, thus providing instantaneous 3D visualization data. However, the CMOS sensor and the image data-based computing unit are always in operation, transmitting the acquired image data to the internal processor for calculation, which results in a waste of power consumption and computing resources. To address the aforementioned problems, this application proposes a low-power control method for a visual touch sensor. The visual touch sensor includes a sensing module, which acquires image information from the touch sensor in a dexterous hand. (Refer to...) Figure 1 In one embodiment, the low-power control method applied to the visual touch sensor includes: S100. Obtain the distance information of the object through the sensing module; S200: Obtain a preset threshold, compare the obtained object distance with the preset threshold, and obtain the comparison result; S300: Control the operation of the sensing module based on the comparison results; wherein... When the distance to an object is greater than a preset threshold, the control perception module operates in depth detection mode to obtain the distance information of the object. When the distance to the object is no greater than a preset threshold, the control perception module operates in image capture mode and acquires image information.

[0024] This application replaces the traditional 2D CMOS camera used to monitor a silicone surface with a sensing module that has distance detection capabilities on existing tactile sensors. The sensing module can be an indirect time-of-flight camera (iToF) or a direct time-of-flight camera (dToF). It can detect the depth of an object in real time and also acquire the distance between the tactile sensor and the object to be touched. In other words, the sensing module can acquire both the depth and distance information of the object. Furthermore, it can be used as a 2D CMOS camera to capture image information from the tactile sensor in a dexterous hand. Its ability to disable some functions when not in contact with an object can reduce system power consumption and save processor computing resources. Alternatively, the sensing module can also be a structured light camera or a combined camera consisting of a proximity sensor and a CMOS camera, which can also achieve depth detection and image capture functions.

[0025] A time-of-flight camera actively emits modulated light, which is reflected when it encounters an external object. The camera receives the reflected light and calculates the time difference between the emitted and reflected light to generate corresponding three-dimensional depth information, thus determining the object's distance. A structured light camera uses projected speckle to determine the object's distance, while a combined camera consisting of a proximity sensor and a CMOS camera can also use the proximity sensor to determine the object's distance. Therefore, when an object is present outside the visual-touch sensor, the sensing module can acquire the distance information between the sensor and the object, including depth information. Understandably, since the distance between the touch module and the visual touch sensor on the dexterous hand is fixed and known, by determining the distance between the visual touch sensor and the object, it is possible to determine how far the touch module of the dexterous hand is from the object and whether the object is about to come into contact with the touch module of the dexterous hand. Based on the judgment result, it is possible to control whether the sensing module acquires and outputs the image information on the touch module, so that when the object comes into contact with the touch module, the visual data is generated based on the geometric changes in the image information. When the distance is far, it is not necessary to activate the image acquisition function, which can save computing power and reduce power consumption.

[0026] This application designs a triggering mechanism based on the distance between the visual-touch sensor and the object. Specifically, it sets a preset distance threshold and determines the operating mode of the sensing module based on the comparison between the object distance and the preset threshold. The sensing module has a depth detection mode and an image capture mode. When the sensing module operates in depth detection mode, it obtains the distance information of the corresponding object. When the sensing module operates in image capture mode, it acquires an amplitude-mask image of the texture information on the sensor to obtain image information and outputs it, so as to generate visualization data based on the image information.

[0027] Specifically, when the determined object distance is greater than a preset threshold, meaning the external object is far from the visual-touch sensor, it can be considered that the object has not made contact with the sensor. In this case, there is no need to use the sensing module to generate visualization data; therefore, it is only necessary to control the sensing module to work in depth detection mode to continue detecting the distance to the object. Conversely, when the determined object distance is not greater than the preset threshold, the distance between the object and the sensor is small enough that it can be considered that the object has made or is about to make contact with the sensing module. In this case, it is necessary to use the sensing module to acquire image information to generate visualization data. That is, the sensing module is controlled to work in image capture mode, start calculating the amplitude mask image at the front end to acquire image information and output it so as to generate visualization data based on the image information. With this setting, the sensing module only activates the image capture function when the object is in contact or about to make contact. When the object is far away, the sensing module does not need to start the real-time calculation module; it only needs to detect the depth or distance information of one or a few points. This can be easily achieved through optical and algorithm design, which saves computing power and reduces system power consumption.

[0028] In this application's technical solution, by replacing the traditional 2D CMOS camera with a sensing module, some functions of the sensing module can be turned off when there is no contact with an object. This solves the problem that traditional visual touch sensors need to work continuously after power-on, and the computing unit continues to calculate and output visualization data even when there is no contact with an object. Furthermore, by utilizing the depth detection function of the sensing module, when the touch sensor is a certain distance away from the object being touched, the image capture and transmission of the sensing module and the operation of the back-end computing module are turned on or off, thereby reducing system power consumption and saving system computing resources to a certain extent. Reference Figure 2 In one embodiment, the step S100 of obtaining the distance information of the object obtained through the sensing module includes: S110, obtaining the distance information of the object multiple times through the sensing module within a preset time period; the step S200 of obtaining a preset threshold, comparing the obtained object distance with the preset threshold, and obtaining the comparison result includes: S210, obtaining the preset threshold, comparing the multiple obtained object distances with the preset threshold respectively, and obtaining multiple comparison results.

[0029] Understandably, in practical applications, a single object distance not exceeding a preset threshold may not necessarily be interpreted as an object approaching the visual touch sensor. Other situations, such as an object suddenly passing in front of the sensor, could lead to misjudgment by the system. Therefore, in this application, multiple consecutive distance data points can be acquired, and the object's distance can be determined based on these multiple data points, reducing the likelihood of system misjudgment. Users can also set corresponding preset thresholds according to their specific applications. For example, if a user desires higher trigger accuracy from the visual touch sensor, a smaller preset threshold can be set to avoid false triggering; conversely, a larger preset threshold can be set to reduce the likelihood of false triggering.

[0030] Specifically, a preset monitoring time T and / or the number of data acquisitions N can be set to acquire multiple distance information. The distance to the object is then determined based on these multiple distance information. The object distances acquired multiple times are then compared using preset thresholds. Finally, the motion information of the monitored object is determined based on the comparison results, thereby controlling the working mode of the sensing module. It is understandable that when the depth acquisition frequency of the sensing module is constant, setting the preset monitoring time T or the number of data acquisitions N serves the same purpose: acquiring N depth values. N is a user-configurable parameter. When N is large, the likelihood of subsequent judgment errors caused by abnormal depth errors will decrease, thus improving the robustness of the algorithm. However, N should not be too large, otherwise it will lead to a certain response delay.

[0031] Optionally, the steps for controlling the operation of the sensing module based on the comparison results include: S310. Control the operation of the perception module based on multiple comparison results; wherein, when the object distances acquired multiple times are all greater than a preset threshold, control the perception module to operate in depth detection mode to obtain distance information; when the object distances acquired multiple times are not greater than the preset threshold, control the perception module to operate in image capture mode to obtain image information.

[0032] This application determines the motion information of the monitored object by judging the relationship between N object distances and a preset threshold. If the object distance is lower than the preset threshold for N consecutive times, it means that the visual touch sensor is close to the target object. At this time, the sensing module works in depth detection mode, that is, the image capture function of the sensing module is enabled to obtain image information and output it to the controller of the visual touch sensor. At the same time, the controller of the visual touch sensor processes the image information to generate corresponding mechanical features and generate visualization data. If the object distance is greater than the preset threshold for N consecutive times, it means that the visual touch sensor is far from the target object. At this time, the sensing module is controlled to work in depth detection mode, that is, the image capture function of the sensing module is disabled. There is no need to transmit images and calculate useless information, thereby reducing system power consumption and saving system computing resources to a certain extent.

[0033] It should be understood that the distances of multiple objects can also be averaged, and then the average distance can be compared with a preset threshold. Based on the comparison result, the sensing module can be controlled to work in the corresponding mode. In addition, users can set different distance comparison methods according to actual application needs, which is not limited in this application. Reference Figure 4 In one embodiment, when the acquired object distance is not greater than a preset threshold, the step of controlling the sensing module to operate in image capture mode and acquire image information includes: S320. When the distance to the acquired object is not greater than a preset threshold, control the perception module to turn off the depth detection mode and control the perception module to work in the image capture mode to acquire image information. S330. Determine the geometric feature change of the image based on the image information, and control the operation of the perception module based on the geometric feature change; wherein, when the geometric feature change is greater than a preset change, control the perception module to continuously acquire image information and output it; when the geometric feature change is not greater than the preset change, control the perception module to enter the depth detection mode to acquire the distance information of the object.

[0034] Optionally, the step of determining the geometric feature changes of the image based on the image information includes: S331. Obtain multiple frames of images at different capture times from the image information, and calculate the geometric and mechanical characteristics of each frame of image; S332. Compare the geometric and mechanical features of multiple frames of images to output the amount of change in geometric features between the multiple frames of images.

[0035] Understandably, when the controller of the visual touch sensor does not require distance information to process the image information captured by the sensing module, the distance information acquired by the sensing module is redundant. Therefore, further, the depth detection function of the sensing module can be turned off when the sensing module is capturing images, thereby further reducing the power consumption of the entire system. Specifically, when the acquired object distance is not greater than a preset threshold, the distance between the object and the visual touch sensor is small enough to be preliminarily considered as the object having already made or about to make contact with the sensor. At this time, the sensing module acquires image information, and the controller of the visual touch sensor acquires multiple frames of images at different capture times based on the image information, calculates the geometric and mechanical features of each frame, and compares the geometric and mechanical features of multiple frames, that is, compares the geometric and mechanical features of multiple frames before and after time according to the time sequence, thereby determining the amount of change in geometric features before and after the image change, and controlling the operation of the sensing module based on the amount of change in geometric features. When the change in geometric features exceeds a preset threshold, the image of the sensing area is determined to have deformed, indicating contact with an object. In this case, the sensing module continuously acquires and outputs image information to generate visualization data. Conversely, when the change in geometric features is less than the preset threshold, the image of the sensing area is considered to be in its original state, meaning no contact with an object has occurred. The sensing module then re-enters depth detection mode to detect the object's distance until the next acquired object distance is no greater than a preset threshold, repeating the process. This configuration allows the sensing module to acquire image information and calculate the geometric and mechanical features of the sensing area pattern when an object approaches. When no object is touched, the sensing module re-enters depth detection mode to continuously detect external objects. This eliminates the need for the vision-touch sensor controller to continuously generate visualization data after power-on and also eliminates the need for continuous depth detection by the sensing module. Each controller and sensing module only operates when needed, further reducing the overall system power consumption. Optionally, the sensing module includes a structured light camera and a fill light. The steps of controlling the sensing module to work in the depth detection mode include: turning off the fill light and controlling the structured light camera to project speckle and perform depth detection. The steps of controlling the sensing module to work in the image capture mode and acquire image information include: turning on the fill light and controlling the structured light camera to stop projecting speckle and perform image capture.

[0036] In one embodiment, the sensing module consists of a structured light camera and a fill light. In depth detection mode, the structured light camera includes a projection unit and a receiving unit. In depth detection mode, the projection unit only needs to project a small speckle at the center, which is received by the receiving unit to obtain depth information. In image capture mode, the fill light is turned on and the speckle is turned off. The receiving unit in the structured light camera captures the tactile image. The switching of the working mode needs to be determined by combining the depth value and the result of geometric feature changes. Specifically, the system determines whether to project speckle and turn on the fill light based on the distance to the object. When the distance to the object is greater than a preset threshold, the sensing module needs to be controlled to enter depth detection mode. At this time, the fill light is turned off, and the structured light camera projects speckle. The depth information is obtained based on the received speckle information to determine the distance to the object. When the distance to the object is not greater than the preset threshold, it is considered that the object has been or will be touched. The structured light camera is then controlled to turn off speckle, and the fill light is turned on for fill light. The structured light camera is switched to image capture mode for image capture. The controller of the visual touch sensor calculates geometric features based on the captured image information and determines that there is no touch when the geometric deformation disappears. The system then switches back to the depth detection mode of the sensing module and repeats the above operations.

[0037] Optionally, the sensing module includes a proximity sensor and a CMOS camera. The steps of controlling the sensing module to operate in depth detection mode include: controlling the CMOS camera to stop working and controlling the proximity sensor to work for depth detection; the steps of controlling the sensing module to operate in image capture mode and acquire image information include: controlling the proximity sensor to stop working and controlling the CMOS camera to work for image capture.

[0038] In one embodiment, the sensing module consists of a proximity sensor and a CMOS camera. In depth detection mode, the CMOS camera is turned off and the proximity sensor is turned on for depth detection. Conversely, in image capture mode, the proximity sensor is turned off and the CMOS camera is turned on to capture the tactile image. Specifically, the proximity sensor and CMOS camera are activated based on the distance to the object. When the object distance is greater than a preset threshold, the sensing module is in depth detection mode. In this case, the CMOS camera is turned off and the proximity sensor is turned on, thus using the proximity sensor to obtain depth information to determine the object's distance. When the object distance is not greater than the preset threshold, it is considered that the object has been or is about to be touched. The proximity sensor is then turned off and the CMOS camera is turned on to capture the tactile image. The controller of the visual-touch sensor calculates geometric features based on the captured image information and determines that there is no touch when the geometric deformation disappears. The system then switches back to depth detection mode and repeats the above operations. Reference Figure 3 In one embodiment, the step of obtaining the preset threshold includes: obtaining a depth monitoring strategy and determining the preset threshold according to the depth monitoring strategy; wherein the depth monitoring strategy is one of single-point monitoring, multi-point monitoring and area array monitoring.

[0039] In practical applications, various depth monitoring methods can be implemented using the sensing module, such as single-point monitoring, multi-point monitoring, and area array monitoring. Therefore, users can set corresponding preset thresholds, i.e. working trigger thresholds, according to actual application needs or the depth monitoring method of the sensing module. For example, a typical setting scheme is to set the threshold as low as possible within the depth testing range of the iToF camera, such as 10mm.

[0040] Accordingly, if the user sets multiple selectable depth monitoring methods using the sensing module, multiple preset thresholds corresponding to different monitoring methods need to be set simultaneously. With this setup, when a preset threshold needs to be obtained, the depth monitoring strategy of the sensing module can be obtained first to determine the sensing module's monitoring method, and then the corresponding preset threshold can be determined based on the sensing module's monitoring method for comparison. It is understandable that different depth monitoring methods of the sensing module will result in different preset thresholds. For example, when the sensing module is set to single-point monitoring, due to its single-point monitoring, there will be a large error in depth judgment for objects with large areas; in this case, a larger preset threshold can be set. Similarly, when the sensing module is set to multi-point monitoring or area array monitoring, its depth judgment accuracy is higher; therefore, a smaller preset threshold can be set accordingly to avoid misjudgments that would cause it to switch operating modes. When the sensing module is set to single-point monitoring, its power consumption is lower, and the amount of data it needs to calculate is also smaller, which can more effectively reduce system power consumption and save system computing resources. In addition, users can also consider whether the controller needs further depth features when performing image processing on image information. For example, some algorithms use multiple data features, including depth features, and their calculation results are more accurate. In this case, the sensing module can adopt an area array monitoring method. If depth features are not needed and only IR / RGB images are required, the sensing module can adopt a multi-point monitoring method, or even a single-point monitoring method would be better. Therefore, users can set the depth monitoring method of the sensing module according to the actual application requirements. This application also proposes a visual touch sensor, comprising: a controller for executing the low-power control method for the visual touch sensor as described above; a sensing module connected to the controller, the sensing module for monitoring distance information of an object and outputting the acquired distance information to the controller; the sensing module is also used to capture image information and output the captured image information to the controller, so that the controller generates corresponding geometric shapes and mechanical data. This application replaces the traditional 2D CMOS camera that monitors the silicone surface with a sensing module that integrates distance detection in the existing visual touch sensor. The sensing module not only outputs standard grayscale images, fulfilling the functionality of a traditional CMOS camera in grayscale silicone pattern designs, but also, in conjunction with depth detection, generates corresponding three-dimensional depth information. This adds a dimension of depth information to the traditional camera's ability to capture image brightness, providing more data features for image analysis. During the upgrade and optimization of the visual touch sensor, the sensing module can both detect the distance between the touch sensor and the object being touched in real time and function as a 2D CMOS camera, requiring minimal additional components and having minimal impact on the overall structure. Furthermore, by utilizing the sensing module's distance detection capabilities, some of its functions can be disabled when not in contact with an object, reducing system power consumption and saving processor computing resources. These saved resources can then be allocated to other modules.

[0041] Optionally, when the sensing module is a time-of-flight camera, the time-of-flight camera includes a laser emitter, such as a VCSEL (Vertical-Cavity Surface-Emitting Laser) array light source. The laser emitter includes multiple sub-light source regions, and each sub-light source region can be controlled independently. By combining different sub-light source regions, single-point beams, multi-point beams, or array beams can be projected onto the object, thereby using a time-of-flight camera including multiple independent sub-light source regions to achieve various depth monitoring methods such as single-point monitoring, multi-point monitoring, and array monitoring.

[0042] Optionally, the sensing module is an indirect time-of-flight camera, which further includes an image sensor and a filtering module. The filtering module is disposed on the image acquisition side of the image sensor. The image sensor includes a pixel array, which includes at least a first pixel, a second pixel, a third pixel, and a fourth pixel. The filtering module includes a filter array, which includes at least a first filter, a second filter, a third filter, and a fourth filter. The first filter, the second filter, the third filter, and the fourth filter filter each filter a different wavelength band. The first filter is configured to correspond to the first pixel, the second filter is configured to correspond to the second pixel, the third filter is configured to correspond to the third pixel, and the fourth filter is configured to correspond to the fourth pixel.

[0043] In one embodiment, the sensing module is an indirect time-of-flight camera. Thus, when designing the indirect time-of-flight camera module, the pattern design on the lower surface of the silicone and the controller can be combined to determine whether color features are needed during image processing. In this way, an infrared + depth camera or an infrared + RGB + depth camera can be selected. It should be noted that the infrared image and color image processing algorithms corresponding to these two types of cameras will be different. In general, indirect time-of-flight cameras have their own dedicated chip for depth processing and data processing.

[0044] When the inner design pattern of the silicone is colored, and color features are also required for image analysis, the indirect time-of-flight camera with IR+ depth has certain limitations because it can only obtain grayscale images and lacks color information. To address this common application scenario, this application proposes a novel color-enabled indirect time-of-flight camera as a visual-touch sensor solution for internal detection devices. The indirect time-of-flight camera includes an image sensor and a filtering module. The filtering module is located on the image acquisition side of the image sensor, and the image sensor includes a pixel array, while the filtering module includes a filter array. Specifically, the pixel array includes at least a first pixel, a second pixel, a third pixel, and a fourth pixel, and the filter array includes at least a first filter, a second filter, a third filter, and a fourth filter. The first, second, third, and fourth filters are filters for different wavelengths, and each filters out different wavelengths of light. Users can set the wavelengths filtered by different filters and the application of pixel channels according to actual application requirements to obtain color images with different characteristics. It should be understood that users can set different pixels to collect different colors of light according to actual application needs. The pixel array here can include more than four types of pixels, and this application does not limit this. For example, the first filter can be set to allow only infrared light to pass through, then the corresponding first pixel is a depth monitoring pixel, which can use infrared light to obtain distance information to realize the monitoring of the distance to the object. The second, third and fourth filters can be set to allow only visible light of different colors to pass through, so that the second, third and fourth pixels can collect information of different colors and provide more diverse data features. Optionally, the indirect time-of-flight camera further includes a processing unit electrically connected to the controller. The processing unit has a depth detection mode and an image capture mode. When the controller controls the processing unit to operate in the depth detection mode, the laser emitter of the indirect time-of-flight camera emits infrared light for depth monitoring, and the image sensor collects the reflected light beam. At this time, the processing unit only acquires the light data collected by the first pixel in the image sensor and outputs the corresponding distance information based on the light data collected by the first pixel. When the controller controls the processing unit to operate in the image capture mode, the processing unit acquires the light data collected by all pixels and outputs the corresponding image information based on the light data collected by all pixels.

[0045] In one embodiment, the first filter can be configured to allow only infrared light to pass through. Correspondingly, the first pixel is a depth monitoring pixel, capable of using infrared light to acquire distance information and thus monitor the distance to an object. Therefore, when the processing unit operates in depth detection mode, it can acquire only the light data collected by the first pixel and block the light data collected by the second, third, and fourth pixels, thereby outputting the corresponding distance information based on the light data collected by the first pixel to determine the distance to the monitored object. Similarly, when the processing unit operates in image capture mode, it can acquire the light data collected by all pixels and output the corresponding image information for the controller to calculate the geometry and mechanical characteristics of the silicone area, thereby generating corresponding visualization information.

[0046] Optionally, the first filter is used to filter out light other than infrared light; the second filter is used to filter out light other than red light and infrared light; the third filter is used to filter out light other than green light and infrared light; and the fourth filter is used to filter out light other than blue light and infrared light.

[0047] Reference Figure 6 As shown, in one specific embodiment, a novel color-coded indirect time-of-flight camera with an RGB+NIR (Near Infrared) layout is proposed. Different band filters are designed above the pixel array. The first filter filters out light other than near-infrared light, the second filter filters out light other than red and near-infrared light, the third filter filters out light other than green and near-infrared light, and the fourth filter filters out light other than blue and near-infrared light. With this configuration, during exposure, the four pixels can capture signals in the red, green, blue, and near-infrared bands. The pixels in the red, green, and blue channels can then recover the color information of the entire image using a demosaic algorithm. The pixels corresponding to the near-infrared band are indirect time-of-flight pixels, which can acquire information such as brightness and depth values. Thus, this indirect time-of-flight camera can acquire full-resolution RGB data, as well as 1 / 4 full-resolution NIR and depth data. Similar to what was mentioned earlier, it can simultaneously output RGB, NIR and depth information, providing more diverse data features for the controller to calculate the geometry and mechanical characteristics of the silicone area. It can also detect the distance between external objects and the visual touch sensor through the light-transmitting material to determine when to activate the image capture function.

[0048] Optionally, the pixel array has a central region and an outer peripheral region; wherein, the first pixel is disposed in the outer peripheral region of the pixel array; and the second, third, and fourth pixels are disposed in the central region of the pixel array. Optionally, the sensing module is one of a time-of-flight camera, a structured light camera, a binocular camera, or a combined camera; wherein the combined camera includes a proximity sensor and a CMOS camera.

[0049] It is understandable that the sensing module can be a time-of-flight camera, i.e., an indirect time-of-flight camera or a direct time-of-flight camera. A time-of-flight camera can capture image information of the tactile pattern and also obtain the distance between the visual-touch sensor and the object to be touched. In other words, the sensing module can acquire distance information of the object and can also be used as a 2D CMOS camera. Disabling some of its functions when not in contact with the object can reduce system power consumption and save processor computing resources to some extent. The sensing module can also be a structured light camera, which can project and receive speckle patterns, thereby determining the distance of the object based on the speckle information. The sensing module can also be a binocular camera, which includes a left camera and a right camera. The left and right cameras respectively acquire left and right images of the object for stereo matching to obtain parallax, thereby determining the distance of the object based on the parallax.

[0050] The sensing module can also be a combined camera, which may include a proximity sensor and a CMOS camera. During depth detection, the proximity sensor is used for distance measurement, and during image capture, the CMOS camera is used for image capture. Similarly, the sensing module can also be a combined camera with other configurations, as long as it has both depth detection and image capture modes. These are not specifically limited here. Furthermore, the sensing module may include a supplementary light, which is activated during image capture to ensure the camera can capture clear tactile images, thereby improving image capture quality and obtaining high-precision geometric shapes and mechanical features. This application also proposes a dexterous hand, comprising: the aforementioned visual-touch sensor; a touch-sensing module, the touch-sensing module being configured corresponding to the sensing module of the visual-touch sensor, the touch-sensing module having a contact area and a light-transmitting area, the contact area being configured corresponding to the central area of ​​the pixel array in the indirect time-of-flight phase, and the light-transmitting area being arranged around the periphery of the contact area; wherein, a touch-sensing pattern is provided on the side of the contact area closer to the sensing module; the contact area is composed of silicone material, and the light-transmitting area is composed of a transparent conductive film.

[0051] Reference Figure 7As shown, the dexterous hand also includes a sensing module, which corresponds to the sensing area of ​​the visual-touch sensor on the dexterous hand. This application proposes a method combining translucent materials and silicone materials. The silicone serves as the contact surface, with a pattern designed and drawn on the inner side for recognition. Specifically, it can be a gray-white pattern or a colored pattern. When the visual-touch sensor is activated, the sensing module captures the pattern image inside the silicone in real time when it is in image capture mode. When the silicone contact point is attached to the surface of an object, the sensing module captures the amplitude mask image at the front end. The image is used to further deduce its geometric shape and mechanical characteristics, thereby providing real-time 3D visualization data. The silicone periphery is made of translucent material, forming an overall finger structure with an inner silicone ring and an outer transparent material ring. The silicone part retains its original functionality, while the transparent material ensures that the camera embedded at the bottom can perceive external image data through the structure, thereby further detecting distance.

[0052] Specifically, the light-transmitting area can be composed of a transparent conductive film. Transparent conductive films are thin-film materials that combine high transparency and conductivity, and are widely used in touchscreens, displays, solar cells, and flexible electronic devices. The conductive materials typically include metals, metal oxides, and carbon-based materials. Currently, there are methods for preparing transparent conductive films that improve light transmittance in the near-infrared band, making this a very suitable material for the structural design of this application. Using it as the light-transmitting part of the light-transmitting area effectively ensures the transmittance of infrared lasers, thereby accurately detecting the distance to objects outside the transparent film. Furthermore, no holes are required; the film itself has good flexibility, and it only needs to be covered onto a silicone finger to achieve the functions described above. It should be understood that the touch-sensing module can also be independently set up from the visual touch sensor and integrated into the dexterous hand. It only needs to be positioned corresponding to the sensing module in the visual touch sensor; this application does not impose any restrictions here.

[0053] For dexterous hands, where the processing unit doesn't require depth features, the overall distance information is actually quite redundant; capturing the depth outside the light-transmitting area is sufficient. Furthermore, even distance information from external objects passing through one or more points on the light-transmitting area is enough for the system to determine the distance of the object to the finger and promptly trigger the IR or RGB image and corresponding processing unit to perform tactile feature calculations. This significantly reduces system power consumption. Therefore, the first pixel can be set as the depth monitoring pixel and positioned in the outer periphery of the pixel array, corresponding to the light-transmitting area, ensuring infrared laser transmittance and accurately detecting the distance to objects outside the transparent film. The second, third, and fourth pixels can be set as RGB channels and positioned in the center of the pixel array, corresponding to the contact area, to capture the pattern image inside the silicone. This image can then be used to further calculate its geometry and mechanical characteristics, providing real-time 3D visualization data. In addition, the overall structure of the sensing module can adopt a punch-hole design. At this time, the corresponding pixel positions can also be set according to the actual hole positions. In addition, the pixel positions can also be set according to the depth monitoring method of the sensing module. For example, in the single-point monitoring method, the depth monitoring pixel can be set in the center of the array, and the corresponding hole position can be opened in the center of the contact area, thereby realizing the single-point monitoring of the sensing module.

[0054] Optionally, the dexterous hand also includes: a dimming module, with the corresponding tactile pattern set on the side of the tactile module close to the sensing module; the dimming module has a transparent state when powered on and a non-transparent state when powered off.

[0055] Reference Figure 8As shown, the dexterous hand also includes a dimming module, which can be a material with variable light transmittance properties, such as a smart dimming film composed of liquid crystal molecules. The following explanation will use a smart dimming film as an example of the dimming module. This application covers the dimming module onto a specially designed tactile pattern in the sensing area. When the dimming module is powered off, light cannot directly penetrate the film because the liquid crystal molecules in the smart dimming film are in a dispersed state, making the film opaque. At this time, the camera can only observe the area inside the finger and collect image information with the pattern. When powered on, the entire finger area is transparent, allowing observation of objects outside the finger area and outputting data in the required format, thus making the system more flexible. In this way, the entire system can be used as a 3D camera or switched to a visual-touch sensor, thus having more operating modes and providing more multi-dimensional information in different application environments. When using this intelligent dimming film, its power-on / off state needs to be determined based on the camera's operating mode. When only the pattern inside the finger needs to be observed, the intelligent dimming film is in a power-off state; when observing objects outside the finger is required, it is switched to a power-on state. Specifically, the power-on and power-off of the dimming module can be set according to the user's actual application needs. For example, the power-on or power-off of the intelligent dimming film can be controlled based on the comparison between the geometric change amount mentioned earlier and the preset change amount, or it can be based on the operating mode of the sensing module. For example, it can be in a power-on state when the sensing module is working in depth detection mode and in a power-off state when the sensing module is working in image capture mode. These are not limited to a single specific scenario. Furthermore, it is understood that users can set the position of the dimming module according to specific application needs. For example, the dimming module can cover only the contact area of ​​the touch module, or it can cover the entire area of ​​the touch module, including both the contact area and the light-transmitting area. Users can flexibly set this according to their specific application requirements.

[0056] The above are merely preferred embodiments of this application and do not limit the scope of the patent application. Any equivalent structural transformations made based on the inventive concept of this application and the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included within the scope of patent protection of this application.

Claims

1. A low-power control method for a visual touch sensor, the visual touch sensor comprising a sensing module, characterized in that, include: The distance information of the object is obtained through the sensing module; Obtain a preset threshold, compare the obtained object distance with the preset threshold, and obtain the comparison result; The operation of the sensing module is controlled based on the comparison result; wherein... When the distance to the object is greater than the preset threshold, the sensing module is controlled to work in depth detection mode to obtain the distance information of the object. When the distance to the object is not greater than the preset threshold, the sensing module is controlled to work in image capture mode and acquire image information; The sensing module is an indirect time-of-flight camera, including an image sensor and a processing unit. The image sensor includes a pixel array, which includes at least a first pixel, a second pixel, a third pixel, and a fourth pixel. When the perception module is working in depth detection mode, the processing unit only acquires the light data collected by the first pixel and outputs the corresponding object distance information based on the light data collected by the first pixel. When the sensing module is operating in image capture mode, the processing unit acquires the light data collected by all pixels and outputs the corresponding image information based on the light data collected by all pixels.

2. The low-power control method for visual touch sensors as described in claim 1, characterized in that, The step of obtaining the distance information of the object through the sensing module includes: The distance information of the object is acquired multiple times within a preset time period through the sensing module; The step of obtaining a preset threshold, comparing the obtained object distance with the preset threshold, and obtaining the comparison result includes: A preset threshold is obtained, and the object distances obtained multiple times are compared with the preset threshold to obtain multiple comparison results.

3. The low-power control method for visual touch sensors as described in claim 2, characterized in that, The step of controlling the operation of the sensing module based on the comparison result includes: The operation of the sensing module is controlled based on multiple comparison results; wherein... When the distance to the object is obtained multiple times and is greater than the preset threshold, the perception module is controlled to work in depth detection mode to obtain distance information. When the distance to the object obtained in multiple tests is not greater than the preset threshold, the sensing module is controlled to work in image capture mode to obtain image information.

4. The low-power control method for visual touch sensors as described in claim 1, characterized in that, The step of controlling the perception module to operate in image capture mode and acquire image information when the acquired object distance is not greater than the preset threshold includes: When the distance to the object is not greater than the preset threshold, the perception module is controlled to turn off the depth detection mode and to work in the image capture mode to obtain image information. The geometric feature changes of the image are determined based on the image information, and the operation of the sensing module is controlled based on the geometric feature changes; wherein, When the change in the geometric features is greater than a preset change, the sensing module is controlled to continuously acquire and output image information; When the change in the geometric features is not greater than a preset change, the sensing module is controlled to enter the depth detection mode to obtain the distance information of the object.

5. The low-power control method for visual touch sensors as described in claim 4, characterized in that, The step of determining the geometric feature changes of the image based on the image information includes: Multiple frames of images captured at different times are obtained from image information, and the geometric and mechanical features of each frame are calculated. The geometric and mechanical features of multiple frames are compared to output the amount of geometric feature change between the multiple frames.

6. The low-power control method for visual touch sensors as described in claim 1, characterized in that, The step of obtaining the preset threshold includes: A depth monitoring strategy is acquired, and a preset threshold is determined based on the depth monitoring strategy; wherein, The depth monitoring strategy is one of single-point monitoring, multi-point monitoring, and area array monitoring.

7. A visual-touch sensor, characterized in that, include: A controller for performing the low-power control method for a visual touch sensor as described in any one of claims 1-6; A sensing module, connected to the controller, is used to monitor the distance information of objects and output the acquired distance information to the controller; The sensing module is also used to capture image information and output the captured image information to the controller, so that the controller can solve the image information and generate corresponding geometric shapes and mechanical data.

8. The visual-touch sensor as described in claim 7, characterized in that, The sensing module further includes a filtering module, which is disposed on the image acquisition side of the image sensor. The filtering module includes a filter array, which comprises at least a first filter, a second filter, a third filter, and a fourth filter. The first, second, third, and fourth filters filter out different wavelengths of light. The first filter is set to correspond to the first pixel, the second filter is set to correspond to the second pixel, the third filter is set to correspond to the third pixel, and the fourth filter is set to correspond to the fourth pixel.

9. The visual-touch sensor as described in claim 8, characterized in that, The first filter is used to filter out light other than infrared light; The second filter is used to filter out light other than red and infrared light; The third filter is used to filter out light other than green light and infrared light; The fourth filter is used to filter out light other than blue light and infrared light.

10. The visual-touch sensor as claimed in claim 7, characterized in that, The pixel array has a central region and an outer peripheral region; wherein... The first pixel is located in the outer peripheral region of the pixel array; The second, third, and fourth pixels are located in the central area of ​​the pixel array.

11. A dexterous hand, characterized in that, include: The visual-touch sensor as described in any one of claims 7-10; A touch-sensing module, corresponding to the sensing module of the visual-touch sensor, is provided. The touch-sensing module has a contact area and a light-transmitting area. The contact area corresponds to the center area of ​​the pixel array of the sensing module, and the light-transmitting area is arranged around the periphery of the contact area. The contact area has a tactile pattern on the side closest to the sensing module; The contact area is composed of silicone material, and the light-transmitting area is composed of a transparent conductive film.

12. The dexterous hand as described in claim 11, characterized in that, The dexterous hand also includes: A dimming module is positioned on the side of the touch module closest to the sensing module, corresponding to the touch pattern. The dimming module has a transparent state when powered on and a non-transparent state when powered off.

Images