Sensing, storing and computing integrated visual tactile sensor and working system and tactile sensing method thereof

Abstract

The invention provides a sensing, storage and calculation integrated visual tactile sensor which comprises a contact module, a lighting module and a camera module. Wherein the contact module is used for converting physical contact information generated when the contact module is in contact with an external object into corresponding optical information; the illumination module is used for providing an illumination environment and enabling the optical information to be continuously propagated in the illumination environment; the camera module comprises a lens and a focal plane sensing processor which are sequentially arranged from front to back; wherein the lens is used for capturing optical information, and the focal plane sensing processor is used for carrying out information conversion on the captured optical information to obtain processed tactile information. Meanwhile, the invention provides a corresponding working system and a touch sensing method. According to the invention, the sensing, storing and computing integrated visual chip is utilized, the computing pressure of an upper computer is shared, the bandwidth pressure of a transmission end is relieved, the energy consumption of the whole system is reduced, no extra hardware equipment is introduced, and the complexity and deployment cost of the system are well controlled.

Description

Technical Field

[0001] The present invention relates to the technical field of visual-tactile sensors, and in particular to a sensing-storage-computing integrated visual-tactile sensor and a working system and a tactile perception method thereof. Background Art

[0002] In the fields of humanoid robots, dexterous hand manipulation, and human-computer interaction, high-quality tactile information is essential for enriching the perception of the external environment by embodied intelligent systems. These application scenarios usually rely on fine tactile texture perception and sensitive detection of normal force and shear force. Compared with tactile sensors based on principles such as resistance, capacitance, and magnetic field, visual tactile sensors have certain advantages due to the high-resolution visual information brought by the built-in camera, efficient and convenient signal acquisition capabilities, and low manufacturing costs. The traditional visual tactile sensor working architecture includes a sensor front end and a host computer back end. The sensor front end is responsible for converting the tactile information generated by external physical contact into visual information captured by the camera, usually image data. Subsequently, this visual information is transmitted to the host computer back end (usually a computer) via wired or wireless means. The host computer runs the corresponding program to perform signal processing and analysis on the received visual information, and finally transmits the processing results to the downstream algorithm to complete specific tasks, such as fabric recognition through tactile texture information, or dexterous operation based on the force and position of contact. However, the above traditional working architecture has problems such as high signal transmission pressure between the sensor front end and the host computer back end, and serious centralization of signal processing operation burden. Figure 1 As shown. This limits the development of visual tactile sensors in larger-scale applications, such as applications in devices such as dexterous hands, robotic arms, and humanoid robots that require a high-density tactile perception range. As the demand for spatial and temporal resolution of tactile information in intelligent robot systems continues to grow, the data size, output frame rate, and operating power consumption of traditional cameras will increase significantly, while also placing higher demands on the bandwidth and stability of signal transmission. On the host computer side, there is an increasing demand for computing power. The increase in tactile signal processing will further squeeze the computing resources of downstream algorithms, which may have a negative impact on the performance of the entire system.

[0003] Existing visual tactile sensor improvement solutions include using event cameras to sparse the features of the output images, enhance time information, and reduce transmission bandwidth, but at the same time, they also sacrifice the ability of high-resolution static perception and significantly increase production costs. Figure 2As shown in the figure, another solution is to introduce additional edge computing devices, such as using an MCU to pre-process the visual and tactile image data, and then transmit the processed data to the host computer for calculation. This solution reduces the transmission burden between the edge computing device and the back end of the host computer, alleviating the computing pressure of the host computer, but introduces additional hardware devices, complicates the system architecture and significantly increases the cost. In addition, the signal transmission pressure between the sensor front end and the edge computing unit has not been reduced, so it is essentially a solution to transfer contradictions rather than resolve them.

[0004] Based on the above background, before the large-scale promotion and application of visual tactile sensors, further improvement and optimization are still needed. Currently, no description or report of similar technology to the present invention has been found, and similar information at home and abroad has not been collected. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present invention provides a vision-tactile sensor with integrated sensing, storage and computing, a working system thereof and a tactile perception method.

[0006] According to one aspect of the present invention, a sensing, storage and computing integrated visual and tactile sensor is provided, comprising: a contact module, an illumination module and a camera module; wherein:

[0007] The contact module is used to convert physical contact information generated when contacting with an external object into corresponding optical information;

[0008] The lighting module is used to provide a lighting environment and enable the optical information to continue to propagate in the lighting environment;

[0009] The camera module includes a lens and a focal plane sensor processor which are arranged in sequence from front to back; wherein the lens is used to capture the optical information, and the focal plane sensor processor is used to convert the captured optical information to obtain processed tactile information.

[0010] According to another aspect of the present invention, a sensing, storage and computing integrated visual and tactile sensor working system is provided, comprising: an integrated sensor front end, a signal transmission end and a host computer back end; wherein:

[0011] The sensor front end adopts a sensing, storage and computing integrated visual and tactile sensor, which is used to convert physical contact information into optical information, and processes the optical information based on a focal plane sensor processor to obtain processed tactile information;

[0012] The signal transmission end is used to send the processed tactile information to the host computer back end;

[0013] The host computer backend is used to execute a downstream algorithm on the received processed tactile information, so as to complete a designated task.

[0014] According to a third aspect of the present invention, a tactile perception method of a sensing, storage and computing integrated visual and tactile sensor is provided, comprising:

[0015] The external object comes into contact with the elastic body of the contact module at the front end of the sensor. Based on different perception principles, the tactile information is converted into optical information by the contact module.

[0016] The optical information continues to propagate in the lighting environment provided by the illumination module in front of the sensor until it is captured by the focal plane sensor processor of the camera module;

[0017] The photosensitive units distributed on the pixel processor array in the focal plane sensor processor capture the optical information and convert it into pixel values ​​and store them in the analog registers in the pixel processor array;

[0018] Directly calculating the pixel values ​​in parallel according to the required tasks and storing them in digital registers within the pixel processor array for subsequent calculations;

[0019] The upper microprocessor in the focal plane sensor processor runs in parallel with the pixel processor array to provide instructions and communications for the operation of the pixel processor array. When the pixel processor array completes the sense-storage-computing operation to obtain visual data, the upper microprocessor continues to perform subsequent calculations and maps the preliminary visual data to the processed tactile information, which is then sent to the upper computer back end to execute downstream algorithms to complete designated tasks.

[0020] Due to the adoption of the above technical solution, the present invention has at least one of the following beneficial effects compared with the prior art:

[0021] The present invention has output flexibility: by introducing the integrated vision chip of sensing, storage and computing into the visual tactile sensor, the output of the sensor is no longer limited to a single image data, but can be customized and programmed, including raw image data, pre-processed tactile data, tactile coding vectors, and even control signals, etc. This greatly improves the flexibility of the output data of the traditional visual tactile sensor and its adaptability to different tasks and system integration.

[0022] The present invention has a balanced architecture: the integrated sensing, storage and computing visual-tactile sensor has its own computing capabilities, which disperses the computing pressure that was originally completely located at the back end of the host computer to the front end of the sensor, reducing the communication bandwidth pressure without introducing additional edge computing unit hardware, thus creating a new visual-tactile sensor working framework.

[0023] The present invention has advanced performance: the pixel array processor used in the integrated sensing, storage and computing visual chip realizes parallel computing functions in hardware and is good at processing image data at high speed and efficiency. This enables the integrated sensing, storage and computing visual-tactile sensor to have the ability to perceive dynamic tactile information at high frequency and have a lower power consumption level, breaking through the performance and cost bottlenecks of traditional visual-tactile sensors.

[0024] The present invention utilizes a sensing, storage and computing integrated vision chip to share the computing pressure of the host computer, alleviate the bandwidth pressure at the transmission end, and reduce the energy consumption of the overall system. At the same time, no additional hardware equipment is introduced, and the system complexity and deployment cost are well controlled. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0026] Figure 1 Schematic diagram of the working structure of a traditional visual tactile sensor in the prior art.

[0027] Figure 2 Schematic diagram of the working architecture of the visual-tactile sensor improved based on the edge computing unit in the prior art.

[0028] Figure 3 This is a schematic diagram of the working structure of a vision-tactile sensor with integrated sensing, storage and computing in a preferred embodiment of the present invention.

[0029] Figure 4 This is a working architecture diagram of a vision-tactile sensor with integrated sensing, storage and computing in a preferred embodiment of the present invention.

[0030] Figure 5 Schematic diagram of the working architecture of a vision-tactile sensor working system with integrated sensing, storage and computing in a preferred embodiment of the present invention.

[0031] Figure 6 This is a workflow diagram of a tactile perception method of a vision-tactile sensor with integrated sensing, storage and computing in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following is a detailed description of the embodiments of the present invention: This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation method and a specific operation process are given. It should be pointed out that for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

[0033] In view of the problem that the existing technology still needs further improvement and optimization before the large-scale promotion and application of visual tactile sensors, an embodiment of the present invention provides a visual tactile sensor with integrated sensing, storage and computing. The sensor combines tactile perception with imaging and computing technology in the sensor. It is a new visual tactile sensor. Its working structure is as follows: Figure 3 As shown in the figure. After the sensor comes into contact with an external object, it first performs high-speed on-chip calculations on the captured raw image information through the advanced on-chip imaging and computing capabilities of its internal vision chip, converting it into useful tactile information, and then transmits this small-sized, high-value pre-processed tactile information to the host computer. The sensor uses a vision chip that integrates sensing, storage and computing to share the computing pressure of the host computer, relieve the bandwidth pressure at the transmission end, and reduce the energy consumption of the overall system. At the same time, no additional hardware equipment is introduced, which effectively controls the system complexity and deployment cost.

[0034] Specifically, Figure 3 As shown, the sensing, storage and computing integrated visual and tactile sensor provided in this embodiment may include: a contact module, a lighting module and a camera module; wherein:

[0035] A contact module, used to convert physical contact information generated when contacting with external objects into corresponding optical information;

[0036] An illumination module, used for providing an illumination environment and enabling optical information to continue to propagate in the illumination environment;

[0037] The camera module includes a lens and a focal plane sensor processor which are arranged in sequence from front to back; wherein the lens is used to capture optical information, and the focal plane sensor processor is used to convert the captured optical information to obtain processed tactile information.

[0038] like Figure 4 The figure is a working architecture diagram of the sensing storage and computing integrated visual tactile sensor provided by this embodiment. Figure 4 As shown, the sensing, storage and computing integrated visual and tactile sensor provided by this embodiment is further described in detail.

[0039] In some preferred embodiments, the contact module may further include: an elastomer and a lens arranged for contact from front to back, the front end of the elastomer may be provided with a coating, and the inside of the elastomer may be provided with a marking body.

[0040] In some preferred embodiments, the contact module is used to convert physical contact information generated when contacting an external object into corresponding optical information, and may further include:

[0041] Different tactile perception principles are used to convert the required tactile information into corresponding optical information based on actual tasks; wherein the perception principle includes: any one or any combination of intensity mapping principle, marker displacement principle and modal fusion principle.

[0042] In some preferred embodiments, the lighting module may further include: a plurality of light sources; the plurality of light sources are arranged in an array between the contact module and the camera module; wherein:

[0043] The light source uses the same color or different color light source;

[0044] The shape of the light source includes a point light source and / or a strip light source;

[0045] The incident position of the light source includes the side, upper side and / or upper side of the contact module.

[0046] In some preferred embodiments, the focal plane sensor processor may further include: a pixel processor array and a host microprocessor running in parallel; wherein:

[0047] A pixel processor array, used to convert light signals into analog electrical signals, and perform calculations based on the analog electrical signals to obtain visual information;

[0048] The upper microprocessor is used to perform command control and data transmission on the pixel processor array; at the same time, it extracts key information related to touch from the visual information to obtain processed tactile information and outputs it.

[0049] In traditional visual tactile sensors, the original image captured by the camera photosensitive chip is directly transmitted to the host computer backend without processing, which will bring two problems: (1) Transmission data bandwidth bottleneck: As the resolution and frame rate of the original image gradually increase, the bandwidth pressure of data transmission gradually increases, and the slight disturbance in the transmission will significantly affect the data quality, which has a negative impact on the stability of the visual tactile system; (2) Centralization of computing pressure: After receiving the original image, the host computer backend will send the data to the CPU / GPU hardware for centralized processing. As the resolution and frame rate of the original image gradually increase, and the number of original images brought by large-scale tactile perception systems (such as multi-finger dexterous hands) increases exponentially, these factors will cause the increasing computing pressure to be concentrated on a single hardware device at the host computer backend, resulting in a shortage of computing resources, thereby reducing the working frequency of the tactile perception algorithm and the performance of the tactile perception system. It may even squeeze the computing resources of other algorithm programs running on the host computer backend, thereby endangering the stability of the entire host computer system. The embodiment of the present invention uses a focal plane sensor processor to solve the above problems very well. The high-speed, low-power on-chip computing power of the pixel processor array based on parallel design is very suitable for processing matrix-type image data, so that the data can be effectively processed without having to be sent to the host computer, which helps to solve the problem of "centralized computing pressure"; furthermore, the introduction of the host microprocessor not only helps the pixel processor array to assist in calculations, but also enhances the logic and code calculation capabilities, which allows part of the functions of the host computer backend CPU to be perfectly shared by the host microprocessor, which helps to solve the problem of "centralized computing pressure". At the same time, the host microprocessor can output custom data through programming, including pre-processed tactile information and even control signals. This size-compressed output data can alleviate the "transmission data bandwidth bottleneck".

[0050] In some preferred embodiments, the focal plane sensor processor is used to convert the captured optical information to obtain processed tactile information, and may further include:

[0051] The photosensitive units in the pixel processor array capture the original light signals in the optical information and convert them into analog electrical signals to obtain pixel values ​​which are stored in analog registers in the pixel processor array; the pixel values ​​stored in the analog registers are used to directly perform parallel numerical calculations and / or logical operations in the digital domain according to the required tasks, obtain corresponding visual information and store it in digital registers in the pixel processor array;

[0052] The upper microprocessor extracts key information related to touch from the visual information to obtain processed tactile information.

[0053] In the working architecture of the visual tactile sensor of the embodiment of the present invention, the problem of "computational pressure centralization" is solved by introducing an edge computing unit, but additional hardware is also introduced, making the overall working framework complicated. The pixel array processor in the integrated sensing, storage and computing visual tactile chip combines the photosensitive element, the computing element and the storage element on the same focal plane, achieving the same function without changing the size of the device or introducing additional hardware.

[0054] In some preferred implementations, the pixel processor array may further include any one or more of the following:

[0055] -First, use a preset threshold to filter out random noise in the optical information, then perform morphological operations to remove and repair independent noise points to obtain the original optical signal;

[0056] - For image data with positional offset and / or visual angle distortion in the spatial plane in the visual information, the image data is corrected by parallel translation and / or distortion correction.

[0057] In some preferred embodiments, the upper microprocessor extracts key information related to touch from visual information to obtain processed tactile information, which may further include any one or more of the following methods:

[0058] - Based on explicit reasoning of mathematical and physical models, known mapping models are used to extract the required key information from visual information and convert it into target tactile information;

[0059] - Based on implicit reasoning of binarized neural networks, known deep learning models are used to extract the required key information from visual information and convert it into target tactile information.

[0060] As mentioned above, in the traditional visual and tactile sensor working framework, the sensor front end needs to transmit the raw data to the host computer back end for data processing, and the introduction of the integrated sensing, storage and computing vision chip disperses the calculation process to the sensor front end, so it is necessary to develop an adaptation algorithm accordingly, otherwise the program of the host computer back end cannot be directly transplanted. For the display reasoning method based on mathematical and physical models, the logical operation and code capabilities of the host microprocessor can be used to solve it. The implicit reasoning method based on deep learning can be implemented by running a binary neural network on a pixel array processor. This method can optimize the hardware architecture of the integrated sensing, storage and computing vision chip in a targeted manner to achieve optimal operating efficiency and reasoning performance.

[0061] Based on the sensing, storage and computing integrated visual and tactile sensor provided in any one of the above embodiments of the present invention, an embodiment of the present invention further provides a sensing, storage and computing integrated visual and tactile sensor working system.

[0062] Specifically, Figure 5 As shown, the sensing, storage and computing integrated visual-tactile sensor working system provided in this embodiment may include: an integrated sensor front end, a signal transmission end and a host computer back end; wherein:

[0063] The sensor front end uses a vision-tactile sensor with integrated sensing, storage and computing to convert physical contact information into optical information, and processes the optical information based on the focal plane sensor processor to obtain processed tactile information;

[0064] The signal transmission end is used to send the processed tactile information to the host computer back end;

[0065] The host computer backend is used to execute downstream algorithms on the received processed tactile information, and then to complete the specified task.

[0066] Based on the sensing, storage and computing integrated visual and tactile sensor provided in any one of the above embodiments of the present invention, an embodiment of the present invention further provides a tactile perception method of the sensing, storage and computing integrated visual and tactile sensor.

[0067] Specifically, Figure 6 As shown, the tactile perception method of the sensing, storage and computing integrated visual and tactile sensor provided in this embodiment may include the following operations:

[0068] S1, the external object comes into contact with the elastic body of the contact module at the front end of the sensor. Based on different perception principles, the tactile information is converted into optical information by the contact module;

[0069] S2, the optical information continues to propagate in the lighting environment provided by the illumination module in front of the sensor until it is captured by the focal plane sensor processor of the camera module;

[0070] S3, the photosensitive units distributed on the pixel processor array in the focal plane sensor processor capture the optical information, convert it into pixel values ​​and store it in the analog registers in the pixel processor array;

[0071] S4, directly calculating the pixel values ​​in parallel according to the required tasks and storing them in digital registers in the pixel processor array for subsequent calculations;

[0072] S5, the upper microprocessor in the focal plane sensor processor runs in parallel with the pixel processor array to provide instructions and communication for the operation of the pixel processor array. When the pixel processor array completes the sense-storage-calculation operation and obtains the visual data, the upper microprocessor continues to perform subsequent calculations and maps the preliminary visual data to the processed tactile information, and sends it to the upper computer back end to execute the downstream algorithm to complete the specified task.

[0073] The technical solutions provided by the above embodiments of the present invention are further described in detail below.

[0074] The sensing, storage and computing integrated visual-tactile sensor and its working system and tactile perception method provided in the above-mentioned embodiments of the present invention adopt a focal plane sensor processor (Focal-Plane Sensor-Processor, FPSP), which has the ability to capture, store and process visual data at the pixel level. Its implementation is derived from the sensing, storage and computing integrated digital-analog hybrid integrated circuit design of non-von Neumann architecture.

[0075] After the perception unit in the circuit converts the light signal into an analog electrical signal, it can directly perform numerical calculations and digital domain (1bit) logic operations based on the analog quantity (voltage or current) without the need for analog-to-digital conversion (ADC) of traditional visual sensors. Its core is a pixel processor array composed of processing elements, and it also integrates additional upper microprocessors (such as RISC-V, ARM, etc.) to perform command control and data transmission on the pixel processor array. The device is good at directly controlling the current of analog quantities to extract and directly output key information from the image, without the need for traditional recording, digital-to-analog conversion, and data transmission to an external computer, avoiding dependence on external computing resources and professional software. It can process image operations at a maximum speed of more than 10,000 frames per second, with an operating power consumption of less than 1.5 watts and a computing energy efficiency of more than 1TOPS / W. By using this device, the signal processing program and some downstream control algorithms running on the host computer can be transplanted to the focal plane sensor processor, taking advantage of its parallelism and the lack of need for external digital-to-analog conversion. High-speed, low-latency, and low-power optical imaging, data processing, feature analysis, and result output of visual and tactile information can be completed directly on the chip. It can also be flexibly programmed according to the different task requirements of the lower computer, thereby outputting customized processed tactile information.

[0076] The following is an introduction to the design of the integrated vision-tactile sensor, its working system, and tactile perception method:

[0077] 1. Hardware design of integrated vision-tactile sensor with sensing, storage and computing

[0078] (1) Main components: Figure 3 As shown in the figure, the integrated vision-tactile sensor for sensing, storage and computing mainly consists of three modules, namely the contact module, the lighting module and the camera module. The contact module is used to convert the tactile information generated when the sensor contacts the external object into optical information; the lighting module provides good lighting conditions for the above optical information conversion; the camera module is responsible for capturing the converted optical information and processing the information through the built-in focal plane sensor processor to obtain the processed tactile information.

[0079] Furthermore, the contact module converts the tactile information generated when the sensor contacts the external object into optical information, and its working method includes:

[0080] Existing visual tactile sensors can be divided into intensity mapping mechanism, marker displacement mechanism and modality fusion mechanism according to different perception principles. Different principles can also be combined to form a combination of two. Among them: Based on the intensity mapping principle, the reflective coating containing metal powder applied on the surface of the elastomer reflects the internal light when it comes into contact with the outside world, causing the visual information received by the camera to change in pixel intensity I_m*n. The pixel intensity I_i can be associated with contact features, such as depth, shape, etc., so it can be used to capture and reconstruct accurate contact textures; Based on the marker displacement principle, the marker array Ma embedded inside the elastomer changes its topological position P_a when it comes into contact with the outside world. The displacement of each marker itself can be described as a direction vector V_i, so that the whole The displacement field of the marker array can be represented by the matrix V_a. The displacement field can be used to calculate the absolute position, relative position, optical flow field, acceleration field, etc. of the marker. The above physical information can be linked to the contact characteristics, such as normal force, tangential force, etc., so it can be used to perceive the mechanical information and dynamic contact information of the contact; based on the principle of modal fusion, using the combination of transparent epidermis and elastomer, the visual information of the contact area can also be directly captured by the camera, such as fine patterns on the contact surface, color distribution, etc. The above visual information is a supplement to the contact characteristics and can be used for visual-tactile fusion perception, proximity prediction, etc. It should be pointed out that the above perception principles can be combined with each other, such as IMM+MDM, MDM+MFM, etc., and the contact module in the integrated visual-tactile sensor of sensing, storage and computing in the embodiment of the present invention can freely use the above different perception principles to convert the required tactile information into corresponding optical features based on the indicators of the actual task.

[0081] Furthermore, the focal plane sensor processor is used to convert the captured optical information to obtain processed tactile information, and its working method includes:

[0082] The photosensitive units in the pixel processor array capture the original light signals in the optical information and convert them into analog electrical signals, and the obtained pixel values ​​are stored in the analog registers in the pixel processor array;

[0083] The pixel values ​​stored in the analog registers are used to directly perform parallel numerical calculations and / or logical operations in the digital domain according to the required tasks, obtain corresponding visual information and store it in the digital registers within the pixel processor array;

[0084] The upper microprocessor extracts key information related to touch from the visual information to obtain processed tactile information.

[0085] In the process of information conversion, it also includes: first, threshold operation is performed, and random noise is filtered out using a suitable threshold, so that the data background becomes clean and convenient for subsequent effective tactile feature extraction; due to the average and global nature of the threshold operation, not all noise can be successfully filtered by the threshold, so morphological operations such as corrosion and expansion operations are required to remove and repair independent noise points. When the image data obtained after denoising is found to have problems such as position offset and perspective distortion on the spatial plane, the image data can be corrected by parallel movement, distortion correction, etc.

[0086] (2) Structural design details: In terms of structural design, since the architecture of the integrated visual-tactile sensor with sensing, storage and computing has significant versatility and generalization, there are no strict restrictions on its detailed design. For the contact module, according to different visual-tactile sensor perception principles, such as intensity mapping method (IMM), marker displacement method (MDM), modal fusion method (MFM), etc., its internal structure may be different. Common structures include elastomers (such as transparent silicone, rubber, etc.), lenses (such as acrylic, glass, etc.), markers (circular, rectangular, triangular, etc.) and coatings (such as reflective coatings of mixed copper powder and aluminum foil). For the lighting module, the same color or different color light sources can be used according to the perception requirements. The shape of the light source can be divided into point light sources and strip light sources. The injection position can be designed to be on the side, above the side or above the contact module. The design of the camera module needs to focus on the field of view, focal length, installation of the focal plane sensor processor, heat dissipation and wiring of the lens. In order to reduce the depth in the focal length direction, a mirror can be introduced to fold the optical imaging path. At the same time, due to the low power consumption characteristics of the focal plane sensor processor, this design has significant advantages in heat dissipation compared to traditional visual tactile sensors.

[0087] 2. Software design of integrated vision-tactile sensor with sensing, storage and computing

[0088] (1) Tactile feature capture: After conversion by the contact module and the lighting module, the original optical features of the visual touch will be captured by the photosensitive units in the pixel processor array, including information such as pixel intensity, distribution and change. The design goal of tactile feature capture is to preprocess the raw data and extract explicit tactile features from the implicit optical features mentioned above. The focal plane sensor processor of the embodiment of the present invention includes a pixel processor array and a host microprocessor running in parallel. The main computing platform is the pixel processor array. First, the time of analog-to-digital conversion can be saved by taking advantage of the hardware integration; second, the computing speed can be improved by using the parallel structure of the array. As mentioned in point 1. In addition, for some complex calculations, the assistance of the host microprocessor may also be required. The tactile features finally extracted vary depending on the design principle of the contact module, and may include pixel intensity mapping (commonly used for contact modules with reflective coatings), marker displacement tracking (commonly used for contact modules with markers), optical flow field calculation (applicable to large-scale dynamic contact information collection scenarios), etc.

[0089] Furthermore, the focal plane sensor processor used in the embodiments of the present invention has the following advantages and innovations:

[0090] Compared with the common USB camera used in the conventional visual-tactile sensor, the embodiment of the present invention uses the focal plane sensor processor to achieve hardware innovation, and obtains the visual-tactile sensor architecture, such as Figure 5 As shown, this is consistent with existing architectures (such as Figure 1 ) or even improve the architecture (as shown Figure 2 As shown in the figure, the pixel processor array of the focal plane sensor processor itself is different. Each unit can not only capture optical information like the photosensitive chip inside a traditional camera, but also integrates multiple analog registers, digital registers, and computing units. This allows the pixel processor itself to complete the three functions of perception, storage, and calculation, and due to its parallel computing characteristics, it is very suitable for high-speed processing of matrix image data. Figure 4 As shown in the figure, when the pixel processor and the upper microprocessor work together, they can complete some more complex perception tasks, such as tactile feature capture and tactile feature inference. Both are core devices inside the focal plane processor, but the pixel processor array tends to perceive images and high-speed parallel operations, while the upper microprocessor tends to top-level control and complex logic operations.

[0091] (2) Tactile feature inference: After completing the tactile feature capture, the captured features will be input into the tactile feature inference module, the design goal of which depends on the requirements of the specific task. For common basic tasks, the visual tactile sensor needs to provide information such as contact texture, depth, shape, posture, force, torque, etc. when the robot interacts with the outside world. Therefore, the tactile feature inference module needs to calculate and solve based on the known tactile features using mathematical physics models or machine learning methods. In some downstream tasks, in addition to the above basic information, further analysis and inference may be required, such as determining whether the contact state is sliding, or classifying and identifying the contacted objects. In higher-level advanced tasks, it may even be necessary to directly implement end-to-end mapping from tactile features to control decisions, such as adjusting the robot's posture in real time according to the contact state, which usually requires fitting with a deep learning model. Due to the complexity of these tasks, the inference module is more suitable for running on a host microprocessor, but it still requires the coordination of the pixel processor array.

[0092] Furthermore, the sensing, storage and computing integrated visual tactile sensor provided in an embodiment of the present invention has a workflow at the hardware and software levels, including: (1) an external object comes into contact with the elastic body of the contact module, and based on different tactile perception principles, the tactile information is converted by the contact module into optical information (such as changes in pixel intensity distribution or movement of a mark position); (2) the optical information continues to propagate in the lighting environment provided by the lighting module until it is captured by the sensor of the camera module, which specifically refers to the focal plane processor in the embodiment of the present invention; (3) the photosensitive units distributed on the pixel processor array in the focal plane sensor capture the optical information and convert it into pixel values ​​and store it in an analog register; (4) according to the program currently running by the pixel processor, the pixel values ​​stored in the analog register can be used for direct parallel computing, such as image preprocessing, or analog-to-digital conversion. The pixel processor is converted and stored in a digital register for subsequent calculations, such as a binary neural network; (5) The upper microprocessor runs in parallel with the pixel processor and is responsible for providing instructions and communication for the pixel processor. When the pixel processor completes the sensing, storage and computing operations and obtains preliminary output data, the upper microprocessor can continue to perform subsequent processing, taking advantage of its own expertise in complex logical operations, and mapping the preliminary output data to the final output data, such as the neural network classification results, or the PWM wave signal for servo control. The specific mapping method depends on the software program and the specific task, which can also be custom programmed; (6) Finally, the above output data is transmitted through GPIO, serial port, Bluetooth, Wifi and other wired or wireless methods, and is sent from the sensing, storage and computing integrated visual and tactile sensor to the upper computer or downstream control equipment, such as servos, robotic arms, etc.

[0093] 3. Application scenarios of integrated vision and tactile sensors

[0094] (1) Compact multimodal visual-tactile sensors: Visual-tactile sensors are developing in the direction of miniaturization and multimodality, but the compactness of the equipment is often in conflict with the goal of multifunctionality. The integrated sensing, storage and computing visual-tactile sensor integrates the photosensitive unit, computing unit and storage unit through the application of the focal plane sensor processor, thereby eliminating the need for additional hardware resources (such as edge computing units), and significantly reducing the complexity of the system and the size of the device while maintaining the same performance. In addition, the sensor can also be flexibly programmed to match different contact modules and lighting modules to adjust its working mode, so that it has the potential to simultaneously output multiple information such as contact texture, contact shape, contact force, contact torque, contact temperature, etc.

[0095] (2) High-speed, low-energy vision-tactile sensors that integrate sensing and actuation: Currently, the output of most vision-tactile sensors is high-resolution image data, which needs to be processed by a host computer before it can be used for downstream control tasks. However, in many application scenarios, the energy consumption, volume, computing power, and data transmission bandwidth of the host computer are all limited. Vision-tactile sensors that integrate sensing, storage, and computing have significant advantages in performing such tasks. Their on-chip parallel computing has the characteristics of low power consumption and high efficiency, and can directly output control signals. They have the potential to achieve sensing-actuation integration and are very suitable for applications in small robot systems and high-speed robot systems.

[0096] (3) Large-scale tactile dexterous hands with multi-sensor integration: Dexterous hand manipulation with enhanced tactile perception is currently a hot development direction. Visual tactile sensors can provide them with high spatial resolution and tactile sensitivity, but they also face the problem of limited transmission bandwidth after large-scale integration and significantly increased computing pressure on the host computer. Visual tactile sensors with integrated sensing, storage and computing can effectively solve these problems. Through distributed on-chip parallel computing, they can reduce the computing pressure caused by the increase in the number of sensor integration. At the same time, since the original image data is converted into sparse and high-value tactile features, a significant increase in communication transmission pressure is avoided. The above characteristics make it very suitable for application in large-scale integrated tactile dexterous hands.

[0097] (4) On-chip neural network operation and feature encoding to assist embodied intelligent tactile perception: The focal plane sensor processor used in the integrated visual tactile sensor has the ability to run on-chip neural networks. In addition to simple classification and regression tasks, it also has the potential to achieve encoding from raw image data to low-dimensional feature space. The volume of the encoded low-dimensional features is significantly reduced, but most of the effective information in the original data is retained with minimal information loss, which makes it suitable for working with advanced models in the host computer, saving the host computer's computing resources on the original data-based encoding tasks, thereby assisting the efficient integration of embodied intelligent systems and tactile perception.

[0098] like Figure 4 As shown, the integrated vision-tactile sensor and tactile perception method provided by the above embodiment of the present invention need to call the tactile perception algorithm for subsequent processing after obtaining satisfactory original optical features through data preprocessing. The design goal of the tactile perception algorithm is to convert the specific optical features contained in different perception principles (such as the degree of change in pixel brightness and distribution, the speed and distance of contact movement) into effective tactile features that can be directly used by the host computer through the integrated vision chip, including texture, depth, normal force, tangential force, torque, etc. It is mainly designed in two parts, tactile feature capture and tactile feature inference.

[0099] The first is to extract and capture tactile features. For the integrated visual tactile sensor based on the principle of intensity mapping, the tactile features include information such as pixel intensity, distribution and change. The pixel intensity can be the value of each pixel in the data. For example, the range of each pixel value I_i in the grayscale image is (0, 255). The pixel intensity distribution refers to the pixel array I_m*n of the entire image. The change in pixel intensity is based on a reference image I_r, and its change is calculated as I_delta = I_m*n-I_r. For the integrated visual tactile sensor based on the principle of marker displacement, its tactile features are the topological position P_n of the marker and its direction vector V_i, so that the displacement field of the entire marker array can be represented by the matrix V_n.

[0100] Then comes the tactile feature inference. After the required tactile features are correctly extracted, they need to be linked to the target tactile information to complete the inference from the tactile features to the tactile information actually required. There are two typical methods. One is explicit reasoning based on mathematical and physical models. Using the known mapping model M, the captured tactile features, such as the pixel intensity change I_delta or the marker array displacement field V_a, are converted into contact depth D_m*n=M(I_delta), contact force field F_m*n=M(V_a), or other target tactile information. The mapping model M can be established and corrected through theoretical modeling, data induction, and actual calibration. The other method is implicit reasoning based on binary neural networks. Similar to the machine learning models widely used on computers, similar algorithm models can also be deployed on the integrated sensing, storage and computing chip, such as lightweight binary convolutional neural networks (Binary-CNN), recurrent neural networks (RNN), etc. The specific usage is similar to method 1, but the network model M trained based on the target tactile information calibration data set is no longer based on mathematical and physical modeling.

[0101] Based on the above description, after obtaining satisfactory original optical features through data preprocessing, the tactile perception algorithm is called for subsequent processing, and the specific optical features contained in different perception principles (such as the degree of change in pixel brightness and distribution, the speed and distance of contact movement) can be converted into effective tactile features that can be directly used by the host computer through the integrated vision chip of sensing, storage and computing, including texture, depth, normal force, tangential force, torque, etc. If the tactile perception algorithm is further extended to the robot control algorithm, for example, the end position of the robot arm and the posture angle of the dexterous hand are considered in the modeling of the mapping model and the collection of training data of the network model, the control signals of robot equipment such as the robot arm and the dexterous hand can be directly inferred from the original optical features, completing high-speed and efficient end-to-end robot control tasks.

[0102] It should be noted that the explicit reasoning based on mathematical and physical models commonly used in tactile sensor research has not yet been applied to the sensing-storage-computing integrated sensor, and the implicit reasoning based on binary neural networks used in the sensing-storage-computing integrated sensor has never been applied to the use of tactile sensors. Therefore, the present invention not only successfully combines the sensing-storage-computing integrated sensor and the visual-tactile sensor in hardware design, but is also a breakthrough in the corresponding tactile perception method.

[0103] All matters not covered in the above embodiments of the present invention are well known in the art.

[0104] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A vision-tactile sensor with integrated sensing, storage and computing, characterized in that: include: A contact module, an illumination module, and a camera module; wherein: The contact module is used to convert physical contact information generated when contacting with an external object into corresponding optical information; The lighting module is used to provide a lighting environment and enable the optical information to continue to propagate in the lighting environment; The camera module includes a lens and a focal plane sensor processor which are arranged in sequence from front to back; wherein the lens is used to capture the optical information, and the focal plane sensor processor is used to convert the captured optical information to obtain processed tactile information.

2. The sensing, storage and computing integrated visual and tactile sensor according to claim 1, characterized in that: The contact module comprises: an elastic body and a lens arranged for contact from front to back, the front end of the elastic body may be provided with a coating, and the interior of the elastic body may be provided with a marking body.

3. The sensing, storage and computing integrated visual and tactile sensor according to claim 2, characterized in that: The contact module is used to convert physical contact information generated when contacting with an external object into corresponding optical information, including: Different tactile perception principles are used to convert the required tactile information into corresponding optical information based on actual tasks; wherein the perception principle includes: any one or any combination of intensity mapping principle, marker displacement principle and modal fusion principle.

4. The sensing, storage and computing integrated visual and tactile sensor according to claim 1, characterized in that: The lighting module comprises: a plurality of light sources; the plurality of light sources are arranged in an array between the contact module and the camera module; wherein: The light source is a same color or different color light source; The shape of the light source includes a point light source and / or a strip light source; The incident position of the light source includes a side surface, an upper side surface and / or an upper side of the contact module.

5. The sensing, storage and computing integrated visual and tactile sensor according to claim 1, characterized in that: The focal plane sensor processor comprises: a pixel processor array and a host microprocessor operating in parallel; wherein: The pixel processor array is used to convert the optical signal into an analog electrical signal, and perform calculations based on the analog electrical signal to obtain visual information; The upper microprocessor is used to perform command control and data transmission on the pixel processor array; at the same time, key information related to touch is extracted from the visual information to obtain processed tactile information and output it.

6. The sensing, storage and computing integrated visual and tactile sensor according to claim 5, characterized in that: The focal plane sensor processor is used to convert the captured optical information to obtain processed tactile information, including: The photosensitive units in the pixel processor array capture the original light signals in the optical information and convert them into analog electrical signals to obtain pixel values ​​which are stored in analog registers in the pixel processor array; the pixel values ​​stored in the analog registers are used to directly perform parallel numerical calculations and / or logical operations in the digital domain according to the required tasks, obtain corresponding visual information and store it in digital registers in the pixel processor array; The upper microprocessor extracts key information related to touch from the visual information to obtain processed tactile information.

7. The sensing, storage and computing integrated visual and tactile sensor according to claim 6, characterized in that: The pixel processor array further includes any one or more of the following: -First, random noise in the optical information is filtered out using a preset threshold, and then morphological operations are performed to remove and repair independent noise points to obtain the original optical signal; - For image data in the visual information that has positional offset and / or visual angle distortion on a spatial plane, the image data is corrected by means of parallel translation and / or distortion correction.

8. The sensing, storage and computing integrated visual and tactile sensor according to claim 5, characterized in that: The upper microprocessor extracts key information related to touch from the visual information to obtain processed tactile information, including any one or more of the following methods: - Based on explicit reasoning of mathematical and physical models, using known mapping models, the required key information is extracted from the visual information and converted into target tactile information; - Based on implicit reasoning of binarized neural networks, known deep learning models are used to extract the required key information from the visual information and convert it into target tactile information.

9. A vision-tactile sensor working system integrating sensing, storage and computing, characterized in that: include: Integrated sensor front end, signal transmission end and host computer back end; among them: The sensor front end adopts a sensing, storage and computing integrated visual and tactile sensor, which is used to convert physical contact information into optical information, and processes the optical information based on a focal plane sensor processor to obtain processed tactile information; The signal transmission end is used to send the processed tactile information to the host computer back end; The host computer backend is used to execute a downstream algorithm on the received processed tactile information, so as to complete a designated task.

10. A tactile perception method of a vision-tactile sensor with integrated sensing, storage and computing, characterized in that: include: The external object comes into contact with the elastic body of the contact module at the front end of the sensor. Based on different perception principles, the tactile information is converted into optical information by the contact module. The optical information continues to propagate in the lighting environment provided by the illumination module in front of the sensor until it is captured by the focal plane sensor processor of the camera module; The photosensitive units distributed on the pixel processor array in the focal plane sensor processor capture the optical information and convert it into pixel values ​​and store them in the analog registers in the pixel processor array; Directly calculating the pixel values ​​in parallel according to the required tasks and storing them in digital registers within the pixel processor array for subsequent calculations; The upper microprocessor in the focal plane sensor processor runs in parallel with the pixel processor array to provide instructions and communications for the operation of the pixel processor array. When the pixel processor array completes the sense-storage-computing operation to obtain visual data, the upper microprocessor continues to perform subsequent calculations and maps the preliminary visual data to the processed tactile information, which is then sent to the upper computer back end to execute downstream algorithms to complete designated tasks.

Images