Human-computer interaction touch pressure evaluation method and system, computing device and medium

Abstract

The invention provides a man-machine interaction tactile pressure evaluation method and system, computing equipment and a medium. The method comprises the following steps: acquiring a product virtual model under a digital twin environment pre-established for a real product; the method comprises the following steps: collecting real-time pose information of a real product, and tested hand motion data, man-machine interaction contact point information and pressure data of a contact point in a man-machine interaction process of a tested hand and the real product; mapping and presenting the spatial position of the tested hand in a digital twin system based on the hand motion data; calculating and generating a hand pressure distribution thermodynamic diagram based on the man-machine interaction contact point information and the pressure data of the contact points, and mapping and presenting the hand pressure distribution thermodynamic diagram to the product virtual model; and evaluating the man-machine interaction pressure level of a real product based on the spatial position of the tested hand and the hand pressure distribution thermodynamic diagram which are mapped and presented in the digital twin system.

Description

Technical Field

[0001] The present invention relates to the technical field of human-computer interaction evaluation, and in particular to a human-computer interaction tactile pressure evaluation method and system, computing equipment and medium. Background Art

[0002] In today's society, people-oriented has become the mainstream of social thought. Ergonomics accounts for an increasingly large proportion of product design factors, making the product's scope of use more extensive and more practical. At the same time, in production and life practice, products are not only used as a tool for life, but more and more people pay attention to the rationality and comfort of products during use. Evaluating the rationality and comfort of ergonomic factors at the contact surface between the product and the human body can make the product more easily accepted by consumers during use and achieve better consumption effects.

[0003] Existing technologies for measuring the contact pressure distribution between hands and objects often rely on more traditional physical methods, such as applying dye to the hands or using clay models to record contact marks. During implementation, these methods are easily interfered by factors such as uneven dyeing or incomplete contact between clay and hands, which can lead to deviations and distortions in the measurement results. For example, the dye may not evenly cover all contact areas, or the clay may not accurately reflect all pressure details during molding. These limitations reduce the reliability and accuracy of the measurement data and are not conducive to accurately analyzing the mechanical properties of the interaction between the hand and the product surface.

[0004] For example, in the hand-staining or clay modeling method, when pressure marks are left on the hand or product surface, they are recorded by taking photos. This method depends largely on the lighting conditions, camera angle, and image resolution when taking photos. These external variables may lead to subjectivity and inconsistency in data collection. Since changes in light intensity may cause color distortion, and different shooting angles may affect the accurate judgment of the size and shape of the pressure area, in this method, it is difficult to objectively and accurately convert the data obtained by taking photos alone into a digital pressure distribution map. Summary of the invention

[0005] In view of this, embodiments of the present invention provide a method and system for evaluating tactile pressure in human-computer interaction, a computing device, and a medium to eliminate or improve one or more defects in the prior art.

[0006] One aspect of the present invention provides a method for evaluating tactile pressure in human-computer interaction, the method comprising: obtaining a product virtual model in a digital twin environment pre-built for a real product; collecting real-time posture information of the real product, as well as the subject's hand motion data, human-computer interaction contact point information and contact point pressure data during the human-computer interaction between the subject's hand and the real product; mapping and presenting the spatial position of the subject's hand in the digital twin system based on the hand motion data; generating a hand pressure distribution heat map based on the human-computer interaction contact point information and the contact point pressure data, and mapping and presenting the hand pressure distribution heat map to the product virtual model; and evaluating the human-computer interaction pressure level of the real product based on the spatial position of the subject's hand and the hand pressure distribution heat map mapped and presented in the digital twin system.

[0007] In some embodiments of the present invention, before collecting the real-time position and posture information of the real product, the method also includes a spatial coordinate system building step, including: establishing a unified spatial coordinate system for the real product in the real environment and the product virtual model in the digital twin environment, and establishing a spatial mapping relationship between each coordinate point of the real product and the product virtual model; wherein, the position coordinates of the real product are collected in real time by using an external position tracker pre-installed in the real environment, and the angle information of the real product is collected in real time by using a first IMU inertial measurement unit pre-installed at the bottom of the real product.

[0008] In some embodiments of the present invention, the method also includes the step of pre-building a virtual model of the product in a digital twin environment, including: using a 3D scanning device to scan the surface geometry information of a real product, generating point cloud data based on the surface geometry information, using three-dimensional modeling software to create a corresponding three-dimensional model of the real product based on the point cloud data, and inputting the three-dimensional model into the digital twin system to obtain a virtual model of the product in the digital twin environment.

[0009] In some embodiments of the present invention, the method further includes a step of optimizing the generated point cloud data, including: performing denoising, hole filling and alignment and splicing processing on the point cloud data.

[0010] In some embodiments of the present invention, the step of establishing a unified spatial coordinate system for the real product in the real environment and the virtual model of the product in the digital twin environment includes: selecting the same position as the origin for the real product in the real environment and the virtual model of the product in the digital twin environment, and selecting coordinate axes with consistent spatial directions; calibrating the size parameters of the virtual model of the product based on the size data of the real product collected in advance; calibrating the position of the origin based on the position coordinates of the real product collected by an external position tracker; and calibrating the spatial direction of the coordinate axis based on the angle information of the real product collected by the first IMU inertial measurement unit.

[0011] In some embodiments of the present invention, the external position tracker includes one or more of an optical tracker, a laser tracker, and a body posture perception sensor; during the human-computer interaction process, the hand motion data, human-computer interaction contact point information, and contact point pressure data are collected through sensors integrated in the hand smart wearable device worn by the subject, and the sensors integrated in the hand smart wearable device include multiple types of flexible sensors, biomechanical sensors, a second IMU inertial measurement unit, and a precision positioning sensor, and the sensors are distributed in the fingers, joints, and palm areas of the hand smart wearable device.

[0012] In some embodiments of the present invention, the step of real-time visualization of the hand pressure distribution heat map in the digital twin system includes: flattening the three-dimensional geometric surface of the product virtual model into a two-dimensional representation, obtaining the two-dimensional texture coordinates of the three-dimensional surface points of the product virtual model, and when human-computer interaction occurs, wrapping the hand pressure distribution heat map to the surface of the product virtual model based on the two-dimensional texture coordinates in the digital twin system.

[0013] In some embodiments of the present invention, during the visualization of human-computer interaction results, the method further includes: identifying the posture of the real product, and when the posture of the real product in the real environment changes due to direct manipulation by the subject, the virtual model of the product in the digital twin environment changes the visualization effect in real time as the posture of the real product changes.

[0014] In some embodiments of the present invention, the hand motion data includes precise position information of hand interaction, hand movement information and hand motion trajectory, the human-computer interaction contact point information includes human-computer interaction contact area and human-computer interaction contact point position, and the contact point pressure data includes hand pressure visualization data, product pressure visualization mapping data and pressure statistical data, and the pressure statistical data includes at least two of single-point pressure index, area pressure value, maximum pressure value and average pressure value.

[0015] In some embodiments of the present invention, the step of evaluating the human-computer interaction stress level of a real product by using the motion data of the subject's hand and the heat map of hand pressure distribution presented in real time as a visualization in the digital twin system includes: determining the respective evaluation scores of the statistical hand motion data, human-computer interaction contact point information and contact point pressure data according to preset evaluation criteria, and calculating the final real product human-computer interaction stress level evaluation result by weighting according to preset weights.

[0016] Another aspect of the present invention provides a human-computer interaction tactile pressure evaluation system, the system comprising: a digital twin system, for obtaining a product virtual model in a digital twin environment pre-built for a real product; collecting real-time posture information of the real product, as well as the subject's hand motion data, human-computer interaction contact point information and contact point pressure data during the human-computer interaction between the subject's hand and the real product; mapping and presenting the spatial position of the subject's hand in the digital twin system based on the hand motion data; generating a hand pressure distribution heat map based on the human-computer interaction contact point information and the contact point pressure data, and mapping and presenting the hand pressure distribution heat map to the product virtual model; and evaluating the human-computer interaction pressure level of the real product based on the spatial position of the subject's hand and the hand pressure distribution heat map mapped and presented in the digital twin system.

[0017] In some embodiments of the present invention, the system also includes: an external position tracker pre-installed in a real environment, used to collect the position coordinates of the real product in real time; and a first IMU inertial measurement unit pre-installed at the bottom of the real product, used to collect the angle information of the real product in real time.

[0018] In some embodiments of the present invention, the external position tracker includes one or more of an optical tracker, a laser tracker, and a body posture perception sensor; during the human-computer interaction process, the hand motion data, human-computer interaction contact point information, and contact point pressure data are collected through sensors integrated in the hand smart wearable device worn by the subject, and the sensors integrated in the hand smart wearable device include multiple types of flexible sensors, biomechanical sensors, a second IMU inertial measurement unit, and a precision positioning sensor, and the sensors are distributed in the fingers, joints, and palm areas of the hand smart wearable device.

[0019] Another aspect of the present invention provides a computing device, including a processor, a memory, and a computer program / instruction stored in the memory, wherein the processor is used to execute the computer program / instruction, and when the computer program / instruction is executed, the computing device implements the steps of the method described in any one of the above embodiments.

[0020] Another aspect of the present invention provides a computer-readable storage medium having a computer program / instruction stored thereon, which implements the steps of the method described in any one of the above embodiments when the computer program / instruction is executed by a processor.

[0021] The human-computer interaction tactile pressure evaluation method and system, computing device and medium proposed in the present invention can establish real products in a real environment and virtual models of products in a digital twin environment in a unified spatial coordinate system, and use an external position tracker to collect the position coordinates of the real product in real time, and use the first IMU inertial measurement unit to collect the angle information of the real product in real time. During the human-computer interaction process, the sensor integrated in the smart wearable device worn by the subject collects the hand motion data, human-computer interaction contact point information and contact point pressure data of the subject's hand in the unified spatial coordinate system, thereby achieving accurate and comprehensive evaluation of the human-computer interaction pressure level of the real product, and getting rid of the problems of physical methods that are difficult to accurately control variables and difficult to digitally display.

[0022] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention may be achieved and obtained by the structures specifically indicated in the specification and the accompanying drawings.

[0023] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of the present application, and do not constitute a limitation of the present invention. In the drawings:

[0025] Figure 1 This is a flow chart of a method for visualizing and evaluating human-computer interaction pressure of a real product in one embodiment of the present invention.

[0026] Figure 2 Schematic diagram of an external position tracker and an IMU inertial measurement unit in one embodiment of the present invention.

[0027] Figure 3 This is a schematic diagram of product calibration in a real environment and a digital twin environment in one embodiment of the present invention.

[0028] Figure 4 The figure is a schematic diagram of coordinate point mapping between a real environment and a digital twin environment in one embodiment of the present invention.

[0029] Figure 5 FIG. 1 is a schematic diagram of the structure of a computing device in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0031] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0032] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0033] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0034] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0035] The design of the hand contact surface of a product is one of the most complex research topics in the field of product human-machine form design. The main reasons are: ① The shape of the palm surface is very complex; ② The palm surface will also produce corresponding deformation under different working postures; ③ When subjected to external mechanical pressure during work, the palm surface will undergo complex deformation, that is, contact deformation; ④ Pain is an important factor in the subjective human-machine evaluation of the contact surface. The palm has abundant sensory nerve endings, a low pain perception threshold, and is more sensitive to adverse feelings such as pain. At the same time, the human-machine interaction contact surface of the hand is mostly in the form of a curved surface, and the contact mechanism is relatively complex. The human-machine performance quality of the contact surface is affected by many factors such as the contact surface morphology, material properties, physical characteristics of the target user group, biomechanical characteristics of the human body, contact posture, vibration characteristics, and subjective psychological feelings. There are many problems in its design methods and evaluation criteria, which need to be studied.

[0036] The existing technical problems in evaluating the human-computer interaction contact surface of a product are: (1) it is difficult to accurately control the variables when using physical methods for measurement; (2) with physical measurement methods, the contact surface is difficult to display digitally; and (3) there is a lack of a visualization method that can reflect the pressure generated by the contact between the hand and the surface of the object on the product.

[0037] In order to overcome the problems existing in the prior art, the present invention aims to propose a human-computer interaction tactile pressure assessment method and system, computing equipment and medium, aiming to innovate the traditional human-computer interaction pressure assessment mode, and realize the precision, visualization and intelligence of the assessment process through highly integrated digital technology.

[0038] Figure 1 This is a flow chart of a method for visualizing the human-computer interaction pressure of a real product in an embodiment of the present invention. The method comprises the following steps:

[0039] Step S110: Obtain a product virtual model in a digital twin environment pre-built for the real product.

[0040] Step S120: collecting real-time posture information of the real product, as well as the subject's hand movement data, human-computer interaction contact point information and contact point pressure data during the human-computer interaction between the subject's hand and the real product.

[0041] In the specific implementation process, during the human-computer interaction process, information such as hand movement data, human-computer interaction contact point information and contact point pressure data can be collected through sensors integrated in the smart wearable device worn by the subject.

[0042] Step S130: Mapping and presenting the spatial position of the subject's hand in the digital twin system based on the hand motion data.

[0043] Step S140: generating a hand pressure distribution heat map based on the human-computer interaction contact point information and the pressure data of the contact points, and mapping the hand pressure distribution heat map to the product virtual model.

[0044] Step S150: Based on the spatial position of the subject's hand and the hand pressure distribution heat map mapped and presented in the digital twin system, the human-computer interaction pressure level of the real product is evaluated.

[0045] The human-computer interaction tactile pressure evaluation method proposed in the present invention can establish the real product in the real environment and the virtual model of the product in the digital twin environment in a unified spatial coordinate system, and use an external position tracker to collect the position coordinates of the real product in real time, and use the first IMU inertial measurement unit to collect the angle information of the real product in real time. During the human-computer interaction process, the sensor integrated in the smart wearable device worn by the subject collects the hand motion data, human-computer interaction contact point information and contact point pressure data of the subject's hand in the unified spatial coordinate system, thereby realizing accurate and comprehensive evaluation of the human-computer interaction pressure level of the real product, and getting rid of the problems of physical methods that are difficult to accurately control variables and difficult to digitally display.

[0046] In some embodiments of the present invention, before collecting the real-time position and posture information of the real product, the method further includes a spatial coordinate system building step, which includes: establishing a unified spatial coordinate system for the real product in the real environment and the product virtual model in the digital twin environment, and establishing a spatial mapping relationship between each coordinate point of the real product and the product virtual model. Among them, the position coordinates of the real product can be collected in real time using an external position tracker pre-installed in the real environment, and the angle information of the real product can be collected in real time using a first IMU inertial measurement unit pre-installed at the bottom of the real product.

[0047] By adopting this embodiment of the invention, a unified spatial coordinate system can be established, thereby establishing an association between the real product in the space and the virtual model of the product in the digital twin environment, preparing for the subsequent human-computer interaction tactile pressure evaluation.

[0048] In some embodiments of the present invention, the method also includes the step of pre-building a virtual model of the product in a digital twin environment, including: using a 3D scanning device to scan the surface geometry information of a real product, generating point cloud data based on the surface geometry information, using a three-dimensional modeling software to create a corresponding three-dimensional model of the real product based on the point cloud data, and inputting the three-dimensional model into the digital twin system to obtain a virtual model of the product in the digital twin environment.

[0049] By adopting the embodiment of the invention, the preparatory steps for the tactile pressure evaluation of human-computer interaction are further refined, and the construction of a product virtual model in a digital twin environment is realized, thereby preparing for the subsequent tactile pressure evaluation of human-computer interaction.

[0050] In some embodiments of the present invention, step S110 also includes an optimization processing step for the generated point cloud data, including: denoising, hole filling and alignment and splicing of the point cloud data. Specifically, it includes: (1) Denoising is the first step in point cloud data processing, which aims to remove noise points caused by sensor errors, environmental factors or other interferences in the data acquisition process. (2) Hole filling is the process of repairing holes or missing areas in the point cloud data. Holes may be caused by incomplete data acquisition, occlusion or damage to the surface of the object. (3) Alignment and splicing is the process of aligning and merging multiple point cloud data sets into a complete three-dimensional model. This is usually used to reconstruct a complete object or scene from point cloud data collected from multiple perspectives or positions.

[0051] By adopting the embodiment of the invention, the point cloud data can be made purer.

[0052] In some embodiments of the present invention, the step of establishing a unified spatial coordinate system for the real product in the real environment and the virtual model of the product in the digital twin environment in step S120 includes:

[0053] (1) The real product in the real environment and the virtual model of the product in the digital twin environment select the same position as the origin and select the coordinate axis with the same spatial direction;

[0054] (2) calibrating the dimensional parameters of the product virtual model based on the dimensional data of the real product collected in advance;

[0055] (3) Calibrate the origin position based on the position coordinates of the real product collected by the external position tracker;

[0056] (4) Calibrate the spatial direction of the coordinate axis based on the angle information of the real product collected by the first IMU inertial measurement unit.

[0057] By adopting this embodiment of the invention, it is possible to utilize the uniformity of the coordinate system calibrated by the external position tracker and the IMU inertial measurement unit, thereby making the virtual model in the digital twin system more accurate.

[0058] In some embodiments of the present invention, the external position tracker includes one or more of an optical tracker, a laser tracker, and a body posture sensing sensor.

[0059] In some further embodiments of the present invention, during the human-computer interaction process, the hand motion data, human-computer interaction contact point information and contact point pressure data are collected through sensors integrated in the hand smart wearable device worn by the subject, and the sensors integrated in the hand smart wearable device include flexible sensors, biomechanical sensors, a second IMU inertial measurement unit and precision positioning sensors. The sensors are distributed in the fingers, joints and palm areas of the hand smart wearable device.

[0060] In some embodiments of the present invention, the step of real-time visualization of the hand pressure distribution heat map in the digital twin system includes: performing a UV unfolding operation on the surface of the product virtual model to obtain two-dimensional texture coordinates of the three-dimensional surface points; when human-computer interaction occurs, wrapping the hand pressure distribution heat map to the surface of the product virtual model based on the two-dimensional texture coordinates in the digital twin system.

[0061] By adopting the embodiment of the invention, the mapping between two-dimensional coordinates and three-dimensional coordinates can be realized. In the prior art, the visualization data of the pressure generated by the contact between the hand and the surface of the object is generally limited to displaying the pressure distribution information directly captured by the sensor. However, if the intention is to accurately correspond these data to the actual contact area of ​​the product, it is necessary to use data mapping technology, combined with computer-aided three-dimensional modeling methods, complex algorithm processing and advanced function simulation methods. This process usually makes it difficult to ensure the accuracy of the mapping, resulting in possible deviations in the final hand contact coordinate position, affecting the accurate restoration of the pressure distribution details in the real interaction situation.

[0062] In some embodiments of the present invention, during the visualization of human-computer interaction results, the method further includes: identifying the posture of the real product, and when the posture of the real product in the real environment changes due to direct manipulation by the subject, the virtual model of the product in the digital twin environment changes the visualization presentation effect in real time as the posture of the real product changes.

[0063] Using this embodiment of the invention, the hand motion data includes the precise position information of hand interaction, hand movement information and hand motion trajectory, the human-computer interaction contact point information includes the human-computer interaction contact area and the human-computer interaction contact point position, the pressure data of the contact point includes hand pressure visualization data, product pressure visualization mapping data and pressure statistical data, and the pressure statistical data includes at least two of the single-point pressure index, regional pressure value, maximum pressure value and average pressure value.

[0064] Furthermore, the step of evaluating the human-computer interaction stress level of a real product by utilizing the real-time visualization of the subject's hand motion data and the hand pressure distribution heat map in the digital twin system includes: determining the respective evaluation scores of the statistical hand motion data, human-computer interaction contact point information, and contact point pressure data according to preset evaluation criteria, and calculating the final real product human-computer interaction stress level evaluation result by weighting according to preset weights.

[0065] By adopting the embodiment of the invention, the human-computer interaction stress level of a real product can be evaluated in a comprehensive manner from different dimensions to obtain a more comprehensive evaluation result.

[0066] In some embodiments of the present invention, the method also includes the step of pre-building a virtual model of the product in a digital twin environment, including: using a 3D scanning device to scan the surface geometry information of a real product, generating point cloud data based on the surface geometry information, using three-dimensional modeling software to create a corresponding three-dimensional model of the real product based on the point cloud data, and inputting the three-dimensional model into the digital twin system to obtain a virtual model of the product in the digital twin environment.

[0067] By adopting the embodiment of the invention, a virtual model of the product in a digital twin environment can be built using 3D scanning equipment, three-dimensional modeling software and a digital twin system.

[0068] During the specific implementation process, the real product needs to be 3D scanned, point cloud processed and 3D modeled.

[0069] In one embodiment of the present invention, a 3D scanning device (such as a laser scanner, a structured light scanner, etc.) is first used to comprehensively scan the real product, collect its surface geometry information, and generate raw point cloud data. Secondly, the raw point cloud is subjected to operations such as denoising, hole filling, and alignment and splicing through point cloud processing to improve data quality. The processed point cloud data is imported into a 3D modeling software for surface reconstruction and solid modeling to create an accurate 3D model, ensuring that the geometric size and shape of the model are consistent with the real product.

[0070] At the same time, the virtual and real spaces are connected through the data interface (connecting the virtual space and the physical world), and the data collected in the physical space is transmitted to the virtual space to update the digital twin model. The real product digital twin model thus constructed provides a digital twin environment for product human-computer interaction.

[0071] In the specific implementation process, it is necessary to establish the same coordinate system in the real world and virtual space, and install positioning sensors in the real world. Determine the three-dimensional coordinates of the key reference points of the product, and define the global coordinate system in the real environment. At the same time, create a matching coordinate system in the virtual environment to ensure that the two are consistent in spatial direction and origin. Figure 2 Schematic diagram of an external position tracker and an IMU inertial measurement unit in one embodiment of the present invention.

[0072] In one embodiment of the present invention, Figure 2 As shown, according to the application requirements, on the one hand, an external position tracker (or positioning sensor system) for sensing and positioning, such as an optical tracker, a laser tracker, and a body perception sensor, etc., pre-installed in the environment is used to locate and track the position of the real product; on the other hand, an IMU inertial measurement unit is installed at the bottom of the real product to clarify the angle, direction, etc. of the real product. The present invention is not limited to this, and the above-mentioned sensor types are only examples.

[0073] During the specific implementation process, it is necessary to calibrate the positions of the real product and the product virtual model, including product position, size, and angle.

[0074] Before interaction, it is first necessary to compare the data obtained through precise measurement and scanning with the actual product of known dimensions, compare the key dimensions of the product virtual model with the actual product, adjust the model proportions, and ensure dimensional accuracy.

[0075] Adjust the global position of the model to ensure that the absolute position of the model in the virtual space is correct. Use positioning sensors to calibrate the real product position, calibrate the product position data, and transmit it to the digital twin system. Adjust the model angle to ensure that the model is in a horizontal direction, and combine the product with the IMU inertial measurement unit to measure the product angle, calibrate the product angle data, and transmit it to the digital twin system so that the posture of the model in the digital twin system is consistent with the real product.

[0076] After completing the fine calibration of the product's position, size, and angle, the digital twin system will accurately establish the spatial correspondence between the sensor system and each coordinate point of the product. On this basis, if the product's posture is adjusted during human-computer interaction, the built-in IMU inertial measurement unit will immediately detect the angle offset and instantly transmit this dynamic data to the positioning sensor system. Through this integrated communication mechanism, the system can quickly calculate and update the precise position information of each coordinate point under the new posture of the product, ensuring that the consistency between the virtual model and the physical product is always accurate, thereby improving the real-time and accuracy of interactive feedback.

[0077] Figure 3 FIG. 1 is a schematic diagram of product calibration in a real environment and a digital twin environment in one embodiment of the present invention. Figure 3 As shown, when calibrating the product, if the physical dimensions of the actual object are 60 cm in height and 40 cm in width, the digital twin system will synchronize the real dimensions to ensure that the virtual model is exactly the same size as the real object. This process not only involves precise matching of dimensions, but also requires ensuring that their spatial positioning is consistent, that is, the products in the real world and the digital twin environment must strictly share the same coordinate reference frame. Any point in the real space, such as a point with coordinates X(x,y,z), must have a completely corresponding position in the virtual coordinate system to the corresponding point X'(x',y',z') in the digital twin, achieving precise point-to-point mapping between the two worlds, ensuring all-round consistency and accuracy from the physical to the digital mirror.

[0078] During the specific implementation process, it is necessary to interact with the product through smart gloves and determine the product contact position through coordinate points.

[0079] Smart gloves integrate a variety of sensor technologies, including flexible sensors, biomechanical sensors, IMU inertial measurement units and precision positioning sensors, which are distributed throughout the fingers, joints and palm areas. They work together to capture every subtle gesture and pressure data of the user with high precision, while ensuring that the spatial position of the gloves when interacting with the real product can be accurately tracked and displayed in the digital twin system.

[0080] When operators (i.e. test subjects, or test subjects for short) wear the smart gloves to directly operate physical products, the gloves can collect detailed gesture movement paths, pressure distribution information of contact points, and precise coordinates of fingers and joints in three-dimensional space at each contact moment. At the same time, they are transmitted to the digital twin system in real time to achieve data fusion with the pre-built high-fidelity virtual product model and hand model.

[0081] By combining the virtual product model with the hand model and positioning data in the digital twin system, every exact contact point between the operator and the product in the virtual environment is located, and a heat map of the hand pressure distribution is accurately mapped, intuitively reflecting the pressure concentration area and distribution pattern in the real interaction scenario.

[0082] Specifically, when the operator wears smart gloves and interacts with the real product and forms contact, the sensor network built into the gloves immediately captures the precise coordinate information of this contact event, and this data is then instantly transmitted to the corresponding virtual hand model in the digital twin system. In the digital twin system, through the precise matching of advanced algorithms and sensor data, real-time spatial alignment and synchronization are achieved between the virtual hand and the virtual product model, ensuring that each contact is not only accurate in position, but also has the same dynamic effect as reality. Once the virtual hand coordinates and the virtual product model coordinates are precisely aligned, the system automatically identifies and records the contact event, and records the coordinate data of each contact in detail. These data are further used to generate a detailed hand pressure distribution heat map in the digital twin system, intuitively and accurately depicting the distribution and intensity changes of pressure in each real interaction.

[0083] Figure 4 Schematic diagram of coordinate point mapping between the real environment and the digital twin environment in one embodiment of the present invention. Figure 4 As shown in the figure, in a real environment, the contact point between the index finger of the smart glove and the real object is detected as point X(x,y,z), and the thumb is point A(a,b,c). Then the digital twin system will automatically synchronize its coordinate point information to X'(x',y',z') and A'(a',b',c').

[0084] During the specific implementation process, data visualization mapping needs to be realized.

[0085] The data collected by the smart gloves is transmitted to the processing system, and the system will generate a visualized heat map of the hand pressure distribution. According to the above contact rules, if contact is made, the coordinate data of each contact will be recorded, and the data will be visualized by specific coordinates. In addition, in the 3D software, the hand pressure distribution heat map is mapped to the virtual product according to the contact and mapping rules.

[0086] In the mapping process, it is necessary to establish the corresponding relationship between the three-dimensional coordinates of the object and the pixel coordinates of its two-dimensional image. In the three-dimensional software, the three-dimensional geometric surface of the product model is flattened into a two-dimensional representation, that is, the UV unfolding operation is performed on each surface of the product model to obtain the UV coordinates (i.e., two-dimensional texture coordinates) of the three-dimensional surface points of the product virtual model, ensuring that the two-dimensional image can be accurately fitted to the three-dimensional model. In this process, it should be ensured that each UV coordinate obtained is consistent with the surface space coordinates before unfolding, ensuring that the precise coordinate information of the contact event can be captured during mapping. Among them, UV unfolding refers to mapping the texture coordinates (usually called UV coordinates) of the surface of the three-dimensional model to a two-dimensional plane so that texture mapping can be easily applied. Among them, UV unfolding is an important process in 3D modeling. It refers to unfolding or flattening the 3D geometry into a 2D representation so that each polygon and face of the 3D object can be bound to the corresponding area in the UV map.

[0087] When the smart glove comes into contact with the product in the real world, the digital twin system determines that a contact event has occurred and obtains the three-dimensional contact coordinate points of the virtual product. The system will automatically match the obtained image with the UV coordinates of the three-dimensional model, automatically "wrap" the hand pressure distribution heat map onto the surface of the three-dimensional model, and optimize and adjust it through the corresponding algorithm.

[0088] In the specific implementation process, it is necessary to realize real-time display of data. When users interact with physical objects, the system detects and identifies their posture in real time, calculates the posture of the physical object identified in the depth camera coordinate system in the projector coordinate system, and adjusts it in real time in the 3D software. Therefore, when the user directly manipulates the objects in the real scene, the model in the virtual scene will also show the same changes as the user's movement and rotation, and the hand pressure data will be mapped in real time according to the corresponding changes.

[0089] The human-computer interaction tactile pressure evaluation method and system, computing device and medium proposed in this invention are designed to capture and analyze the hand pressure distribution of users when operating physical products, and then realize the visualization of hand pressure data on the product through the digital twin system. The core advantage of this system is that it can intuitively map abstract pressure data to a three-dimensional digital model, thus providing researchers with an innovative human-computer interaction research and analysis tool.

[0090] In the specific human-computer interaction evaluation process, it can be divided into the following steps:

[0091] 1) Preparation before the assessment

[0092] First, the physical product to be evaluated needs to be scanned to create a three-dimensional digital model of the product. Second, the same coordinate system needs to be established in the real world and virtual space, and relevant sensors need to be installed. At the same time, the product position, size, angle, etc. need to be calibrated. Finally, the smart gloves need to be prepared, connected, and tested.

[0093] 2) Human-computer interaction evaluation

[0094] By wearing smart gloves, people interact with real physical products and obtain hand pressure data. At the same time, the digital twin system will accurately determine whether the hand has actually interacted with the product through coordinate points. When the digital twin system determines that an interaction event has occurred, the system will determine the location of the interaction event on the product through the interaction coordinate points and perform data mapping. When people need to interact with the product multiple times, the digital twin system will determine the interaction events that have occurred in real time and record and map the data.

[0095] 3) Visual analysis of human-computer interaction pressure of physical products based on digital twin technology

[0096] The system can provide a scientific analysis method for research in the field of human-computer interaction, allowing researchers to intuitively observe the specific state of human-product interaction. By superimposing visual data of hand pressure distribution on the surface of the product and accurately capturing each interaction event, the system can not only intuitively observe the specific form of the contact surface, but also quantify the details of pressure distribution during the interaction. This visualization method significantly improves the scientific nature and ergonomics of human-computer interaction design, making the product more in line with the user's physiological and psychological needs in terms of form and function.

[0097] The human-computer interaction tactile pressure evaluation system proposed in the present invention combines data sensing methods, product three-dimensional scanning methods, point cloud processing methods, product 3D simulation modeling methods, data three-dimensional mapping methods, and real-time data analysis methods. In a real-life on-site environment, the mechanical behavior in the physical world is accurately reproduced in a virtual way. As the contact state between the hand and the product changes, the pressure distribution on the 3D model is updated in real time and dynamically visualized. Its interactive evaluation experience enables users to observe and analyze real-time changes in surface pressure of objects from different perspectives, as well as real-time changes of the contact surface following the object, so as to better conduct product human-computer interaction analysis.

[0098] The advantages of the human-computer interaction tactile pressure evaluation method proposed in the present invention are:

[0099] (1) By accurately reproducing the mechanical behavior in the physical world through virtual means, we can overcome the limitations of the existing pressure evaluation methods for real product human-computer interaction through traditional physical methods, and display the effective contact surface and pressure distribution data of the hand in a digital way.

[0100] (2) Visual display of the effective contact surface of the hand of a real product based on the digital twin system: When the hand is in contact with the real product, the mechanical data of the effective contact surface is accurately mapped to the product in the real world through the digital twin system, so that the mechanical data of the effective contact surface can be visualized in combination with the product.

[0101] (3) Accurate location of the effective contact surface between the real product and the hand in the digital twin system: The system ensures that the fine pressure distribution data generated by each interaction between the hand and the product in the real world can be accurately mapped in the virtual environment of the digital twin, and the location of each contact is calibrated. At the same time, the effective contact pressure surface data generated is transmitted to the system and intuitively presented on the real product simulated in the digital twin system.

[0102] (4) In the digital twin system, the pressure surface changes in real time with the interaction in the real world: In the application of the digital twin system, with the dynamic interaction between the hand and the product in the real world, the system can instantly capture and respond and accurately map the pressure distribution surface in the virtual environment, including real-time changes in the product angle and position.

[0103] (5) Visual analysis of hand pressure data combined with specific product contact surfaces: When performing data analysis, the hand pressure distribution data can be combined with the visualized product surface pressure data in combination with the specific product surface contacted by the hand to obtain more data analysis results.

[0104] (6) Visual analysis of pressure in various environments and throughout the life cycle of various products: The system can be applied indoors and outdoors, and can be combined with different types of products, such as home appliances, furniture, handheld products, etc., to map pressure data onto the products at different stages for visual analysis.

[0105] Corresponding to the above method, the present invention also provides a human-computer interaction tactile pressure evaluation system, the system comprising: a digital twin system, for obtaining a product virtual model in a digital twin environment pre-built for a real product; collecting real-time posture information of the real product, as well as the subject's hand motion data, human-computer interaction contact point information and contact point pressure data during the human-computer interaction between the subject's hand and the real product; mapping and presenting the spatial position of the subject's hand in the digital twin system based on the hand motion data; generating a hand pressure distribution heat map based on the human-computer interaction contact point information and the contact point pressure data, and mapping and presenting the hand pressure distribution heat map to the product virtual model; based on the spatial position of the subject's hand and the hand pressure distribution heat map mapped and presented in the digital twin system, evaluating the human-computer interaction pressure level of the real product.

[0106] In some embodiments of the present invention, the system also includes: an external position tracker pre-installed in a real environment, used to collect the position coordinates of the real product in real time; and a first IMU inertial measurement unit pre-installed at the bottom of the real product, used to collect the angle information of the real product in real time.

[0107] Furthermore, in some embodiments of the present invention, the external position tracker includes one or more of an optical tracker, a laser tracker, and a body posture sensing sensor. In the process of human-computer interaction, the hand motion data, human-computer interaction contact point information, and contact point pressure data are collected by sensors integrated in the hand smart wearable device worn by the subject, and the sensors integrated in the hand smart wearable device include multiple types of flexible sensors, biomechanical sensors, second IMU inertial measurement units, and precision positioning sensors, and the sensors are distributed in the fingers, joints, and palm areas of the hand smart wearable device.

[0108] The core advantage of the human-computer interaction tactile pressure evaluation system proposed by the present invention is that it can observe the visualization data of the specific contact surface during human-computer interaction through the product. In addition, the system can obtain multiple evaluation index data, for example, the evaluation index data may include the precise position of hand interaction, hand pressure visualization data, product pressure visualization mapping data, pressure statistics data (at least two of single-point pressure value, regional pressure value, maximum pressure, average pressure value, etc.) and human-computer interaction contact area, etc.

[0109] The human-computer interaction tactile pressure evaluation system proposed in the present invention can be widely used in multiple fields, such as industrial design, user experience, human-computer interaction, medical and health, sports, etc., and covers different product types, including household products, handheld devices, sports equipment, medical equipment, etc. In ergonomic research in different fields, through the precise quantification of force, the system can define the reasonable range of hand strength, guide the optimal design of the product contact surface, ensure that the product fully takes into account ergonomic principles while meeting functionality, and provide users with a more comfortable and safe use experience.

[0110] Corresponding to the above method, the present invention also provides a computing device, which includes a processor and a memory, wherein the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the device / system implements the steps of the above method. This computing device is the computing processing core of the digital twin system.

[0111] See also Figure 5The computing device 00 includes: a processor 01, a memory 02, and a computer program stored in the memory 02 and executable on the processor 01. When the processor 01 executes the computer program, the human factor data server access control method provided in the above method embodiment is implemented.

[0112] Among them, the processor 01 is connected to the memory 02, such as through a bus 03. The processor 01 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It can implement or execute various exemplary logic blocks, modules and circuits described in combination with the contents disclosed in this application. The processor 01 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of DSP and microprocessors, etc. The bus 03 may include a path to transmit information between the above components. The bus 03 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 130 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The memory 02 is used to store a computer program corresponding to the human factor data server access control method of the above embodiment of the present application, and the computer program is controlled and executed by the processor 01. The processor 01 is used to execute the computer program stored in the memory 02 to implement the content shown in the above method embodiment.

[0113] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0114] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0115] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0116] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for evaluating tactile pressure in human-computer interaction, characterized in that: include: Get a virtual model of the product in a pre-built digital twin environment for the real product; Collecting real-time posture information of a real product, as well as hand motion data of the subject, human-computer interaction contact point information, and contact point pressure data during the human-computer interaction between the subject's hand and the real product; Mapping and presenting the spatial position of the subject's hand in the digital twin system based on the hand motion data; Generate a hand pressure distribution thermogram based on the human-computer interaction contact point information and the pressure data of the contact points, and map the hand pressure distribution thermogram to the product virtual model; Based on the spatial position of the subject's hand and the heat map of hand pressure distribution mapped in the digital twin system, the human-computer interaction pressure level of the real product is evaluated.

2. The method according to claim 1, characterized in that Before collecting the real-time position and posture information of the real product, the method further includes a step of establishing a spatial coordinate system, including: A unified spatial coordinate system is established for the real product in the real environment and the virtual model of the product in the digital twin environment, and a spatial mapping relationship between each coordinate point of the real product and the virtual model of the product is established; wherein, an external position tracker pre-installed in the real environment is used to collect the position coordinates of the real product in real time, and a first IMU inertial measurement unit pre-installed at the bottom of the real product is used to collect the angle information of the real product in real time.

3. The method according to claim 1, characterized in that: The method also includes the step of pre-building a product virtual model in a digital twin environment, including: Use a 3D scanning device to scan the surface geometry information of a real product, generate point cloud data based on the surface geometry information, use 3D modeling software to create a corresponding 3D model of the real product based on the point cloud data, and input the 3D model into the digital twin system to obtain a virtual model of the product in the digital twin environment.

4. The method according to claim 3, characterized in that: The method also includes an optimization processing step for the generated point cloud data, including: performing denoising processing, hole filling processing and alignment and splicing processing on the point cloud data.

5. The method according to claim 1, characterized in that The step of establishing a unified spatial coordinate system for the real product in the real environment and the virtual model of the product in the digital twin environment includes: The real product in the real environment and the virtual model of the product in the digital twin environment select the same position as the origin, and select the coordinate axis with the same spatial direction; Calibrate the product virtual model size parameters based on the real product size data collected in advance; Calibrate the origin position based on the position coordinates of the real product collected by an external position tracker; The spatial direction of the coordinate axis is calibrated based on the angle information of the real product collected by the first IMU inertial measurement unit.

6. The method according to claim 1, characterized in that The external position tracker includes one or more of an optical tracker, a laser tracker, and a body posture sensing sensor; During the human-computer interaction process, the hand motion data, human-computer interaction contact point information and contact point pressure data are collected through sensors integrated in the hand smart wearable device worn by the subject. The sensors integrated in the hand smart wearable device include multiple types of flexible sensors, biomechanical sensors, a second IMU inertial measurement unit and precision positioning sensors. The sensors are distributed in the fingers, joints and palm areas of the hand smart wearable device.

7. The method according to claim 1, characterized in that The step of presenting the hand pressure distribution thermal map in real time visualization in the digital twin system includes: The three-dimensional geometric surface of the product virtual model is flattened into a two-dimensional representation to obtain the two-dimensional texture coordinates of the three-dimensional surface points of the product virtual model. When human-computer interaction occurs, the hand pressure distribution heat map is wrapped onto the surface of the product virtual model based on the two-dimensional texture coordinates in the digital twin system.

8. The method according to claim 1, characterized in that: During the visualization process of the human-computer interaction results, the method also includes: identifying the posture of the real product. When the posture of the real product in the real environment changes due to direct manipulation by the subject, the virtual model of the product in the digital twin environment changes the visualization effect in real time as the posture of the real product changes.

9. The method according to claim 1, characterized in that: The hand motion data includes precise hand interaction position information, hand movement information and hand motion trajectory; the human-computer interaction contact point information includes human-computer interaction contact area and human-computer interaction contact point position; the contact point pressure data includes hand pressure visualization data, product pressure visualization mapping data and pressure statistical data; the pressure statistical data includes at least two of single-point pressure index, regional pressure value, maximum pressure value and average pressure value.

10. The method according to claim 7, characterized in that The step of evaluating the human-computer interaction pressure level of a real product by using the motion data of the subject's hand and the hand pressure distribution heat map presented in real time in the digital twin system, includes: The evaluation scores of the statistical hand motion data, human-computer interaction contact point information and contact point pressure data are determined according to the preset evaluation standards, and the final evaluation results of the real product human-computer interaction pressure level are calculated according to the preset weights.

11. A human-computer interaction tactile pressure evaluation system, characterized in that: The system comprises: The digital twin system is used to obtain a virtual model of the product in a digital twin environment pre-built for the real product; collect the real-time posture information of the real product, as well as the subject's hand movement data, human-computer interaction contact point information and contact point pressure data during the human-computer interaction between the subject's hand and the real product; Based on the hand motion data, the spatial position of the subject's hand is mapped and presented in the digital twin system; based on the human-computer interaction contact point information and the pressure data of the contact points, a hand pressure distribution heat map is generated, and the hand pressure distribution heat map is mapped and presented to the product virtual model; based on the spatial position of the subject's hand and the hand pressure distribution heat map mapped and presented in the digital twin system, the human-computer interaction pressure level of the real product is evaluated.

12. The system according to claim 11, characterized in that The system also includes: Pre-installed external location trackers in real-world environments to collect real-time location coordinates of real products; The first IMU inertial measurement unit pre-installed on the bottom of the real product is used to collect the angle information of the real product in real time.

13. The system according to claim 12, characterized in that The external position tracker includes one or more of an optical tracker, a laser tracker, and a body posture sensing sensor; During the human-computer interaction process, the hand motion data, human-computer interaction contact point information and contact point pressure data are collected through sensors integrated in the hand smart wearable device worn by the subject. The sensors integrated in the hand smart wearable device include multiple types of flexible sensors, biomechanical sensors, a second IMU inertial measurement unit and precision positioning sensors. The sensors are distributed in the fingers, joints and palm areas of the hand smart wearable device.

14. A computing device comprising a processor, a memory and a computer program / instructions stored in the memory, characterized in that: The processor is used to execute the computer program / instructions. When the computer program / instructions are executed, the computing device implements the steps of the method according to any one of claims 1 to 10.

15. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images