Multi-modal perception blind person shopping guide method, device and equipment and storage medium

Abstract

The invention discloses a multi-modal perception blind person shopping guide method, and relates to the technical field of blind person shopping guide, and the method comprises the steps: carrying out the optimization of touch coding parameters according to the multi-modal touch perception characteristics of a user, and generating a personalized touch coding scheme of the user; mapping an array of electroactive polymer haptic cells into a dynamic touchpad based on the haptic coding scheme and environmental information within the target area; obtaining user interaction data, dynamically adjusting the complexity of the touch coding scheme, and updating the dynamic touch pad; and according to the spatial audio technology and the updated dynamic touchpad, guiding the user to a position corresponding to the target goods to complete shopping. According to the method, the unique tactile perception ability of each user is fully considered, excessive information is prevented from being transmitted in a complex environment, the user can receive and understand the tactile information more easily, the user can gradually adapt to the complex tactile information in the using process through a dynamic adjustment mechanism, and adaptability and practicability are improved.

Description

Technical Field

[0001] This application relates to the field of shopping guidance technology for the blind, and more specifically, to a multimodal sensing method, apparatus, device, and storage medium for shopping guidance for the blind. Background Technology

[0002] Enabling blind people to shop independently in complex environments, especially in shopping scenarios, has always been a challenging task. With increasing societal emphasis on accessible environments, providing effective shopping assistance tools for visually impaired individuals has become a crucial research direction in the field of human-computer interaction. Traditional shopping assistance methods for the blind, such as canes and guide dogs, can meet basic needs in open environments, but in complex indoor environments like supermarkets and shopping malls, these methods often fail to provide sufficient environmental information and precise shopping guidance. Particularly in shopping scenarios, blind users not only need route navigation but also access to multi-dimensional environmental information such as product locations and shelf layouts, placing higher demands on the information presentation capabilities of shopping guides.

[0003] Currently, the mainstream blind shopping guide technology on the market is mainly based on vibration and tactile feedback. It adopts a standardized tactile coding scheme and uses vibration signals with fixed frequency, intensity, and duration to represent different shopping guide instructions. It conveys direction and obstacle information to users through vibration patterns. Some advanced technologies combine ultrasonic or infrared sensors to detect the distance to obstacles and encode distance information through changes in vibration intensity or frequency. This vibration and tactile feedback then alerts the blind person, thus achieving the shopping guide function.

[0004] However, existing technologies have several limitations: in complex environments, they often transmit too much information simultaneously, leading to tactile cognitive overload; tactile feedback has a low dimension, encoding information only through limited parameters such as vibration intensity, frequency, and duration, making it difficult to express complex environmental spatial structures and affecting shopping guide efficiency; and they lack adaptive support for the user's tactile learning process, failing to dynamically adjust interaction strategies as the user's abilities improve. These problems severely limit the practicality and effectiveness of tactile shopping guide devices in complex environments such as shopping, resulting in blind users still facing difficulties such as insufficient information access, low shopping guide efficiency, and safety risks in their daily lives. Summary of the Invention

[0005] To address at least one deficiency or improvement need in the prior art, the present invention provides a multimodal perception-based shopping guide method, apparatus, device, and storage medium for the blind. This invention solves the problems in the prior art where complex environments easily transmit too much information, leading to tactile cognitive overload; information can only be encoded using limited parameters such as vibration intensity, frequency, and duration, making it difficult to express complex environmental spatial structures and affecting shopping guide efficiency; and there is a lack of adaptive support for the user's tactile learning process, making it impossible to dynamically adjust the interaction strategy as the user's ability improves.

[0006] To achieve the above objectives, according to a first aspect of the present invention, a multimodal perception-based shopping guide method for the blind is provided, comprising: The tactile coding parameters are optimized based on the user's multimodal tactile perception characteristics to generate a personalized tactile coding scheme for the user. Based on the tactile coding scheme and environmental information within the target area, the electroactive polymer tactile unit array is mapped into a dynamic touchpad; Acquire user interaction data, dynamically adjust the complexity of the haptic coding scheme, and update the dynamic touchpad; Using spatial audio technology and an updated dynamic touchpad, users are guided to the location of their desired goods to complete their shopping.

[0007] In one possible implementation, the haptic coding parameters are optimized based on the user's multimodal haptic perception characteristics to generate a personalized haptic coding scheme for the user, and the implementation further includes: Based on the characteristics of multimodal tactile perception and the preset optimization algorithm, the tactile coding parameters are optimized to obtain the target tactile coding parameter set; The initial tactile coding is adjusted based on the target tactile coding parameter set to generate a user-personalized tactile coding scheme.

[0008] In one possible implementation, the target tactile coding parameter set includes spatial resolution characteristics, intensity sensitivity curves, frequency discrimination characteristics, and temporal resolution characteristics; adjusting the initial tactile coding based on the target tactile coding parameter set to generate a user-personalized tactile coding scheme further includes: An encoding scheme that adjusts the density and width of the encoded path lines based on spatial resolution characteristics; A nonlinear intensity mapping function is constructed based on the intensity sensitivity curve to optimize the coding scheme for obstacle distance and hazard level; Based on frequency discrimination characteristics, determine the coding scheme that corresponds frequency ranges with different information types; An encoding scheme that adjusts the presentation speed and temporal pattern of tactile sequences based on time-resolved characteristics.

[0009] In one possible implementation, acquiring user interaction data, dynamically adjusting the complexity of the haptic coding scheme, and updating the dynamic touchpad also includes: Based on user interaction data, assess the user's current tactile cognitive load; The complexity of the tactile coding scheme is dynamically adjusted based on the user's current tactile cognitive load. The dynamic touchpad is updated based on the adjusted haptic coding scheme.

[0010] In one possible implementation, assessing the user's current tactile cognitive load based on user interaction data also includes: A cognitive load index calculation model was constructed, and the dynamic threshold of cognitive load in the model was dynamically adjusted based on historical interaction data. The current interaction data is input into the adjusted cognitive load index calculation model to assess the user's current tactile cognitive load.

[0011] In one possible implementation, the complexity of the haptic coding scheme is dynamically adjusted based on the user's current haptic cognitive load, and it also includes: When the user's current tactile cognitive load exceeds the preset threshold, the line width of the encoding path is increased and the stimulation interval is extended. When the user's current tactile cognitive load is less than the preset threshold, the curvature and slope of the encoded path are increased, and obstacle contour encoding is set.

[0012] In one possible implementation, the user is guided to the location of the target goods to complete the shopping process using spatial audio technology and an updated dynamic touchpad, and the method further includes: Based on spatial audio technology and the updated dynamic touchpad, a mapping relationship between tactile sensation and spatial audio is established; Based on the mapping relationship between tactile sensation and spatial audio, the shopping process can be completed by confirming the location of the target goods.

[0013] According to a second aspect of the present invention, a multimodal sensing shopping guide device for the blind is also provided, comprising: The encoding module is configured to optimize tactile encoding parameters based on the user's multimodal tactile perception characteristics, generating a personalized tactile encoding scheme for the user. The mapping module is configured to map an array of electroactive polymer haptic units into a dynamic touchpad based on a haptic coding scheme and environmental information within the target area. The update module is configured to acquire user interaction data, dynamically adjust the complexity of the haptic coding scheme, and update the dynamic touchpad. The shopping guide module is configured to use spatial audio technology and an updated dynamic touchpad to guide users to the corresponding location of the target product to complete their shopping.

[0014] According to a third aspect of the present invention, a multimodal sensing blind shopping guide device is also provided, comprising at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit performs the steps of any of the above-described multimodal sensing blind shopping guide methods.

[0015] According to a fourth aspect of the invention, a storage medium is also provided, which stores a computer program executable by a multimodal sensing blind shopping guide device, wherein when the computer program is run on the multimodal sensing blind shopping guide device, the multimodal sensing blind shopping guide device performs the steps of any of the above-described multimodal sensing blind shopping guide methods.

[0016] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects: This invention provides a multimodal perception-based shopping guide method for the blind. It optimizes tactile encoding parameters based on the user's multimodal tactile perception characteristics to generate a personalized tactile encoding scheme. This fully considers each user's unique tactile perception ability, avoiding tactile cognitive overload caused by transmitting too much information in complex environments, allowing users to more easily receive and understand tactile information. Based on the optimized tactile encoding scheme and environmental information within the target area, an array of electroactive polymer tactile units is mapped to a dynamic touchpad, presenting environmental information in a richer and more diverse way. This breaks through the limitations of traditional parameters, helping to more accurately and comprehensively express complex environmental spatial structures, thereby improving shopping guide efficiency. By acquiring user interaction data, the complexity of the tactile encoding scheme is dynamically adjusted, and the dynamic touchpad is updated. This allows the interaction strategy to be adjusted as the user's ability improves, providing tactile information complexity that matches their ability. This enables users to gradually adapt to more complex tactile information through continuous use, improving adaptability and practicality. Spatial audio technology can provide users with direction and location information from an auditory perspective. It complements and works synergistically with the tactile information provided by the dynamic touchpad to form a multimodal perceptual experience. This provides users with more comprehensive and accurate environmental information, helping them better understand and navigate the shopping environment, thereby significantly improving the shopping guide effect. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating an embodiment of the multimodal perception-based shopping guide method for the blind provided by the present invention; Figure 2 A flowchart illustrating an embodiment of the personalized coding scheme provided by the present invention; Figure 3 Provided by the present invention Figure 1 A schematic flowchart of an embodiment of step S103; Figure 4 A schematic diagram of the structure of an embodiment of the multimodal sensing blind shopping guide device provided by the present invention; Figure 5 This is a schematic diagram of the structure of a multimodal sensing shopping guide device for the blind provided in an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0020] The terms "first," "second," "third," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or apparatus.

[0021] This invention provides a multimodal sensing method, apparatus, device, and storage medium for guiding blind shoppers, which will be described below.

[0022] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the multimodal perception-based shopping guidance method for the blind provided by the present invention. In a specific embodiment of the present invention, a multimodal perception-based shopping guidance method for the blind is disclosed, comprising: S101. Optimize the tactile coding parameters based on the user's multimodal tactile perception characteristics to generate a personalized tactile coding scheme for the user. S102. Based on the tactile coding scheme and environmental information within the target area, the electroactive polymer tactile unit array is mapped into a dynamic touchpad; S103. Obtain user interaction data, dynamically adjust the complexity of the haptic coding scheme, and update the dynamic touchpad. S104. Using spatial audio technology and the updated dynamic touchpad, guide the user to the location of the target goods to complete the shopping.

[0023] In the above embodiments, different blind individuals may have varying sensitivities and comprehensions regarding the intensity, frequency, duration, and patterns of tactile stimuli. Therefore, to more conveniently assist users in shopping guidance, it is necessary to first comprehensively collect multimodal tactile perception characteristic data of users. This includes using professional tactile perception testing equipment to allow users to experience tactile stimuli of different intensities, frequencies, and patterns, and recording user feedback, such as whether they can accurately perceive the stimuli, the comfort level of the perception, and the ease of understanding. Then, based on this detailed data, the tactile coding parameters are optimized, ultimately generating a tactile coding scheme that fully meets the personalized needs of users.

[0024] The environmental information within the target area contains rich details, such as the layout of the shelves, the types and locations of goods, and the direction of aisles. Electroactive polymer tactile unit arrays are materials with unique properties; they can deform in response to electrical signals, thus simulating different tactile sensations. By matching and mapping the parameters in the tactile coding scheme with the environmental information within the target area, the electroactive polymer tactile unit array can be transformed into a dynamic touchpad. When the location of a shelf needs to be indicated, the tactile units in the corresponding area of ​​the touchpad will produce specific protrusions or texture changes, allowing the user to perceive the outline and location of the shelf through touch.

[0025] Different types of goods can be distinguished by changing the frequency or intensity of the tactile pattern. For example, the food section might use a softer, more regular tactile pattern, while the daily necessities section might use a more rhythmic or complex pattern. Moreover, the dynamic touchpad is not static; it updates in real time as the shopping environment changes, ensuring that users always have accurate environmental information.

[0026] During shopping navigation using the dynamic touchpad, user interaction data is continuously collected, including the force, speed, dwell time, touch path, and user feedback when touching the touchpad. In-depth analysis of this data helps understand user behavior and adaptation to the current haptic coding scheme. If users experience difficulty understanding complex haptic patterns, or if they can easily handle the current level of haptic information complexity, the complexity of the haptic coding scheme is dynamically adjusted accordingly. After adjustment, the dynamic touchpad is immediately updated to match the new scheme, ensuring that the haptic coding scheme remains synchronized with user capabilities and needs, providing a more personalized and efficient shopping guidance service.

[0027] Spatial audio technology is a technique that simulates the propagation of sound in three-dimensional space, allowing users to perceive the direction, distance, and location of sounds through hearing. In guiding visually impaired shoppers, spatial audio technology is combined with an updated dynamic touchpad. When a user needs to find a target item, the spatial audio technology provides sound cues with specific direction and distance based on the item's location. For example, if the target item is to the user's right front, a sound from that direction is emitted, and the sound intensity gradually increases as the distance between the user and the item decreases. Simultaneously, the dynamic touchpad provides real-time tactile navigation information, such as indicating which direction to move in and when to turn.

[0028] Users can more accurately and quickly determine their location and the location of target goods by simultaneously experiencing tactile information on the touchpad and listening to spatial audio cues. During navigation, if users encounter obstacles or go in the wrong direction, they can be corrected in time through tactile and audio feedback to ensure they can successfully reach the location of the target goods and complete their shopping task. This multimodal perception-based shopping guide method for the blind provides a new, efficient, and user-friendly shopping experience for the blind through personalized tactile coding, dynamic touchpad mapping, interactive data-driven scheme adjustments, and collaborative navigation of spatial audio and tactile feedback, greatly improving their autonomy and convenience during the shopping process.

[0029] Compared with existing technologies, this embodiment provides a multimodal perception-based shopping guide method for the blind. It optimizes tactile encoding parameters based on the user's multimodal tactile perception characteristics to generate a personalized tactile encoding scheme. This fully considers each user's unique tactile perception capabilities, avoiding tactile cognitive overload caused by transmitting too much information in complex environments, allowing users to more easily receive and understand tactile information. Based on the optimized tactile encoding scheme and environmental information within the target area, an array of electroactive polymer tactile units is mapped to a dynamic touchpad, presenting environmental information in a richer and more diverse way. This breaks through the limitations of traditional parameters, helping to more accurately and comprehensively express complex environmental spatial structures, thereby improving shopping guide efficiency. By acquiring user interaction data, the complexity of the tactile encoding scheme is dynamically adjusted, and the dynamic touchpad is updated. This allows the interaction strategy to be adjusted as the user's abilities improve, providing tactile information complexity that matches their capabilities. This enables users to gradually adapt to more complex tactile information through continuous use, improving adaptability and practicality. Spatial audio technology can provide users with direction and location information from an auditory perspective. It complements and works synergistically with the tactile information provided by the dynamic touchpad to form a multimodal perceptual experience. This provides users with more comprehensive and accurate environmental information, helping them better understand and navigate the shopping environment, thereby significantly improving the shopping guide effect.

[0030] In some embodiments of the present invention, the tactile coding parameters are optimized based on the user's multimodal tactile perception characteristics to generate a personalized tactile coding scheme for the user, and the method further includes: Based on the characteristics of multimodal tactile perception and the preset optimization algorithm, the tactile coding parameters are optimized to obtain the target tactile coding parameter set; The initial tactile coding is adjusted based on the target tactile coding parameter set to generate a user-personalized tactile coding scheme.

[0031] In the above embodiments, an adaptive psychophysical measurement method is first used to measure the user's tactile perception characteristics in the following four dimensions: Spatial resolution measurement: The Two-Point Discrimination Test was used. Two points were randomly selected on the DEA array of the shopping cart handle and activated simultaneously. The distance between the two points was gradually decreased to determine the minimum distance threshold that the user could distinguish as two independent touch points. (That is, the smallest spatial distance that the user can distinguish).

[0032] The measurement range covers different areas of the user's palm and fingers, generating a spatial resolution distribution map: ; in, Represents the set of spatial resolution distributions of user u. Represents the position coordinates in the DEA array The spatial resolution threshold measured at that location, This represents the position coordinates on the DEA array.

[0033] Intensity sensitivity measurement: An adaptive up-and-down adjustment method is used to determine the minimum noticeable difference (JND) for users under different reference intensities.

[0034] The intensity JND function was obtained by measurement: ; in, Indicates user The intensity JND function describes the user's sensitivity to changes in tactile stimulation at different baseline intensities; , , They represent the first , , A benchmark strength level; Indicates the total number of baseline strength levels tested; This represents the minimum intensity change that a user can perceive.

[0035] Intensity sensitivity data directly affects the selection of intensity parameters when designing tactile feedback, ensuring that changes in tactile signal intensity exceed the user's perception threshold so that information can be effectively perceived.

[0036] Frequency discrimination capability measurement: A frequency judgment task is used to measure the user's ability to distinguish vibration signals of different frequencies. The frequency discrimination threshold function is obtained as follows: ; in, , , They represent the first , , One reference frequency; Indicates the number of reference frequency points tested; Indicates the smallest distinguishable difference; Indicates user The frequency discrimination capability characteristic function describes the frequency resolution accuracy of the user in different frequency ranges.

[0037] Temporal resolution measurement: measures the minimum time interval in which a user can distinguish between two adjacent tactile stimuli in time, as well as the response time to perceive changes in tactile patterns.

[0038] Furthermore, the time-resolved characteristics were obtained: ; in, Indicates user The time-resolved characteristics describe the user's ability to perceive time-related tactile stimuli; The minimum distinguishable time interval is the shortest time difference between which a user can perceive two independent stimuli rather than a single consecutive stimulus. The average response time is the average time required for a user to respond correctly from receiving a tactile stimulus.

[0039] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the personalized coding scheme provided by the present invention. In some embodiments of the present invention, the target tactile coding parameter set includes spatial resolution characteristics, intensity sensitivity curves, frequency discrimination characteristics, and temporal resolution characteristics; adjusting the initial tactile coding based on the target tactile coding parameter set to generate a user-personalized tactile coding scheme further includes: S201. An encoding scheme that adjusts the density and width of the encoding path lines according to the spatial resolution characteristics; S202. Construct a nonlinear intensity mapping function based on the intensity sensitivity curve to optimize the coding scheme for obstacle distance and hazard level; S203. Based on the frequency discrimination characteristics, determine the coding scheme that corresponds the frequency range to different information types; S204. An encoding scheme for adjusting the presentation speed and temporal pattern of tactile sequences based on time-resolved characteristics.

[0040] In the above embodiments, spatial resolution characteristics determine the positioning accuracy of tactile stimuli on the skin surface, the intensity sensitivity curve reflects the user's perception threshold and comfort range for pressure amplitude, frequency discrimination characteristics characterize the user's ability to distinguish different vibration frequencies, and temporal discrimination characteristics describe the user's ability to perceive the temporal sequence of tactile pulses. Based on the dynamic adjustment mechanism of this parameter set, a tactile coding scheme adapted to individual physiological characteristics can be generated. Specific implementation paths include: For different sensory areas (such as the highly sensitive area of ​​the fingertips and the low-sensitivity area of ​​the palm), an adaptive variable density coding strategy is adopted: 0.2-0.5mm ultra-fine lines are used with high-density path planning in sensitive areas, while 1-2mm wide lines are used with sparse paths in non-sensitive areas. At the same time, a dynamic line width adjustment mechanism is introduced to automatically adjust the visual-tactile dual-modal matching degree of the lines according to the ambient light intensity to ensure multi-sensory consistency.

[0041] The distances to obstacles in physical space (0.1-10m) and their danger levels (1-5) are mapped to a set of tactile stimulus intensity parameters, where: Logarithmic coordinate compression mapping is used in the 0.1-1m close-range zone to enhance near-end hazard perception; Linear mapping is implemented in the 1-5m medium distance range to maintain the accuracy of distance judgment; In the 5-10m distance range, exponential decay mapping is used to prevent information overload.

[0042] The hazard level is superimposed by pulse width modulation (PWM), and the high hazard signal is superimposed with a high-frequency vibration component of 100-300Hz to enhance alertness.

[0043] The user frequency discrimination threshold (DF) was determined through a two-pulse interval discrimination experiment (2IFC paradigm), and a three-level frequency coding channel was established. Low-frequency channel (5-50Hz): Maps basic navigation information (direction guidance, path prompts); Intermediate frequency channel (50-200Hz): Encodes environmental status information (temperature changes, material identification); High-frequency channel (200-500Hz): Transmits emergency alarm information (collision warning, fault).

[0044] An orthogonal frequency design principle is adopted to ensure that the minimum frequency interval between channels is ≥1.5×DF, and a harmonic suppression algorithm is used to eliminate spectral aliasing. For complex information scenarios, a hybrid coding strategy of time division multiplexing (TDM) and frequency division multiplexing (FDM) is implemented.

[0045] Based on the determination of user time-resolved threshold (TRT) through event-related potential (ERP) experiments, a dynamic temporal coding framework is constructed: Simple instructions use single-pulse encoding, with pulse width = TRT × 0.8; The sequence instructions implement variable interval encoding, with the base interval = TRT × 1.2 and the critical node interval shortened to TRT × 0.6; The continuous stream information uses adaptive rate adjustment, with an initial rate of 1 / TRT, which is dynamically adjusted by ±30% based on the user's cognitive load.

[0046] Introducing rhythmic patterns enhances information recognizability. By generating tactile rhythmic templates with statistical regularity through Markov chain models, complex temporal patterns conform to the capacity of human short-term memory (7±2 information units).

[0047] The multimodal perception fusion algorithm achieves cross-modal alignment of touch, vision and hearing, and is equipped with a real-time biofeedback monitoring module. It can dynamically optimize the coding parameters based on the user's skin conductance response (GSR) and heart rate variability (HRV) indicators, and finally form a bio-adaptive closed-loop tactile interaction.

[0048] Please see Figure 3 , Figure 3 Provided by the present invention Figure 1 A flowchart illustrating an embodiment of step S103. In some embodiments of the present invention, acquiring user interaction data, dynamically adjusting the complexity of the haptic coding scheme, and updating the dynamic touchpad further includes: S301. Based on user interaction data, assess the user's current tactile cognitive load; S302. Dynamically adjust the complexity of the tactile coding scheme based on the user's current tactile cognitive load; S303. Update the dynamic touchpad according to the adjusted haptic coding scheme.

[0049] In the above embodiments, a multi-source data acquisition module is constructed to acquire user interaction data (generally finger exploration behavior data) in real time. Through a multi-point touch sensing layer on the tactile interface, the module records the user's finger exploration patterns, including but not limited to: Dwell time distribution Records the time a user's finger stays at different locations; Scan speed : Measures the speed at which a user's finger moves on a haptic interface; Pressure changes Records the amount of pressure applied by the user at various locations; User physiological response data: Data on the user's physiological state is collected through wearable sensors. Heart rate variability : Reflects the user's state of tension and stress; Skin conductance : Indicates the user's emotional activation level; Body temperature changes : To help determine the user's emotional state; Navigation behavior data: Collecting user behavior metrics during the navigation process. Navigation speed The speed at which users push their shopping carts changes; Path deviation The deviation between the user's actual path and the optimal path; Number of pauses and duration , Records user pauses during navigation.

[0050] By constructing a cognitive load index calculation model and dynamically adjusting the model's cognitive load threshold based on historical interaction data, the current interaction data is input into the adjusted cognitive load index calculation model to assess the user's current tactile cognitive load.

[0051] The density of tactile elements is dynamically adjusted based on the user's current tactile cognitive load. Sparse coding (element spacing ≥ 5mm) is used under high load, and high-density coding (element spacing ≤ 2mm) is enabled under low load, thus dynamically adjusting the complexity of the tactile coding scheme.

[0052] Intensity dimension: Implement non-linear intensity compression. When the user's current tactile cognitive load is >70%, the original intensity mapping range is dynamically compressed from [0,100%] to [0,60%] to prevent perceptual saturation.

[0053] Timing dimension: The user's operation rhythm is predicted by the particle filter algorithm. During high-speed operation (TPV>300mm / s), the tactile pulse interval is automatically extended (≥150ms) to ensure information recognizability.

[0054] The dynamic touchpad employs a composite driving structure of piezoelectric ceramic array and shape memory alloy, supporting real-time switching of local surface deformation and texture. An integrated field-effect transistor array enables zoned energy management, with high-priority tactile signals triggering high-frequency driving in local areas, while low-priority signals utilize a low-frequency energy-saving mode.

[0055] Based on the adjusted encoding parameters, multi-channel driving waveforms are generated through direct digital frequency synthesis technology, supporting the simultaneous output of 16 independent tactile signals. A dynamic bias compensation algorithm is introduced to correct the hysteresis effect of electrodeformable materials in real time, ensuring that the deformation accuracy reaches ±0.05mm.

[0056] The touchpad is divided into 16 independent control areas, and time-division multiplexing technology is used to achieve parallel driving of multiple areas, with area switching time of less than 2ms, supporting continuous presentation of complex paths. For high-frequency operation areas, higher timing resolution is dynamically allocated to prevent information loss.

[0057] In some embodiments of the present invention, assessing the user's current tactile cognitive load based on user interaction data further includes: A cognitive load index calculation model was constructed, and the dynamic threshold of cognitive load in the model was dynamically adjusted based on historical interaction data. The current interaction data is input into the adjusted cognitive load index calculation model to assess the user's current tactile cognitive load.

[0058] In the above embodiments, based on multimodal interaction data, this application constructs a cognitive load index calculation model: ; in, The cognitive load index is used to quantify the user's current level of cognitive processing stress. Indicates the exploration mode parameters; This is a haptic efficiency parameter, representing the efficiency with which a user recognizes and processes haptic information; Indicates physiological response parameters; This parameter represents the environmental complexity, characterizing the information complexity of the current environment. This represents the cognitive load calculation function.

[0059] ; in, Indicates the exploration mode parameters; , , These represent the weighting coefficients for dwell time, scanning speed, and pressure magnitude, respectively. Indicates the total number of sampling points; Indicates the user's finger position at coordinates The length of time spent at the place; Indicates the total duration of the observation; Indicates the user's time Finger scanning speed; This indicates the total number of pressure sampling points; Indicates the user's time ,Location The magnitude of the pressure applied at the location.

[0060] ; in, This is a haptic efficiency parameter, representing the overall efficiency of the user in recognizing and processing haptic information; Indicates the total number of tactile patterns; An index representing a specific tactile pattern, ranging from 1 to K; Indicates tactile pattern The accuracy of recognition; For tactile mode Average response time; , These represent tactile modes. The weighting coefficients for accuracy and response time; This is the reciprocal of the response time, representing the response speed; a larger value indicates a faster user response.

[0061] ; in, These are physiological response parameters, representing a comprehensive assessment of stress and emotional state calculated by the user based on physiological indicators. , , These represent the weighting coefficients of heart rate variability, skin conductance response, and body temperature change in the calculation of physiological response parameters, respectively. Indicates time Normalized heart rate variability index; For time Normalized skin conductance index; Indicates time Normalized body temperature change index.

[0062] ; in, This is an environmental complexity parameter, representing the comprehensive evaluation value of the information complexity of the current environment; , , These represent the weighting coefficients of the number of obstacles, path complexity, and decision point density in the calculation of environmental complexity, respectively. The number of obstacles; For path complexity; Decision point density.

[0063] Based on users' historical data, dynamically adjust the threshold level of cognitive load: Personalized baseline establishment: By recording users' performance at different load levels, personalized cognitive load baselines are established. ; in, For users The cognitive load baseline value represents the average cognitive load level of the user over a period of time; Indicates the total length of the measurement period; This represents the integral operation from time 0 to time T, used to accumulate cognitive load over the entire time period; For users At the point of time Real-time cognitive load level; Let be the time derivative, representing the time increment during the integration process.

[0064] Adaptive threshold calculation: Dynamically adjust the acceptable threshold of cognitive load based on the user's learning progress. ; in, For users In time The cognitive load threshold represents the maximum cognitive load level that the user is considered to be able to tolerate at that moment. For users The cognitive load baseline value represents the user's basic cognitive load tolerance. The adjustment range for cognitive load represents the maximum amount of cognitive load that can be increased relative to the baseline value. This is an adjustment function based on the learning state, with values ​​ranging from [0, 1]. Indicates the user's time The function value increases as the user's learning level improves, thereby appropriately raising the cognitive load threshold.

[0065] The user's current tactile cognitive load can be directly input into the adjusted cognitive load index calculation model by inputting the current interaction data. This invention will not elaborate further on this.

[0066] In some embodiments of the present invention, the complexity of the tactile coding scheme is dynamically adjusted based on the user's current tactile cognitive load, and the method further includes: When the user's current tactile cognitive load exceeds the preset threshold, the line width of the encoding path is increased and the stimulation interval is extended. When the user's current tactile cognitive load is less than the preset threshold, the curvature and slope of the encoded path are increased, and obstacle contour encoding is set.

[0067] In the above embodiments, when the multimodal sensor detects that the user's tactile cognitive load exceeds a threshold (e.g., CLI > 70%), a cognitive decompression coding strategy is immediately initiated. In the spatial dimension, a Gaussian blur algorithm is used to perform visual-tactile joint widening processing on the encoded path, dynamically expanding the original 0.3mm line width to 0.8-1.5mm. The expansion coefficient is linearly positively correlated with the CLI value (γ = 0.01 × CLI), ensuring that the user can still accurately perceive the path direction under high load conditions.

[0068] Meanwhile, the Voronoi diagram algorithm is used to simplify the complex path by skeletonization, retaining key topological features such as curvature abrupt changes and eliminating 30-50% of redundant branches, so that the path complexity matches the user's information processing capabilities. In terms of the temporal dimension, a dynamic interval model is established based on the user's reaction time distribution. The stimulus interval is nonlinearly extended by the formula T_new = T_base × (1 + 0.02×(CLI-75)) (T_base baseline interval 120ms), and the pulse width is compressed from 80ms to 50-60ms. Combined with a 20% amplitude increase to compensate for perception loss, an anti-interference coding mode of "wide interval - short pulse - high intensity" is formed.

[0069] When a user is determined to be in a low cognitive load state (CLI < 30%), the system automatically switches to a cognitive enhancement coding mode to improve information transmission efficiency. In the spatial dimension, a controllable curvature (curvature radius adjustable from 5-20mm) is added to the coding path using the Bézier curve algorithm, while a gradient slope design (gradual slope angle from 15° to 45°) is introduced to enable three-dimensional deformation perception on flat paths, increasing the information density per unit area by 2-3 times.

[0070] To address environmental perception needs, an obstacle contour encoding submodule is activated, mapping spatial distance information to boundary vibrations of different amplitudes (high amplitude for near distances / low amplitude for far distances), and parallel encoding of multiple obstacle contours is achieved through phase modulation technology. In terms of temporal dimension, variable-interval pulse sequences (interval times following a Weiber distribution) are used instead of equally spaced stimuli, leveraging the higher sensitivity of humans to random temporal sequences to improve information recognition rate.

[0071] In some embodiments of the present invention, guiding the user to the location corresponding to the target goods to complete the shopping, based on spatial audio technology and an updated dynamic touchpad, further includes: Based on spatial audio technology and the updated dynamic touchpad, a mapping relationship between tactile sensation and spatial audio is established; Based on the mapping relationship between tactile sensation and spatial audio, the shopping process can be completed by confirming the location of the target goods.

[0072] In the above embodiments, a cross-modal mapping framework for tactile and spatial audio is constructed using multi-dimensional perception fusion technology. In the spatial audio dimension, a 3D audio engine based on HRTF (Head-Related Transfer Function) is employed to encode the target cargo location as a sound field distribution with azimuth, elevation, and distance attenuation, achieving sub-degree-level positioning accuracy through binaural time difference and binaural sound level difference. In the tactile dimension, the dynamic touchpad, based on an updated tactile coding scheme, maps the same target location to a multi-parameter tactile signal containing amplitude gradient, texture density, and deformation contour. A joint probability distribution of tactile and audio parameters is established through a Bayesian inference model, enabling the touchpad to automatically generate a matching tactile guidance mode when the user receives spatial audio cues, forming a two-way verification mechanism of "auditory positioning - tactile confirmation."

[0073] A hierarchical guidance algorithm is used to provide end-to-end support from environmental awareness to coarse and fine localization. Environmental cognition layer: Ambient noise is collected in real time through a distributed microphone array on the supermarket ceiling. Beamforming technology is used to generate a noise field map and dynamically adjust the signal-to-noise ratio (SNR≥15dB) of spatial audio and the driving intensity of tactile signals to ensure that guidance information can still be clearly perceived in noisy environments.

[0074] Coarse positioning layer: When a user enters the target shelf area, the spatial audio pulse sequence encoding and the progressive deformation of the touchpad are activated to help the user quickly locate the approximate position through the synchronization of bimodal stimulation.

[0075] Precision positioning layer: Within 1m of the target, switch to high-precision encoding mode. Spatial audio uses frequency modulation technology to transmit distance information, while the touchpad generates a texture pattern that matches the outline of the goods through a micro-vibration array, achieving centimeter-level positioning accuracy.

[0076] To address perceptual interference in complex shopping scenarios, a multi-level fault-tolerance mechanism is integrated: Modal compensation layer: When spatial audio is blocked and fails, the touchpad automatically enhances the salience of the tactile signal by raising the vibration frequency to the sensitive area of ​​the human ear to generate perceptible air vibration, forming an alternative guidance path of "tactile-bone conduction"; conversely, when the touchpad is blocked, spatial audio simulates the directional cues of tactile feedback by dynamically adding directional noise.

[0077] User Adaptive Layer: Based on the k-means clustering algorithm, analyze users' historical interaction data to generate personalized mapping parameters for different user types (e.g., prioritize low-frequency tactile signals and high-contrast spatial audio for elderly users).

[0078] Environmental Adaptive Layer: Real-time monitoring of user movement speed via UWB positioning within the supermarket. When the speed exceeds 1.5m / s, the system automatically simplifies tactile coding and compresses the guiding rhythm of spatial audio (pulse frequency increased to 2Hz) to prevent cognitive collapse caused by information overload.

[0079] Ultimately, through the deep integration of tactile and spatial audio, intelligent guidance was achieved throughout the entire process, from environmental perception and path planning to task verification, providing an innovative solution for barrier-free shopping and smart retail.

[0080] To better implement the multimodal perception-based blind shopping guide method in the embodiments of the present invention, based on the multimodal perception-based blind shopping guide method, please refer to the corresponding... Figure 4 , Figure 4 This is a schematic diagram of a structure of an embodiment of the multimodal sensing blind shopping guide device provided by the present invention. The embodiment of the present invention provides a multimodal sensing blind shopping guide device 400, comprising: The encoding module 410 is configured to optimize the tactile encoding parameters based on the user's multimodal tactile perception characteristics to generate a personalized tactile encoding scheme for the user. The mapping module 420 is configured to map the electroactive polymer haptic unit array into a dynamic touchpad based on the haptic coding scheme and environmental information within the target area. The update module 430 is configured to acquire user interaction data, dynamically adjust the complexity of the haptic coding scheme, and update the dynamic touchpad. The shopping guide module 440 is configured to guide users to the corresponding location of the target goods to complete the shopping process based on spatial audio technology and an updated dynamic touchpad.

[0081] It should be noted that the device 400 provided in the above embodiments can implement the technical solutions described in the above method embodiments. The specific implementation principles of the above modules or units can be found in the corresponding content in the above method embodiments, and will not be repeated here.

[0082] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of a multimodal sensing shopping guide device for the blind provided in an embodiment of the present invention. Based on the above-described multimodal sensing shopping guide method for the blind, the present invention also provides a multimodal sensing shopping guide device for the blind, which can be a computing device such as a mobile terminal, desktop computer, laptop, handheld computer, or server. The multimodal sensing shopping guide device 500 for the blind includes a processor 510, a memory 520, and a display 530. Figure 5 Only some components of the multimodal sensing shopping guide device for the blind are shown. However, it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead.

[0083] In some embodiments, the memory 520 may be an internal storage unit of the multimodal sensing blind shopping guide device 500, such as a hard disk or memory of the multimodal sensing blind shopping guide device 500. In other embodiments, the memory 520 may be an external storage device of the multimodal sensing blind shopping guide device 500, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the multimodal sensing blind shopping guide device 500. Furthermore, the memory 520 may include both internal and external storage units of the multimodal sensing blind shopping guide device 500. The memory 520 is used to store application software and various types of data installed on the multimodal sensing blind shopping guide device 500, such as the program code of the multimodal sensing blind shopping guide device 500. The memory 520 may also be used to temporarily store data that has been output or will be output. In one embodiment, a multimodal sensing blind shopping guide program 540 is stored in the memory 520. The multimodal sensing blind shopping guide program 540 can be executed by the processor 510 to implement the multimodal sensing blind shopping guide method of the various embodiments of this application.

[0084] In some embodiments, processor 510 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in memory 520 or process data, such as executing a multimodal perception-based shopping guide method for the blind.

[0085] In some embodiments, display 530 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 530 is used to display information from the multimodal sensing visual aid device 500 and to display a visual user interface. Components 510-530 of the multimodal sensing visual aid device 500 communicate with each other via a bus.

[0086] In one embodiment, when the processor 510 executes the multimodal sensing blind shopping guide program 540 in the memory 520, the steps in the multimodal sensing blind shopping guide method described above are implemented.

[0087] This embodiment also provides a computer-readable storage medium storing a multimodal sensing shopping guide program for the blind, which, when executed by a processor, performs the following steps: The tactile coding parameters are optimized based on the user's multimodal tactile perception characteristics to generate a personalized tactile coding scheme for the user. Based on the tactile coding scheme and environmental information within the target area, the electroactive polymer tactile unit array is mapped into a dynamic touchpad; Acquire user interaction data, dynamically adjust the complexity of the haptic coding scheme, and update the dynamic touchpad; Using spatial audio technology and an updated dynamic touchpad, users are guided to the location of their desired goods to complete their shopping.

[0088] In summary, this invention provides a multimodal perception-based shopping guide method for the blind. It optimizes tactile encoding parameters based on the user's multimodal tactile perception characteristics, generating a personalized tactile encoding scheme. This fully considers each user's unique tactile perception capabilities, avoiding tactile cognitive overload caused by transmitting too much information in complex environments, allowing users to more easily receive and understand tactile information. Based on the optimized tactile encoding scheme and environmental information within the target area, an array of electroactive polymer tactile units is mapped to a dynamic touchpad, presenting environmental information in a richer and more diverse way. This breaks through the limitations of traditional parameters, helping to more accurately and comprehensively express complex environmental spatial structures, thereby improving shopping guide efficiency. By acquiring user interaction data, the complexity of the tactile encoding scheme is dynamically adjusted, and the dynamic touchpad is updated. This allows the interaction strategy to be adjusted as the user's abilities improve, providing tactile information complexity that matches their capabilities. This enables users to gradually adapt to more complex tactile information through continuous use, improving adaptability and practicality. Spatial audio technology can provide users with direction and location information from an auditory perspective. It complements and works synergistically with the tactile information provided by the dynamic touchpad to form a multimodal perceptual experience. This provides users with more comprehensive and accurate environmental information, helping them better understand and navigate the shopping environment, thereby significantly improving the shopping guide effect.

[0089] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method. The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, as well as magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic cards or optical cards, nanometers (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0090] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0091] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0092] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some service interfaces; indirect couplings or communication connections between apparatuses or units may be electrical or other forms.

[0093] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0094] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0095] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0096] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0097] The foregoing description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Those skilled in the art will readily conceive of embodiments of this disclosure upon considering the specification and practicing the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described herein. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

[0098] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0099] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal perception-based shopping guide method for the blind, characterized in that, The method comprises the following steps: According to the multi-modal tactile perception characteristics of the user, the tactile encoding parameters are optimized to generate a user-specific tactile encoding scheme; Based on the tactile encoding scheme and the environmental information in the target area, an array of electroactive polymer tactile units is mapped into a dynamic touchpad; Obtain user interaction data, dynamically adjust the complexity of the tactile encoding scheme, and update the dynamic touchpad; According to the spatial audio technology and the updated dynamic touchpad, the user is guided to the corresponding position of the target goods to complete the shopping.

2. The multimodal perception based shopping guide method for the blind person as claimed in claim 1, wherein, According to the multi-modal tactile perception characteristics of the user, the tactile encoding parameters are optimized to generate a user-specific tactile encoding scheme, which further comprises: According to the multi-modal tactile perception characteristics and the preset optimization algorithm, the target tactile encoding parameter set is obtained by optimizing the tactile encoding parameters; Based on the target tactile encoding parameter set, the initial tactile encoding is adjusted to generate a user-specific tactile encoding scheme.

3. The multimodal perception based shopping guide method for blind people as claimed in claim 2, wherein, The target tactile encoding parameter set includes spatial resolution characteristics, intensity sensitivity curve, frequency discrimination characteristics and time resolution characteristics; based on the target tactile encoding parameter set, the initial tactile encoding is adjusted to generate a user-specific tactile encoding scheme, which further comprises: According to the spatial resolution characteristics, the density and width of the encoding path line are adjusted to generate an encoding scheme; Based on the intensity sensitivity curve, a nonlinear intensity mapping function is constructed to optimize the encoding scheme of obstacle distance and danger level; According to the frequency discrimination characteristics, the encoding scheme of the correspondence between the frequency interval and different information types is determined; Based on the time resolution characteristics, the presentation speed and timing mode of the tactile sequence are adjusted to generate an encoding scheme.

4. The multimodal perception based shopping guide method for blind people as claimed in claim 1 wherein, The method further comprises the following steps: Based on the user interaction data, the current tactile cognitive load of the user is evaluated; The complexity of the tactile encoding scheme is dynamically adjusted according to the current tactile cognitive load of the user; The dynamic touchpad is updated according to the adjusted tactile encoding scheme.

5. The multimodal perception based shopping guide method for blind people as claimed in claim 4, wherein, The method further comprises the following steps: A cognitive load index calculation model is constructed, and the cognitive load dynamic threshold of the model is dynamically adjusted according to historical interaction data; The current interaction data is input into the adjusted cognitive load index calculation model to evaluate the current tactile cognitive load of the user.

6. The multimodal perception based shopping guide method for blind people as claimed in claim 4, wherein, The method further comprises the following steps: When the current tactile cognitive load of the user is greater than a preset compliance value, the line width of the encoding path is increased, and the stimulation time interval is prolonged; When the current tactile cognitive load of the user is less than a preset compliance value, the curvature and slope of the encoding path are increased, and the obstacle contour encoding is set.

7. The multimodal perception based shopping guide method for blind people as claimed in claim 1 wherein, The method further comprises the following steps: According to the spatial audio technology and the updated dynamic touchpad, the mapping relationship between tactile and spatial audio is established; Based on the mapping relationship between tactile and spatial audio, the corresponding position of the target goods is confirmed to complete the shopping.

8. A multi-modal perception based shopping guide device for the blind, characterized in that, The method comprises the following steps: a coding module configured to optimize haptic coding parameters according to the multimodal haptic perceptual characteristics of the user, and generate a user-individualized haptic coding scheme; a mapping module configured to map an array of electroactive polymer haptic units into a dynamic touchpad based on the haptic coding scheme and environmental information within a target area; an updating module configured to obtain user interaction data, dynamically adjust the complexity of the haptic coding scheme, and update the dynamic touchpad; a guiding module configured to guide the user to the location of the target item according to spatial audio technology and the updated dynamic touchpad, and complete the shopping.

9. A multi-modal perception based shopping guide device for the blind, characterized in that, A computer program product, comprising at least one storage medium having stored computer program instructions that, when executed at a processing unit, cause the processing unit to execute the steps of the multimodal perceptual blind guiding method according to any one of claims 1-7.

10. A storage medium, characterized by A computer program product, comprising at least one storage medium having stored computer program instructions that, when executed at a multimodal perceptual blind guiding device, cause the multimodal perceptual blind guiding device to execute the steps of the multimodal perceptual blind guiding method according to any one of claims 1-7.

Images