Map browsing auxiliary device based on multi-mode tactile feedback

Abstract

The embodiment of the utility model provides a map browsing auxiliary device based on multi-mode tactile feedback, and the device comprises an image obtaining unit which is used for obtaining an image; the temperature sensing unit is used for wearing and outputting a temperature sensing signal based on the temperature sensing adjusting signal; the control unit is connected with the image acquisition unit and connected with the temperature sensing unit through a flexible substrate, and the control unit is used for controlling the temperature sensing unit to sense the temperature when the image comprises a target area image of a target position in a map pointed by a finger and environment temperature information. And outputting the temperature sensing adjusting signal aiming at the environment temperature of the target position to the temperature sensing unit. According to the embodiment of the invention, the map browsing auxiliary device can improve the capability of expressing the environment temperature information displayed on the terminal user interface for displaying the electronic map to the visually impaired crowd when the visually impaired crowd reads the map.

Description

Technical Field

[0001] This application relates to the field of guide vision technology, specifically to a map browsing assistance device based on multimodal tactile feedback. Background Technology

[0002] In related technologies, some wearable devices can already help visually impaired people read maps. However, in order to increase the readability of maps, some electronic map manufacturers display ambient temperature information on the user interface of the terminal displaying the electronic map. Visually impaired people cannot understand this information through means other than voice. Utility Model Content

[0003] To address the shortcomings of existing technologies, this application proposes a map browsing assistance device based on multimodal haptic feedback, aiming to improve the ability of the map browsing assistance device to convey the ambient temperature information displayed on the terminal user interface of the electronic map to visually impaired people when they are reading maps.

[0004] To achieve the above objectives, this application provides a map browsing assistance device based on multimodal haptic feedback, comprising:

[0005] Image acquisition unit, used to acquire images;

[0006] A temperature sensing unit for wearable devices, and for outputting a temperature sensing signal based on a temperature sensing adjustment signal;

[0007] The control unit is connected to the image acquisition unit and to the temperature sensing unit via a flexible substrate. The control unit is used to output a temperature sensing adjustment signal for the ambient temperature of the target location to the temperature sensing unit when the image includes an image of the target area where a finger points to a target location on a map and ambient temperature information.

[0008] In one embodiment, the device further includes:

[0009] The map indicator unit includes multiple indicator electrodes, which are connected to the control unit via a flexible substrate. The map indicator unit is wearable and outputs a first electrotactile signal based on map element indication signals through one or more of the multiple indicator electrodes. The first electrotactile signal is used to indicate map elements at the target location.

[0010] The control unit is also configured to output the map element indication signal to the map indication unit when the image includes the target area image.

[0011] In one embodiment, the device further includes:

[0012] A voice interaction unit is connected to the control unit, and the voice interaction unit is used to acquire the user's interactive voice.

[0013] The map indicator unit is further configured to output a second electrotactile signal based on a target direction signal through one or more of the plurality of indicator electrodes, wherein the second electrotactile signal is used to indicate the direction in which the target location points to the location of the target map element;

[0014] The control unit is also configured to output the target direction signal to the map indication unit when the interactive voice includes a voice inquiry about the location of the target map element.

[0015] In one embodiment, the map indicator unit is further configured to output a third electrotactile signal based on a navigation direction signal through one or more of the plurality of indicator electrodes, wherein the third electrotactile signal is used to indicate the direction of the target location relative to a target route, the target route being a route to a location of interest;

[0016] The control unit is further configured to output the navigation direction signal to the map indication unit when the interactive voice includes voice content requesting a route to the location of interest.

[0017] In one embodiment, the map indicator unit is further configured to output a fourth electrotactile signal based on a path condition signal through one or more of the plurality of indicator electrodes, wherein the fourth electrotactile signal is used to indicate the path condition of the target route;

[0018] The control unit is further configured to output the path condition signal to the map indication unit when the interactive voice includes voice content requesting the path condition of the target route and the image includes path condition information.

[0019] In one embodiment, the map elements include one of the following: road elements; building elements; green space elements; water body elements; and points of interest elements.

[0020] In one embodiment, the plurality of indicator electrodes includes at least one of the following: one or more fingertip indicator electrodes; one or more finger pad indicator electrodes; one or more finger back indicator electrodes; one or more fingertip edge indicator electrodes.

[0021] In one embodiment, the temperature sensing signal includes at least one of a first temperature sensing signal and a second temperature sensing signal, wherein the temperature corresponding to the first temperature sensing signal is higher than the temperature corresponding to the second temperature sensing signal;

[0022] The temperature sensing unit includes:

[0023] A heating module is connected to the control unit via a flexible substrate, and the heating module is used to output a first temperature sensing signal based on the temperature sensing adjustment signal;

[0024] The heat absorption module is connected to the control unit via a flexible substrate, and the heat absorption module is used to output the second temperature sensing signal based on the temperature sensing adjustment signal.

[0025] In one embodiment, the voice interaction unit is further configured to play a response voice based on the response signal;

[0026] The control unit is further configured to generate and output the response signal for the map element to the map indication unit when the image includes the target area image.

[0027] In one embodiment, the voice interaction unit includes:

[0028] A voice acquisition module is connected to the control unit, and the voice acquisition module is used to acquire the interactive voice.

[0029] A voice playback module is connected to the control unit, and the voice playback module is used to play the reply voice.

[0030] This application provides a map browsing assistance device based on multimodal haptic feedback, comprising: an image acquisition unit for acquiring images; a temperature sensing unit for wearing and outputting a temperature sensing signal based on a temperature sensing adjustment signal; and a control unit connected to the image acquisition unit and connected to the temperature sensing unit via a flexible substrate. The control unit, when the image includes an image of a target area on the map pointed to by a finger and ambient temperature information, outputs the temperature sensing adjustment signal to the temperature sensing unit for the ambient temperature of the target location. By setting the temperature sensing unit and the control unit outputting the temperature sensing adjustment signal to the temperature sensing unit, the map browsing assistance device can interact with the user through the temperature sensing signal output by the temperature sensing unit, allowing the user to immerse themselves in understanding the ambient temperature of the target location through tactile sensation. This effectively conveys the ambient temperature information displayed on the terminal user interface of the electronic map to the user, thereby improving the map browsing assistance device's ability to express this information. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the structure of a map browsing auxiliary device provided in one embodiment of this application;

[0032] Figure 2 yes Figure 1 A schematic diagram illustrating the application of map browsing assistance devices in the text;

[0033] Figure 3 yes Figure 1 Schematic diagram of the control unit;

[0034] Figure 4 This is a schematic diagram of the structure of a map browsing auxiliary device provided in another embodiment of this application;

[0035] Figure 5 yes Figure 4 A schematic diagram illustrating the application of map browsing assistance devices in the text;

[0036] Figure 6 yes Figure 4 Schematic diagram of the control unit;

[0037] Figure 7 This is a schematic diagram of the structure of a map browsing auxiliary device provided in another embodiment of this application;

[0038] Figure 8A and Figure 8B This is a schematic diagram of the application of the indicator electrode provided in this application;

[0039] Figures 9A to 9E This is a schematic diagram illustrating the application of the indicator electrode provided in this application when it indicates map elements;

[0040] Figures 10A to 10B This is a schematic diagram illustrating the application of the indicator electrode provided in this application when indicating path conditions.

[0041] Figures 11A to 11C This is a schematic diagram illustrating the application of the indicator electrode provided in this application when indicating weather conditions.

[0042] Figure label:

[0043] Image acquisition unit 100, temperature sensing unit 200, heating module 210, heat absorption module 220, control unit 300, map indicator unit 400, indicator electrode 410, fingertip indicator electrode 411, finger pad indicator electrode 412, finger back indicator electrode 413, fingertip edge indicator electrode 414, voice interaction unit 500, voice acquisition module 510, and voice playback module 520. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0045] It should be noted that, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0046] In related technologies, some wearable devices can already help visually impaired people read maps. However, in order to increase the readability of maps, some electronic map manufacturers display ambient temperature information on the user interface of the terminal displaying the electronic map. Visually impaired people cannot understand this information through means other than voice.

[0047] To improve the ability of map browsing assistance devices to convey ambient temperature information displayed on the terminal user interface of an electronic map to visually impaired individuals, this application provides a map browsing assistance device based on multimodal haptic feedback. The device includes: an image acquisition unit for acquiring images; a temperature sensing unit for wearing and outputting a temperature sensing signal based on a temperature adjustment signal; and a control unit connected to the image acquisition unit and connected to the temperature sensing unit via a flexible substrate. The control unit outputs a temperature adjustment signal to the temperature sensing unit for the ambient temperature of the target location when the image includes an image of the target area pointed to by a finger on the map and ambient temperature information. By setting up the temperature sensing unit and the control unit outputting the temperature adjustment signal to the temperature sensing unit, the map browsing assistance device can interact with the user through the temperature sensing signal output by the temperature sensing unit, allowing the user to immerse themselves in understanding the ambient temperature of the target location through tactile sensation. This effectively conveys the ambient temperature information displayed on the terminal user interface of the electronic map to the user, thereby improving the map browsing assistance device's ability to convey this information.

[0048] See Figure 1 and 2 In this embodiment, the map browsing assistance device based on multimodal haptic feedback may include an image acquisition unit 100, a temperature sensing unit 200, and a control unit 300. The image acquisition unit 100 is used to acquire images. The temperature sensing unit 200 is worn and outputs a temperature sensing signal based on a temperature adjustment signal. The control unit 300 is connected to the image acquisition unit 100 and to the temperature sensing unit 200 via a flexible printed circuit (FPC). The control unit 300 can output a temperature adjustment signal for the ambient temperature of the target location to the temperature sensing unit when the image includes an image of the target area where a finger is pointing on the map and ambient temperature information.

[0049] In one embodiment, the image acquisition unit 100 can be a head-mounted camera, a miniature camera, a glasses camera, etc., and is not specifically limited here. The image acquisition unit 100 can be connected to the control unit 300 via a wired connection or via a wireless connection (such as Bluetooth, WIFI, etc.), and is not specifically limited here.

[0050] In one embodiment, the temperature sensing unit 200 can be a patch device (with components capable of outputting temperature sensing signals) that adheres to human skin (such as the skin of fingers, arms, etc.), or it can be wearable clothing (such as finger cots, gloves, clothes, etc.) with added or embedded components capable of outputting temperature sensing signals, etc., and so on. The specific components capable of outputting temperature sensing signals can be Peltier effect (i.e., thermoelectric effect) patches, modules composed of electrochemical cooling elements and heating wires, etc., and so on. Furthermore, the temperature sensing unit 200 can include one or more components capable of outputting temperature sensing signals, and this is not specifically limited.

[0051] In one embodiment, the temperature sensing signal may include at least one of a first temperature sensing signal and a second temperature sensing signal, wherein the temperature corresponding to the first temperature sensing signal is higher than the temperature corresponding to the second temperature sensing signal. The temperature sensing unit 200 may include a heating module 210 and a heat-absorbing module 220. Both the heating module 210 and the heat-absorbing module 220 are connected to the control unit 300 via a flexible substrate. The heating module 210 can be used to output a first temperature sensing signal based on a temperature sensing adjustment signal, and the heat-absorbing module 220 can be used to output a second temperature sensing signal based on the temperature sensing adjustment signal. Both the heating module 210 and the heat-absorbing module 220 may be Peltier effect patches (a Peltier effect patch with the skin contact surface being the heat-exciting surface is the heating module 210, and a Peltier effect patch with the skin contact surface being the heat-absorbing surface is the heat-absorbing module 220); the heating module 210 may be a module composed of a heating wire, and the heat-absorbing module 220 may be a module composed of an electrochemical cooling element, etc., etc., without specific limitations here.

[0052] It should be noted that the heating module 210 and the heat absorption module 220 can work independently (i.e., the final output temperature sensing signal is either the first temperature sensing signal or the second temperature sensing signal), or they can work simultaneously by adjusting the temperature (i.e., one releases heat while the other absorbs the heat released by the former to a certain extent, so as to adjust to a temperature that cannot be formed when either of them works alone. For example, the lowest temperature that the heating module 210 can generate is X degrees, and a temperature lower than X degrees can be generated through the heat absorption module 220). The specifics are not limited here.

[0053] In one embodiment, the temperature sensing signal refers to a physical signal capable of exchanging heat with human skin. For example, when the temperature sensing adjustment signal is a signal to lower the temperature, the temperature sensing unit 200 begins to absorb heat under the influence of the temperature sensing adjustment signal, and the skin part in contact with the temperature sensing unit 200 can feel the temperature drop to a certain value; when the temperature sensing adjustment signal is a signal to raise the temperature, the temperature sensing unit 200 begins to release heat under the influence of the temperature sensing adjustment signal, and the skin part in contact with the temperature sensing unit 200 can feel the temperature rise to a certain value.

[0054] See Figure 3 In one embodiment, the control unit 300 refers to a substrate integrating control, output signal generation, and output signal amplification functions. The control unit 300 may include a control board and a first relay. The input and control terminals of the first relay are connected to the control board, and the output terminal of the first relay is connected to the temperature sensing unit 200 via a flexible substrate. The control board can generate a control signal (digital signal) and a temperature adjustment signal (analog signal) for adjusting the temperature sensing. The control signal output by the control board can activate the first relay, allowing the temperature adjustment signal to be output to the temperature sensing unit 200. This enables the temperature sensing unit 200 to receive precise temperature adjustment control from the control board and effectively improves the operational stability of the temperature sensing unit 200. The control board can be an Arduino board, a Raspberry Pi, an STM series development board, etc., and is not specifically limited here.

[0055] Specifically, the control unit 300 has a built-in trained neural network model, enabling it to perform specific information processing (in this embodiment, this includes, but is not limited to, identifying target locations, identifying map images, identifying weather information, identifying map elements, speech recognition, speech generation, and route planning). At the beginning of using the map browsing assistance device, the control unit 300 can acquire images by controlling the image acquisition unit 100 to perform image capture. The image acquisition unit 100 can then return the acquired images to the control unit 300. After receiving the images acquired by the image acquisition unit 100, the control unit 300 can first identify whether the acquired image includes a map region image. If a map region image is identified, finger region image recognition is performed on the map region image. If a finger region image is identified, a map coordinate system is established based on the map region image. Then, the position of the fingertip relative to the map coordinate system is determined based on the finger region image, thereby determining the target location the finger is pointing to on the map. To improve readability, some existing electronic maps display ambient temperature information on the user interface using icons, pop-ups, and information boxes. To this end, the control unit 300 can perform image recognition on the acquired images to identify areas with ambient temperature information. After identifying these areas, the control unit extracts the ambient temperature information to determine the ambient temperature at the target location. Then, the control unit 300 generates a temperature-sensing adjustment signal based on the ambient temperature at the target location and outputs it to the temperature-sensing unit 200. The temperature-sensing unit 200 can absorb or release heat based on the temperature-sensing adjustment signal to output a temperature-sensing signal to the skin in contact with it. Upon receiving the temperature-sensing signal, the skin can fully perceive the ambient temperature at the location indicated on the electronic map.

[0056] In one embodiment, the neural network model built into the control unit 300 can be a TravelPreview Assistant (TPA) trained based on a Large Language Model (LLM) with reasoning capabilities, capable of accurate image recognition. The TPA can be trained using LLMs such as Deepseek and ChatGLM.

[0057] See Figure 4 and Figure 5In one embodiment, the map browsing assistance device may further include a map indicator unit 400, which may include a plurality of indicator electrodes 410. The plurality of indicator electrodes 410 are connected to the control unit 300 via a flexible substrate (FPC). The map indicator unit 400 is wearable and outputs a first electrotactile signal based on a map element indicator signal through one or more of the plurality of indicator electrodes 410. The first electrotactile signal can be used to indicate map elements at target locations. The control unit 300 may also be used to output the map element indicator signal to the map indicator unit 400 when the image includes an image of the target area.

[0058] In one embodiment, the map indicator unit 400 may be a patch device (patch electrode) that is attached to human skin (such as the skin of fingers, arms, etc.), or it may be clothing with patch electrodes added or embedded (such as finger cots, gloves, clothes, etc.), etc., and the specific details are not limited here.

[0059] In one embodiment, the plurality of indicator electrodes 410 may include a plurality of indicator electrodes 410 for a specific purpose. For example, assuming that the plurality of indicator electrodes 410 need to be attached to human skin, the indicator electrodes 410 attached to different parts of the body have different specific purposes. For example, the indicator electrode 410 attached to the hand is used to indicate direction, and the electrotactile signals indicating direction (such as the first, second, and third electrotactile signals mentioned below) are all indicated by the indicator electrode 410 attached to the hand. The indicator electrode 410 attached to the forearm is used to indicate signals of specific information (such as the fourth electrotactile signal representing the path status mentioned below).

[0060] In one embodiment, the plurality of indicator electrodes 410 may include a plurality of indicator electrodes 410 without a specified purpose, which collectively express information through electrotactile signals and their distribution locations. For example, assuming that the plurality of indicator electrodes 410 need to be attached to a finger, when specific information needs to be output, electrotactile signals can be output through one or more indicator electrodes 410 at specific locations. For example, each finger of a hand may have indicator electrodes 410 attached. When direction needs to be output, electrotactile signals can be output through the indicator electrodes 410 on the thumb and index finger to indicate a specific direction, and electrotactile signals can be output through the indicator electrodes 410 on the thumb, index finger, and ring finger to indicate another specific direction, and so on.

[0061] In one embodiment, the first electrotactile signal refers to an electrical signal that comes into contact with the human body. Specifically, the first electrotactile signal can be categorized into vibrations of different intensities based on map element indicator signals.

[0062] See Figure 6In one embodiment, the control unit 300 may further include multiple second relays, a channel signal generator, and an amplifier. A control board is connected to the channel signal generator, and the amplifier is also connected to the channel signal generator. The input terminal of each second relay is connected to the amplifier, and the control terminal of each second relay is connected to the control board. The output terminal of each second relay is connected to multiple indicator electrodes 410 in the map indicator unit 400 via a flexible substrate. Thus, when the control unit 300 generates a control signal (a digital signal) and a map element information signal (an analog signal) carrying map element information of the target location, the channel signal generator can modulate a specific frequency and voltage modulation signal based on this map element information signal. The modulation signal is amplified by the amplifier to generate a specific map element indicator signal. The control signal can turn on one or more second relays, thereby outputting the specific map element indicator signal generated by the amplifier to the map indicator unit 400 via the second relays. The channel signal generator can generate a modulation signal based on map elements, allowing users to more clearly distinguish the specific map elements of the target location using map element indicator signals of different frequencies and voltages.

[0063] Specifically, after the control unit 300 identifies the map-containing area in the image acquired by the image acquisition unit 100 using its built-in neural network model, it can further perform map element recognition on the map-containing area to determine the map elements displayed on the user interface. Once the target location on the map is determined, the specific map element at that location is identified based on the target location's position in the map coordinate system. Then, a corresponding map element indication signal is generated based on the specific map element and output to the map indication unit 400. This allows one or more indicator electrodes 410 to output a first electrotactile signal, informing the user of the map element at the target location. Since each indicator electrode 410 contacts a different part of the skin, outputting a first electrotactile signal from one or more indicator electrodes at different locations can represent different map elements. Therefore, the output of the first electrotactile signal by one or more indicator electrodes 410 assists the user in reading the map. Combined with the output of a temperature sensing signal from the temperature sensing unit 200, this allows the user to immerse themselves in map reading.

[0064] In one embodiment, map elements may include one of the following: road elements; building elements; green space elements; water body elements; and points of interest (POIs). A POI refers to an element on the map that the user is interested in. A POI can be a subordinate map element selected from road elements, building elements, green space elements, and water body elements. For example, a POI could be an entrance / exit element within building elements, a grassland element within green space elements, a reservoir element within water body elements, etc. Furthermore, POIs can be determined by the user through pre-settings on a map browsing aid device, etc., and are not specifically limited here.

[0065] See Figure 7 In one embodiment, the map browsing assistance device may further include a voice interaction unit 500, which is connected to the control unit 300. The voice interaction unit 500 can be used to acquire the user's interactive voice. The map indication unit 400 can also be used to output a second electrotactile signal based on a target direction signal through one or more of a plurality of indication electrodes 410, wherein the second electrotactile signal can be used to indicate the direction from the target location to the target map element location. The control unit 300 can also be used to output a target direction signal to the map indication unit when the interactive voice includes a voice inquiry about the target map element location.

[0066] In one embodiment, the voice interaction unit 500 can be a separate microphone, an earphone with an audio input device, etc., and is not specifically limited here. Furthermore, the connection between the voice interaction unit 500 and the control unit 300 can be a Bluetooth wireless connection or a wired connection, and is not specifically limited here.

[0067] In one embodiment, the second electrotactile signal can be a short-duration stimulation signal, or a continuous stimulation signal that guides the user to continuously point from the target location to the target map element location until the user accurately points to the target map element location and then stops, etc., and is not specifically limited here.

[0068] For example, when the second electrotactile signal is a short-lived stimulation signal, the user can understand the orientation of the target map element relative to the target location through the second electrotactile signal output by the specific indicator electrode 410, thus assisting the user in finding the target map element location.

[0069] For example, when the second electrotactile signal is a continuous stimulation signal, the user can understand the orientation of the target map element relative to the target location through the second electrotactile signal output by the specific indicator electrode 410, and move their finger to continuously point at the target map element location. In this process, the control unit 300 can continuously acquire images through the image acquisition unit 100 to output the latest target direction signal, so that the second electrotactile signal is continuously updated until the finger points to the target map element location.

[0070] In one embodiment, the target direction signal refers to a signal carrying directional information indicating the direction from the target location to the target map element location. The directional information from the target location to the target map element location in the target direction signal is output to the user by a second electrotactile signal. The direction from the target location to the target map element location includes both horizontal and vertical directions.

[0071] For example, suppose that the target location pointed to by the user's fingertip on the map is A, the target map element location is B, and B is located northeast of A. Then, one or more indicator electrodes 410 indicating the northeast direction will receive the target direction signal and output a second electrotactile signal.

[0072] For example, suppose that the target location A is pointed to by the user's fingertip on the map, and the target location A is also the location of the target map element. In physical space, the fingertip is directly above the target location A on the map. Then, one or more indicator electrodes 410 indicating the vertical downward direction will receive the target direction signal and output the second electrotactile signal.

[0073] for Figure 7 In the illustrated map browsing assistance device, after the control unit 300 acquires the user's interactive voice through the voice interaction unit 500, it performs content recognition on the interactive voice through the built-in neural network model. After recognizing that the interactive voice includes the content of inquiring about the location of the target map element, it first determines the location of the inquired target map element (i.e., the target map element position) based on the established map coordinate system. Then, it determines the target direction based on the target map element position and the target position, and then outputs the target direction signal to the map indication unit 400 so that the corresponding indication electrode 410 outputs the second electrotactile signal.

[0074] In one embodiment, the control unit 300 can also determine the target map element of the inquiry as the aforementioned point of interest element based on the interactive voice.

[0075] By outputting a second electrotactile signal through one or more indicator electrodes 410, the user can be assisted in finding target map elements on the map, thereby making it easier for the user to read the map and improving the convenience of map reading.

[0076] In one embodiment, the voice interaction unit 500 can be used not only to acquire interactive voice but also to play response voice based on a response signal, thereby enabling two-way voice interaction between humans and machines. Specifically, the voice interaction unit 500 may include a voice acquisition module 510 and a voice playback module 520. Both the voice acquisition module 510 and the voice playback module 520 are connected to the control unit 300. The voice acquisition module 510 can be used to acquire interactive voice, and the voice playback module 520 can be used to play response voice based on a response signal. The voice acquisition module 510 can be a microphone, and the voice playback module 520 can be a wireless headset, Bluetooth speaker, or other similar device.

[0077] In one embodiment, when the voice interaction unit 500 includes a voice playback module 520, after the control unit 300 outputs a map element indication signal to the map indication unit 400, it will also generate a response signal based on the map elements of the target location, thereby enabling the voice playback module 520 to play the corresponding response voice to the user, informing the user of the specific map elements of the target location through voice.

[0078] In one embodiment, the map indication unit 400 may also be used to output a third electrotactile signal based on a navigation direction signal through one or more of a plurality of indication electrodes 410, wherein the third electrotactile signal may be used to indicate the direction of the target location relative to a target route, the target route being a route to a location of interest; the control unit 300 may also be used to output a navigation direction signal to the map indication unit 400 when the interactive voice includes voice content requesting a route to a location of interest.

[0079] In one embodiment, the direction of the target location relative to the target route refers to the direction from the target location as the starting point, along the target route, towards the location of interest. Here, when the target location is a non-road element (i.e., the user does not point to a road element), the direction of the target location relative to the target route can include the direction guiding the user from the target location to the nearest road element, and the direction of the route from that location to the location of interest after the user points to the nearest road element; when the target location is a road element, the direction of the target location relative to the target route can include the direction of the route from the target location to the location of interest. The location of interest refers to a location on the map that the user is interested in. This location can be a location displayed on the current terminal's user interface, or a location that cannot be displayed on the current terminal's user interface (e.g., a location that cannot be displayed after zooming in on the electronic map), and is not specifically limited here.

[0080] Specifically, after the control unit 300 acquires the user's interactive voice through the voice interaction unit 500, it performs content recognition on the interactive voice through the built-in neural network model. After recognizing that the interactive voice includes a request for a route to a location of interest, it first determines the requested location of interest based on the established map coordinate system, and then performs path planning based on the location of interest and the target location to obtain the navigation information corresponding to the planned path. Then, it generates a navigation direction signal based on the navigation information, and then outputs the navigation direction signal to the map indicator unit 400. Similar to the method of indicating the target direction described above, the navigation direction signal causes the corresponding indicator electrode 410 to output a third electrotactile signal.

[0081] By outputting a third electrotactile signal through one or more indicator electrodes 410, the user can be assisted in reading navigation routes on the map, thereby making it easier for the user to read the map and improving the convenience of map reading.

[0082] In one embodiment, the map indication unit 400 may also be used to output a fourth electrotactile signal based on a path condition signal through one or more of a plurality of indication electrodes 410, wherein the fourth electrotactile signal is used to indicate the path condition of the target route; the control unit 300 may also be used to output a path condition signal to the map indication unit 400 when the interactive voice includes voice content requesting the path condition of the target route and the image includes path condition information.

[0083] In one embodiment, the path condition signal refers to a signal carrying path condition information such as whether the target route is congested.

[0084] Specifically, some existing electronic maps, in order to improve readability, use pop-ups or display roads in specific colors on the user interface to indicate road congestion status. Therefore, the control unit 300 can also perform image recognition on the acquired images to obtain road condition information. After identifying the area image containing road condition information, it extracts the road condition information from this area image (e.g., a green path indicates a clear road, while a red path indicates congestion), thus determining the road condition. Then, the control unit 300 can generate a road condition signal based on the road condition and output it to the map indicator unit 400. The map indicator unit 400 can then cause one or more corresponding indicator electrodes 410 to output a fourth electrotactile signal based on the road condition signal. After receiving the fourth electrotactile signal, the human skin can more easily perceive the road condition on the target route.

[0085] In one embodiment, the map indication unit 400 is further configured to output a fifth electrotactile signal based on weather conditions via one or more of a plurality of indication electrodes 410, wherein the fifth electrotactile signal is used to indicate the weather conditions of the area to which the target location belongs; the control unit 300 may also be configured to output a weather condition signal to the map indication unit 400 when the interactive voice includes voice content requesting weather conditions of the target location, and when the image includes weather condition information. Here, weather conditions may include sunny, cloudy, and rainy / snowy weather.

[0086] Specifically, to improve readability, some existing electronic maps display weather conditions for the target location's area via pop-ups, information boxes, etc., on the user interface of the terminal. To this end, the control unit 300 can also perform image recognition on the acquired images to identify the weather conditions of the target location's area. After identifying the area image containing the weather conditions of the target location's area, it extracts the weather condition information from this area image, thereby determining the weather conditions of the target location's area. Then, the control unit 300 can generate a weather condition signal based on the determined weather conditions and output it to the map indicator unit 400. The map indicator unit 400 can then cause one or more corresponding indicator electrodes 410 to output a fifth electrotactile signal based on the weather condition signal. After receiving the fourth electrotactile signal, the human skin can more easily perceive the weather conditions of the target location's area.

[0087] In one embodiment, the control unit 300 can also be used to output an adjusted signal to the map indication unit 400 when the interactive voice includes voice content requesting adjustment of the electrotactile signal intensity, so that the indication electrode 410 outputs the adjusted electrotactile signal. The electrotactile signal referred to herein can be any one of the first to fifth electrotactile signals, without specific limitation. Furthermore, the adjustment of the electrotactile signal intensity can be to decrease or increase the voltage of the electrotactile signal, without specific limitation.

[0088] Adjusting the intensity of the electrotactile signal via voice improves the convenience and comfort of the user when using the device.

[0089] In one embodiment, multiple indicator electrodes 410 can represent information by being attached to a single finger; specifically, see [link to relevant documentation]. Figures 8A to 8B The plurality of indicator electrodes 410 include at least one of the following: one or more fingertip indicator electrodes 411; one or more finger pad indicator electrodes 412; one or more finger back indicator electrodes 413; one or more fingertip edge indicator electrodes 414.

[0090] The following example illustrates how to output specific information. It's important to note that... Figures 9A to 11C In the diagram shown, the dark indicator electrode 410 on the finger indicates that the indicator electrode is outputting an electrotactile signal.

[0091] See Figures 9A to 9E For map elements at the target location:

[0092] When the map element is a road element, the fingertip indicator electrode 412 outputs a continuous first electrotactile signal through the map element indicator signal to express the road element through continuous stimulation of the fingertip.

[0093] When the map element is a building element, the fingertip indicator electrode 411, the back of the finger indicator electrode 413, and the fingertip edge indicator electrode 414 all output a continuous first electrotactile signal through the map element indicator signal, so as to express the building element through continuous stimulation of the fingertip, the back of the finger, and the fingertip edge.

[0094] When the map element is a green space element, one or more of the three types of indicator electrodes, namely the fingertip indicator electrode 411, the fingertip indicator electrode 412, and multiple fingertip edge indicator electrodes 414, randomly output a first electrotactile signal for a short period of time to express the green space element by randomly and briefly stimulating the fingertip and the fingertip edge.

[0095] When the map element is a water element, a short-duration first electrotactile signal is sequentially output from the fingertip indicator electrode 411 to multiple fingertip edge indicator electrodes 414 to express the water element through a spreading short-duration stimulation from fingertip to fingertip edge.

[0096] When a map element is a point of interest element, one or more of the multiple fingertip pointing electrodes 411 randomly output a first electrotactile signal for a short period of time to express the point of interest element by randomly stimulating the fingertip.

[0097] For directional indications (including the direction from the target location to the target map element location and the direction from the target location to the target route):

[0098] When the direction is a single direction (such as directly in front, directly to the left, directly to the right, directly below, or directly behind), the direction is indicated by outputting a continuous electrotactile signal through one of the fingertip indicator electrode 411, the finger pad indicator electrode 412, the back of the finger indicator electrode 413, and the fingertip edge indicator electrode 414.

[0099] For example, if the target map element is located directly in front of the target location, the fingertip indicating electrode 411 outputs a continuous second electrotactile signal; if the target map element is located directly to the left of the target location, the left fingertip edge indicating electrode 414 outputs a continuous second electrotactile signal; if the target map element is located directly to the right of the target location, the right fingertip edge indicating electrode 414 outputs a continuous second electrotactile signal; if the target map element is located directly behind the target location, the back of the finger indicating electrode 413 outputs a continuous second electrotactile signal; if the target map element location coincides with the target location, the fingertip indicating electrode 412 outputs a continuous second electrotactile signal.

[0100] For example, if the navigation direction is directly in front of the target position, the fingertip indicator electrode 411 outputs a continuous third electrotactile signal; if the navigation direction is directly to the left of the target position, the left fingertip edge indicator electrode 414 outputs a continuous third electrotactile signal; if the navigation direction is directly to the right of the target position, the right fingertip edge indicator electrode 414 outputs a continuous third electrotactile signal; if the navigation direction is directly behind the target position, the back of the finger indicator electrode 413 outputs a continuous third electrotactile signal; if the position of interest coincides with the target position, the fingertip indicator electrode 412 outputs a continuous third electrotactile signal.

[0101] When the direction is a composite direction (such as left front, left rear, right front, right rear), the direction is represented by two continuous electrotactile signals output by the fingertip indicator electrode 411, the back of the finger indicator electrode 413, and the fingertip edge indicator electrode 414.

[0102] For example, if the target map element is located to the left front of the target position, the fingertip indicator electrode 411 and the left fingertip edge indicator electrode 414 will output a continuous second electrotactile signal; if the target map element is located to the left rear of the target position, the finger back indicator electrode 413 and the left fingertip edge indicator electrode 414 will output a continuous second electrotactile signal; if the target map element is located to the right front of the target position, the fingertip indicator electrode 411 and the right fingertip edge indicator electrode 414 will output a continuous second electrotactile signal; if the target map element is located to the right rear of the target position, the finger back indicator electrode 413 and the right fingertip edge indicator electrode 414 will output a continuous second electrotactile signal.

[0103] For example, if the navigation direction is to the left front of the target position, the fingertip indicator electrode 411 and the left fingertip edge indicator electrode 414 will output a continuous third electrotactile signal; if the navigation direction is to the left rear of the target position, the finger back indicator electrode 413 and the left fingertip edge indicator electrode 414 will output a continuous third electrotactile signal; if the navigation direction is to the right front of the target position, the fingertip indicator electrode 411 and the right fingertip edge indicator electrode 414 will output a continuous third electrotactile signal; if the navigation direction is to the right rear of the target position, the finger back indicator electrode 413 and the right fingertip edge indicator electrode 414 will output a continuous third electrotactile signal.

[0104] See Figures 10A to 10B Regarding the indication of path conditions:

[0105] When the target route is unobstructed, a continuous low-frequency fourth electrotactile signal is output through the fingertip indicator electrode 412 to express that the target route is unobstructed through continuous low-frequency stimulation of the fingertip.

[0106] When the target route is congested, a continuous high-frequency fourth electrotactile signal is output through the fingertip indicator electrode 412 to express the congestion of the target route through high-frequency continuous stimulation of the fingertip.

[0107] See Figures 11A to 11C Regarding weather conditions,

[0108] When the weather is sunny, a continuous fifth electrotactile signal is output through the finger back indicator electrode 413 to express that the weather is sunny through continuous stimulation of the finger back.

[0109] When the weather is cloudy, a continuous fifth electrotactile signal is output through the fingertip indicator electrode 411, the back of the finger indicator electrode 413 and multiple fingertip edge indicator electrodes 414 to express that the weather is cloudy by continuously stimulating the fingertip, the back of the finger and the fingertip edge.

[0110] When the weather is rainy or snowy, a fifth electrotactile signal is randomly output for a short period of time through one or more of the fingertip indicator electrode 411, the back of the finger indicator electrode 413 and the multiple fingertip edge indicator electrodes 414, so as to express that the weather is rainy or snowy by randomly stimulating the fingertip and the fingertip edge.

[0111] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0112] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, or the existence of B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one of a, b, and c can represent: the existence of a alone, the existence of b alone, the existence of c alone, the simultaneous existence of a and b, the simultaneous existence of a and c, the simultaneous existence of b and c, or the simultaneous existence of a, b, and c, where a, b, and c can be single or multiple.

[0113] In the embodiments of this application, "instruction" can include direct and indirect instructions, as well as explicit and implicit instructions. The information indicated by a certain piece of information is called the information to be instructed. In the specific implementation process, there can be many ways to instruct the information to be instructed, such as, but not limited to, directly instructing the information to be instructed, such as instructing the information to be instructed itself or its index. It can also indirectly instruct the information to be instructed by instructing other information, where there is a relationship between the other information and the information to be instructed. It can also instruct only a part of the information to be instructed, while the other parts are known or pre-agreed upon. For example, the instruction of specific information can be achieved by using a pre-agreed (e.g., protocol-defined) arrangement of various pieces of information, thereby reducing the instruction overhead to some extent.

[0114] In the embodiments of this application, the terms and English abbreviations are exemplary examples given for ease of description and should not be construed as limiting the application in any way. This application does not preclude the possibility of defining other terms that can achieve the same or similar functions in existing or future agreements.

[0115] In the embodiments of this application, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first," "second," or "third" may explicitly or implicitly include one or more of that feature.

[0116] The embodiments described above are merely examples of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application.

Claims

1. A map browsing assistance device based on multimodal haptic feedback, characterized in that, include: Image acquisition unit, used to acquire images; A temperature sensing unit for wearable devices, and for outputting a temperature sensing signal based on a temperature sensing adjustment signal; The control unit is connected to the image acquisition unit and to the temperature sensing unit via a flexible substrate. The control unit is used to output a temperature sensing adjustment signal for the ambient temperature of the target location to the temperature sensing unit when the image includes an image of the target area where a finger points to a target location on a map and ambient temperature information.

2. The apparatus according to claim 1, characterized in that, The device further includes: The map indicator unit includes multiple indicator electrodes, which are connected to the control unit via a flexible substrate. The map indicator unit is wearable and outputs a first electrotactile signal based on map element indication signals through one or more of the multiple indicator electrodes. The first electrotactile signal is used to indicate map elements at the target location. The control unit is also configured to output the map element indication signal to the map indication unit when the image includes the target area image.

3. The apparatus according to claim 2, characterized in that, The device further includes: A voice interaction unit is connected to the control unit, and the voice interaction unit is used to acquire the user's interactive voice. The map indicator unit is further configured to output a second electrotactile signal based on a target direction signal through one or more of the plurality of indicator electrodes, wherein the second electrotactile signal is used to indicate the direction in which the target location points to the location of the target map element; The control unit is also configured to output the target direction signal to the map indication unit when the interactive voice includes a voice inquiry about the location of the target map element.

4. The apparatus according to claim 3, characterized in that, The map indicator unit is also configured to output a third electrotactile signal based on a navigation direction signal through one or more of the plurality of indicator electrodes, wherein the third electrotactile signal is used to indicate the direction of the target location relative to a target route, the target route being a route to a location of interest; The control unit is further configured to output the navigation direction signal to the map indication unit when the interactive voice includes voice content requesting a route to the location of interest.

5. The apparatus according to claim 4, characterized in that, The map indicator unit is also configured to output a fourth electrotactile signal based on a path condition signal through one or more of the plurality of indicator electrodes, wherein the fourth electrotactile signal is used to indicate the path condition of the target route; The control unit is further configured to output the path condition signal to the map indication unit when the interactive voice includes voice content requesting the path condition of the target route and the image includes path condition information.

6. The apparatus according to claim 2, characterized in that, The map elements include one of the following: road elements; building elements; green space elements; water body elements; points of interest elements.

7. The apparatus according to claim 2, characterized in that, The plurality of indicated electrodes include at least one of the following: one or more fingertip indicated electrodes; one or more finger pad indicated electrodes; one or more finger back indicated electrodes; one or more fingertip edge indicated electrodes.

8. The apparatus according to claim 1, characterized in that, The temperature sensing signal includes at least one of a first temperature sensing signal and a second temperature sensing signal, wherein the temperature corresponding to the first temperature sensing signal is higher than the temperature corresponding to the second temperature sensing signal; The temperature sensing unit includes: A heating module is connected to the control unit via a flexible substrate, and the heating module is used to output a first temperature sensing signal based on the temperature sensing adjustment signal; The heat absorption module is connected to the control unit via a flexible substrate, and the heat absorption module is used to output the second temperature sensing signal based on the temperature sensing adjustment signal.

9. The apparatus according to claim 3, characterized in that, The voice interaction unit is also used to play a response voice based on the response signal; The control unit is further configured to generate and output the response signal for the map element to the map indication unit when the image includes the target area image.

10. The apparatus according to claim 9, characterized in that, The voice interaction unit includes: A voice acquisition module is connected to the control unit, and the voice acquisition module is used to acquire the interactive voice. A voice playback module is connected to the control unit, and the voice playback module is used to play the reply voice.

Images