Multi-mode interactive sensing method, device and system for dynamic panoramic environment information

Abstract

The invention provides a multi-mode interactive perception method, device and system for dynamic panoramic environment information, and the method comprises the steps: obtaining a recommendation list of environment objects, and then recommending the environment objects to a visual impaired user through a multi-mode interaction mode based on the recommendation list. The multi-modal interaction comprises multiple dimensions such as audio guidance, tactile level feedback, coverage exploration, understanding, recall and social interaction. A recommendation list is generated by an environment recommendation model from the dynamic panoramic video, the model ranking a plurality of environment objects according to a comprehensive interest score. The comprehensive interest scoring comprehensively considers the scores of visual impaired users for aesthetics, freshness and demand factors. And the environment recommendation model iteratively updates association weights of aesthetics, freshness and demand factors by using touch level feedback, and further iteratively updates a comprehensive interest score and a recommendation list. The method can dynamically adapt to user preferences and provide personalized environment perception assistance, so that the independence and life quality of visually impaired users are enhanced.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multimodal interactive perception method, device, and system for dynamic panoramic environmental information. Background Art

[0002] With the continuous advancement of science and technology, more and more technologies are being applied to improve the quality of life for people with disabilities. For people with blindness and low vision (BLV), exploring and perceiving unfamiliar environments has always been a significant challenge. Traditional assistive devices, such as canes and guide dogs, while somewhat helpful for people with BLV in their daily travel, are significantly inadequate in providing environmental information and enhancing the user experience. Blind people are generally limited to passively receiving basic route descriptions and are unable to engage in in-depth, interactive environmental exploration. Summary of the Invention

[0003] In order to solve the technical problems existing in the prior art, the present invention provides a multimodal interactive perception method, device and system for dynamic panoramic environmental information, which can dynamically adapt to user preferences and provide personalized environmental perception assistance, thereby enhancing the independence and quality of life of visually impaired users.

[0004] The present invention provides a multimodal interactive perception method for dynamic panoramic environmental information, which is applied to a user terminal device. The method comprises: obtaining a recommendation list for perception of environmental objects; based on the recommendation list, recommending environmental objects to a visually impaired user through multimodal interaction, so as to assist the visually impaired user in environmental perception from multiple dimensions; the multimodal interaction comprises audio guidance and tactile hierarchical feedback; the multiple dimensions comprise exploration, understanding, recall and social interaction dimensions; wherein the recommendation list is a list of several environmental objects sorted by comprehensive interest scores, generated by an environmental recommendation model based on a dynamic panoramic environmental video; the comprehensive interest score is a score that comprehensively considers the aesthetics, novelty and demand factors of the visually impaired user; the environmental recommendation model is used to iteratively update the associated weights of aesthetics, novelty and demand factors based on the tactile hierarchical feedback, so as to iteratively update the comprehensive interest score and the recommendation list.

[0005] According to a multimodal interactive perception method of dynamic panoramic environmental information provided by the present invention, when the visually impaired user is exploring the environment, information of recommended environmental objects is described to the visually impaired user through audio based on the recommendation list; the information of the recommended environmental objects includes the name, orientation and attributes of the recommended environmental objects; the exploration action of the visually impaired user is determined; the exploration action is used to characterize the preference information of the visually impaired user for the currently recommended environmental objects; the preference information includes likes and dislikes; and the recommendation list is dynamically adjusted based on the exploration action.

[0006] According to a multimodal interactive perception method of dynamic panoramic environmental information provided by the present invention, when the visually impaired user understands the environment, based on the recommendation list, a two-layer hierarchical scene interaction architecture is adopted to form several environmental objects in a main layer graph structure and a sub-layer graph structure; the nodes of the main layer graph structure are coarse-grained environmental objects, and the edges of the main layer graph structure are the relationships between the coarse-grained objects; the nodes of the sub-layer graph structure are fine-grained environmental objects, and the edges of the sub-layer graph structure are the relationships between the fine-grained environmental objects; based on the recommendation list, the information of the coarse-grained environmental objects in the main layer graph structure is described to the visually impaired user through audio; the understanding action of the visually impaired user is determined; the understanding action is used to represent the need of the visually impaired user to continue to understand the information of finer-grained environmental objects under the currently recommended coarse-grained environmental object; based on the understanding action, the information of the fine-grained environmental objects in the sub-layer graph structure corresponding to the currently recommended coarse-grained environmental object is described to the visually impaired user through audio.

[0007] According to a multimodal interactive perception method of dynamic panoramic environmental information provided by the present invention, when the visually impaired user is recalling the environment, the recall action of the visually impaired user is determined; the recall action is used to characterize the visually impaired user's recall need for historical environmental objects; based on the recall action, historical multimodal interactive environmental perception information is retrieved, and the information of the historical environmental objects is described to the visually impaired user through audio.

[0008] According to a multimodal interactive perception method of dynamic panoramic environmental information provided by the present invention, when the visually impaired user performs environmental social interaction, the social interaction action of the visually impaired user is determined; the social interaction action is used to represent the current visually impaired user's need to share information about his or her own historical environmental objects with another visually impaired user; based on the social interaction action, historical multimodal interactive environmental perception information is retrieved, and the historical multimodal interactive environmental perception information is sent to the receiving device of the other visually impaired user, so that the current visually impaired user can share the information about the historical environmental objects with the other visually impaired user.

[0009] According to a multimodal interactive perception method of dynamic panoramic environment information provided by the present invention, the method further includes: converting the dynamic panoramic environment video into a semantic graph sequence through a scene graph generation algorithm, so as to input the semantic graph sequence into the environment recommendation model; the environment recommendation model is based on the environment video training sample, and is trained by using graph mask self-supervised learning and multimodal attention mechanism; the environment recommendation model includes a background network, an aesthetic network, a freshness network and a demand network; the background network is used to identify background objects in the semantic graph sequence through the background attention mechanism, and calculate the score of the background object to obtain a background score; the aesthetic network is used to identify the background objects in the semantic graph sequence through the aesthetic attention mechanism The interesting objects that appear continuously in the semantic graph sequence within a preset time period are scored, and the scores of the interesting objects are calculated to obtain an aesthetic score; the freshness network is used to identify the freshness objects that appear for the first time in the semantic graph sequence compared with the previous frame through the freshness attention mechanism, and calculate the scores of the freshness objects to obtain a freshness score; the demand network is used to identify the physiological and safety-related demand objects for the visually impaired user in the semantic graph sequence through the demand attention mechanism, and calculate the scores of the demand objects to obtain a demand score; the comprehensive interest score is a score determined by comprehensively considering the background score, the aesthetic score, the freshness score and the demand score.

[0010] According to a multimodal interactive perception method of dynamic panoramic environmental information provided by the present invention, the environmental recommendation model also includes a user interaction adapter; the user interaction adapter is used to dynamically adjust the weights of the background score, the aesthetic score, the freshness score and the demand score based on the tactile hierarchical feedback using a maximum likelihood estimation algorithm.

[0011] The present invention also provides a multimodal interactive perception device for dynamic panoramic environmental information, which is applied to a user terminal device, and the device includes: a recommendation list acquisition module, which is used to obtain a recommendation list for environmental object perception; an environmental object recommendation module, which is used to recommend environmental objects to visually impaired users through multimodal interaction based on the recommendation list, so as to assist the visually impaired users in environmental perception from multiple dimensions; the multimodal interaction includes audio guidance and tactile hierarchical feedback; the multiple dimensions include exploration, understanding, recall and social interaction dimensions; wherein, the recommendation list is a list of several environmental objects sorted by comprehensive interest scores generated by the environmental recommendation model based on the dynamic panoramic environmental video; the comprehensive interest score is a score that comprehensively considers the aesthetics, novelty and demand factors of the visually impaired user; the environmental recommendation model is used to iteratively update the associated weights of aesthetics, novelty and demand factors based on the tactile hierarchical feedback, so as to iteratively update the comprehensive interest score and the recommendation list.

[0012] The present invention also provides a multimodal interactive perception system for dynamic panoramic environmental information, comprising: an acquisition device for capturing dynamic panoramic environmental video and transmitting the dynamic panoramic environmental video to a server; the server for running an environmental recommendation model based on the dynamic panoramic environmental video to generate a recommendation list; the recommendation list is a list of several environmental objects sorted by comprehensive interest scores; the comprehensive interest score is a score that comprehensively considers the aesthetics, novelty and demand factors of visually impaired users; the environmental recommendation model is used to iteratively update the associated weights of aesthetics, novelty and demand factors based on the tactile hierarchical feedback to iteratively update the comprehensive interest score and the recommendation list; the user terminal device is used to execute the above-mentioned multimodal interactive perception method for dynamic panoramic environmental information.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the multimodal interactive perception method for dynamic panoramic environment information as described in any one of the above is implemented.

[0014] The present invention provides a multimodal interactive perception method, device and system for dynamic panoramic environmental information, which includes obtaining a recommendation list of environmental objects, and then recommending environmental objects to visually impaired users through multimodal interaction based on the recommendation list. Multimodal interaction includes audio guidance and tactile hierarchical feedback, covering multiple dimensions such as exploration, understanding, recall and social interaction. The recommendation list is generated by an environmental recommendation model based on dynamic panoramic video, and the model sorts multiple environmental objects according to a comprehensive interest score. The comprehensive interest score comprehensively considers the visually impaired user's score on aesthetics, novelty and demand factors. The environmental recommendation model uses tactile hierarchical feedback to iteratively update the associated weights of aesthetics, novelty and demand factors, and then iteratively updates the comprehensive interest score and recommendation list. The present invention can dynamically adapt to user preferences and provide personalized environmental perception assistance, thereby enhancing the independence and quality of life of visually impaired users. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 This is a flow chart of a multimodal interactive perception method for dynamic panoramic environmental information provided by the present invention.

[0017] Figure 2It is a structural schematic diagram of a multimodal interactive perception device for dynamic panoramic environmental information provided by the present invention.

[0018] Figure 3 It is a structural schematic diagram of a multimodal interactive perception system for dynamic panoramic environmental information provided by the present invention.

[0019] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0020] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0021] Connecting with the natural environment is crucial for a person's physical and mental well-being. Millions of people with blindness and low vision (BLV) worldwide also yearn to actively explore these unknown wonders. However, existing research often assumes that BLV individuals have clearly defined their needs and focuses solely on providing functional assistance such as navigation and obstacle avoidance (e.g., guide dogs). This leads to passive dependence on others in unfamiliar environments, leading to a strong sense of helplessness. This severely hinders their independent exploration during dynamic sightseeing, subsequent memory retention, and interaction with peers. The vast amount of visual information in unfamiliar environments far exceeds the perceptual capacity of BLV individuals, resulting in excessive cognitive load. Therefore, helping BLV individuals understand and enjoy unfamiliar environments is an urgent need.

[0022] Please refer to Figure 1 , Figure 1 A schematic flow chart of a multimodal interactive perception method for dynamic panoramic environmental information provided by the present invention.

[0023] The present invention provides a multimodal interactive perception method for dynamic panoramic environmental information, which is applied to a user terminal device 3 and includes: 101: Get a recommendation list of environmental object perception; 102: Based on the recommendation list, environmental objects are recommended to visually impaired users through multimodal interaction to assist visually impaired users in environmental perception from multiple dimensions; multimodal interaction includes audio guidance and tactile hierarchical feedback; multiple dimensions include exploration, understanding, recall and social interaction dimensions; among them, the recommendation list is a list of several environmental objects sorted by comprehensive interest scores generated by the environmental recommendation model based on dynamic panoramic environmental video; the comprehensive interest score is a score that comprehensively considers the aesthetics, novelty and demand factors of visually impaired users; the environmental recommendation model is used to iteratively update the associated weights of aesthetics, novelty and demand factors based on tactile hierarchical feedback to iteratively update the comprehensive interest score and recommendation list.

[0024] This example describes in detail the implementation of a multimodal interactive perception method for dynamic panoramic environmental information. This method aims to provide visually impaired users with a platform that enables them to actively explore and deeply understand unfamiliar environments, achieve long-term memory, and effectively communicate. It reconstructs massive amounts of information in complex scenes into structured, personalized semantic graphs, enhancing their independence and quality of life.

[0025] The present invention integrates a lightweight, portable consumer-grade device combination. First, a user-worn capture device 1, such as a panoramic camera, captures real-time panoramic video of the user's surroundings. This video data is transmitted to server 2 via a wireless network. Server 2 can be a laptop, and the network is a local area network (LAN), offering high communication reliability with virtually no latency or interference. The environmental recommendation model deployed on server 2 uses image recognition and artificial intelligence technologies to analyze environmental objects in the video stream, such as trees, buildings, and pedestrians, and ranks these objects according to a comprehensive interest score. This comprehensive interest score takes into account aesthetics, novelty, and needs of the visually impaired user to create a personalized recommendation list. The recommendation list is then sent to the user's terminal device, such as a smartphone or dedicated assistive device. Based on the recommendation list, user terminal device 3 recommends environmental objects to the user through a multimodal interaction paradigm combining audio guidance and tactile hierarchical feedback, covering four dimensions: exploration, comprehension, recall, and social interaction. Audio guidance provides users with real-time environmental information, such as object names, locations, and features. For example, when a visually impaired user is walking outdoors, headphones provide audio description, such as "There is a big tree 10 meters ahead." Haptic feedback, through vibration or touch response on user terminal device 3, provides real-time information about the user's interaction with the environment. Users provide feedback on recommended objects through interactions with user terminal device 3, such as by touching the screen or issuing voice commands. This feedback is used to update the environmental recommendation model on server 2. The environmental recommendation model iteratively updates the associated weights of aesthetics, novelty, and demand factors based on the user's tactile feedback, further optimizing the overall interest score and recommendation list. This means that as the user interacts with user terminal device 3, the model learns the user's preferences and adjusts the recommended environmental objects accordingly, making recommendations more personalized and accurate. For example, if a visually impaired user is exploring a park, user terminal device 3 might recommend "the bench in front" and "the fountain on the left." The user can touch the bench icon to obtain more information, such as the bench's material and design details. At the same time, user terminal device 3 adjusts the weights of future recommendations based on the user's interactive feedback, ranking objects that the user is more interested in higher in future recommendations.

[0026] Furthermore, the contextual recommendation model can learn user behavior patterns in different environments and automatically adjust its recommendation strategy to suit different scenarios. This adaptive learning capability enables the contextual recommendation model to continuously optimize over time, providing users with a more personalized and accurate contextual perception experience. In this way, visually impaired users can navigate and explore their environment more confidently and independently.

[0027] It is important to note that modern smartphones have powerful computing capabilities, and in theory, the environmental recommendation model can be run entirely on a smartphone without the need for a laptop. However, this requires significant engineering work to optimize mobile deployment, which is an important direction for future work to make user terminal devices more practical and accessible for real-world applications.

[0028] Experiments have shown that this method, by allowing visually impaired users to lead their exploration paths and dynamically adjust information supply, enhances their positive emotions and memory accuracy. Both emotional valence (pleasure) and arousal (activation) are significantly increased, providing them with unprecedented pleasure and unforgettable experiences. This method opens the door for the visually impaired to independently explore the world, propelling the development of a barrier-free society into a new stage.

[0029] As a preferred embodiment, when a visually impaired user is exploring the environment, information about recommended environmental objects is described to the visually impaired user through audio based on a recommendation list; the information about the recommended environmental objects includes the name, location and attributes of the recommended environmental objects; the exploration action of the visually impaired user is determined; the exploration action is used to characterize the preference information of the visually impaired user for the current recommended environmental object; the preference information includes likes and dislikes; and the recommendation list is dynamically adjusted based on the exploration action.

[0030] In this embodiment, when a visually impaired user explores an environment, server 2 first uses an environmental recommendation model to generate a recommendation list. This list, based on a dynamic panoramic video of the environment, ranks multiple environmental objects according to a comprehensive interest score. This comprehensive interest score integrates the visually impaired user's aesthetic, novelty, and needs, significantly enhancing their sense of immersion in the environment. User terminal device 3 then uses audio to describe the recommended environmental objects to the visually impaired user, including the object's name, location, and attributes. For example, user terminal device 3 might play an audio message: "There is a wooden bench on your left, about two meters away." Simultaneously, user terminal device 3 monitors the visually impaired user's exploration actions to determine their preference for the recommended environmental objects. Exploration actions may include, but are not limited to, touching, clicking, or performing specific gestures on the device. These actions indicate a user's like or dislike for a particular environmental object. For example, if a user touches or clicks the bench icon after hearing the description, user terminal device 3 will record this action as preference information.

[0031] The exploration interface of the user terminal device 3 of this embodiment is designed for visually impaired users. It prompts interesting, novel or necessary environmental objects in the surrounding area in real time, allowing users to provide tactile feedback based on their preferences for the currently recommended environmental objects, thereby enhancing the user's interactive experience with the environment. Tactile feedback technology is used to allow users to perceive and select environmental objects by touching the screen. The interface design takes into account the special needs of visually impaired users, adopts a simple and intuitive layout, and distinguishes different environmental objects through different tactile patterns or vibration patterns. Users browse the list of recommended environmental objects by touching and sliding on the exploration interface. Each object is associated with a specific tactile feedback pattern. For example, trees may be associated with a rough touch, while water bodies may be associated with a smooth, undulating touch. Users can get detailed descriptions by touching these objects, such as the name, location, characteristics, etc. of the object.

[0032] Based on these exploration actions, the recommendation list can be dynamically adjusted. If a user expresses a preference for an environmental object, the object's weight in the recommendation list increases, ranking it higher in future recommendations. Conversely, if a user expresses a dislike for an object, the object's weight decreases, reducing its frequency of appearance in future recommendations. This dynamic adjustment mechanism can better adapt to a user's personal preferences and provide a more personalized environmental perception experience. For example, a user may be particularly interested in a newly discovered cafe (freshness) or have an urgent need for a nearby restroom (need). Users can express their preferences through simple tactile actions, such as a long press or double-click. This preference feedback is sent back to Server 2 in real time to update the environmental recommendation model.

[0033] Furthermore, it is possible to learn the user's general preferences for different types of environmental objects and adjust the recommendation algorithm based on these preferences. For example, if the user terminal device 3 finds that the user generally shows a high interest in trees and water bodies, it may give these types of objects a higher initial weight when generating the recommendation list.

[0034] As a preferred embodiment, when a visually impaired user understands the environment, based on the recommendation list, a two-layer hierarchical scene interaction architecture is adopted to form several environmental objects in the main layer graph structure and the sub-layer graph structure; the nodes of the main layer graph structure are coarse-grained environmental objects, and the edges of the main layer graph structure are the relationships between coarse-grained objects; the nodes of the sub-layer graph structure are fine-grained environmental objects, and the edges of the sub-layer graph structure are the relationships between fine-grained environmental objects; based on the recommendation list, the information of the coarse-grained environmental objects in the main layer graph structure is described to the visually impaired user through audio; the visually impaired user's understanding action is determined; the understanding action is used to represent the visually impaired user's need to continue to understand the information of finer-grained environmental objects under the currently recommended coarse-grained environmental object; based on the understanding action, the information of the fine-grained environmental objects in the sub-layer graph structure corresponding to the currently recommended coarse-grained environmental object is described to the visually impaired user through audio.

[0035] In this embodiment, the user terminal device 3 first adopts a two-layer hierarchical scene interaction architecture based on the recommendation list to enhance the visually impaired user's cognition and understanding of environmental information. The hierarchical graph structure promotes semantic association memory and supports subsequent social sharing. In the main layer graph structure, the nodes represent coarse-grained environmental objects, such as trees, buildings, etc., which are the main elements that users first come into contact with during environmental exploration. The edges of the main layer graph structure represent the relationships between these coarse-grained objects, such as proximity or inclusion relationships. The sub-layer graph structure contains fine-grained environmental objects that provide more detailed information, such as the texture of leaves, windows of buildings, etc. The edges of the sub-layer graph structure represent the relationships between these fine-grained environmental objects, such as the relationship between part and whole.

[0036] The user terminal device 3 describes the information of the coarse-grained environmental objects in the main layer graph structure to the visually impaired user through audio, for example: "There is a big tree in front of you with a thick trunk and lush leaves." Users express their interest in finer-grained information by touching or clicking nodes in the main layer graph, and these actions are recognized as understanding actions.

[0037] Based on these understanding actions, the user terminal device 3 describes the information of the fine-grained environmental objects in the sub-layer structure corresponding to the currently recommended coarse-grained environmental object to the visually impaired user through audio, for example: "The leaves of the big tree are green, shaped like palms, and may feel a little rough to the touch." Such descriptions help users build a deeper understanding of the environment.

[0038] The user terminal device 3 also includes a graph-structured cognitive enhancement interface in its interface design, which distills complex environmental information into a sparse, hierarchical graph structure. The graph-structured cognitive enhancement interface consists of a main-layer graph structure interface and a sub-layer graph structure interface. The main-layer graph structure interface is a simplified information presentation layer interface formed by a recommendation list and scene topology mapping. In this interface, environmental objects are organized into a simplified graph structure. This simplified graph structure helps visually impaired users quickly grasp the overall layout and key objects of the environment. The sub-layer graph structure interface allows visually impaired users to independently zoom in on specific nodes in the main-layer graph structure interface based on their perceived scene hierarchy, presenting specific nodes and their associated relationships. When a user is interested in a node in the main-layer graph structure, they can select it by touch or voice command. The sub-layer graph structure interface then displays detailed information about the node and its relationships with other nodes, such as adjacent paths and nearby facilities. This two-layer hierarchical scene interaction architecture allows visually impaired users to independently select the level of information presentation and the degree of detail based on their needs and interests. Users can start from the macro main layer graph structure interface and gradually move down to the micro sub-layer graph structure interface to gain a more detailed understanding of specific environmental objects and their associated information.

[0039] As a preferred embodiment, when a visually impaired user performs environmental recall, the recall action of the visually impaired user is determined; the recall action is used to characterize the visually impaired user's recall needs for historical environmental objects; based on the recall action, historical multimodal interactive environmental perception information is retrieved, and the information of the historical environmental objects is described to the visually impaired user through audio.

[0040] In this embodiment, the user terminal device 3 is equipped with advanced data storage and retrieval capabilities, capable of recording every user interaction, including touch operations, audio feedback selections, and tactile feedback preferences, enabling users to replay the guided tour for an immersive experience. This data is associated with the user's environmental perception information at a specific time and place, forming a historical record of multimodal interactive environmental perception information.

[0041] The user terminal device 3 first needs to identify the visually impaired user's recall action. These actions may include touching a specific area on the screen, issuing a specific voice command, or performing a preset gesture to indicate that the user wishes to recall an environmental object encountered during a previous exploration. For example, the user might trigger a recall action by double-clicking the screen or saying "I want to recall that fountain."

[0042] Once the user terminal device 3 recognizes the recall action, it retrieves relevant multimodal interactive environmental perception information from stored historical data. This information may include audio descriptions of environmental objects, tactile feedback, and the user's interaction history with these objects. The user terminal device 3 integrates this information and prepares an audio description of the historical environmental objects to the user, allowing the user to re-experience the environmental perception at that time.

[0043] For example, if the user's recall action points to a fountain in a park that has been explored before, the user terminal device 3 will retrieve the audio description and tactile feedback related to the fountain from the historical data, and then describe the characteristics of the fountain to the user through the audio player, such as "The fountain you encountered before is located in the center of the park, with a water column about three meters high and surrounded by smooth stone benches."

[0044] As a preferred embodiment, when a visually impaired user performs environmental social interaction, the social interaction action of the visually impaired user is determined; the social interaction action is used to represent the current visually impaired user's need to share information about his or her own historical environmental objects with another visually impaired user; based on the social interaction action, historical multimodal interactive environmental perception information is retrieved, and the historical multimodal interactive environmental perception information is sent to a receiving device of another visually impaired user, so that the current visually impaired user can share information about historical environmental objects with another visually impaired user.

[0045] The visually impaired have a strong desire to share experiences and a strong sense of community. In this embodiment, the user terminal device 3 first needs to identify the visually impaired user's social interaction actions. These actions may include specific touch gestures, voice commands, or device vibrations, indicating that the user wishes to share their historical environmental object information with another visually impaired user. For example, a user might trigger a social interaction action by performing a specific gesture on the touch screen or saying "Share this scene."

[0046] Once the user terminal device 3 recognizes a social interaction action, it automatically retrieves relevant multimodal interactive environment perception information from historical data. This information includes audio descriptions of environmental objects, tactile feedback, and the user's interaction history with these objects. The user terminal device 3 packages this information into a shared package and sends it via a wireless network to a receiving device of another visually impaired user, such as a smartphone, tablet, or other wearable device.

[0047] After receiving the information, the receiving device will play an audio description of the environmental object through its audio output device. It may also provide additional sensory information through a tactile feedback device, allowing the other visually impaired user to experience the shared environmental object. Furthermore, the receiving device can provide interactive feedback options, allowing users to request more information or comment on the shared content, thereby promoting social interaction between the two visually impaired users. This allows users to transform fragments of scenes and emotional memories from their journey into shareable digital content. This feature not only promotes knowledge exchange within the visually impaired community, but also strengthens social connections through collective memory, forming a unique space for cultural interaction.

[0048] For example, if a visually impaired user encounters an interesting sculpture while exploring a park and wishes to share their discovery with a friend, they can perform a social interaction action. The user terminal device 3 will recognize this action, retrieve detailed information about the sculpture from previous explorations, and then send this information to the friend's device. The friend can then hear a description of the sculpture and perhaps even feel its surface texture through haptic feedback, allowing them to experience the environmental object without being there in person.

[0049] This recall-based retrieval function not only helps visually impaired users better recall and understand their experiences, but also enhances their social interactions with others. By reviewing and sharing past environmental perception information, visually impaired users can participate more deeply in social activities, improving their social engagement and quality of life. Furthermore, this function can be used for educational and training purposes, helping visually impaired users learn how to better use their user terminal devices to explore and understand their environment.

[0050] As a preferred embodiment, it also includes: converting the dynamic panoramic environment video into a semantic graph sequence through a scene graph generation algorithm, so as to input the semantic graph sequence into the environment recommendation model; the environment recommendation model is based on the environment video training samples and is trained using graph mask self-supervised learning and a multimodal attention mechanism; the environment recommendation model includes a background network, an aesthetic network, a freshness network and a demand network; the background network is used to identify background objects in the semantic graph sequence through a background attention mechanism, and calculate the scores of the background objects to obtain a background score; the aesthetic network is used to identify interesting objects that appear continuously in the semantic graph sequence within a preset time period through an aesthetic attention mechanism, and calculate the scores of the interesting objects to obtain an aesthetic score; the freshness network is used to identify freshness objects that appear for the first time in the semantic graph sequence compared to the previous frame through a freshness attention mechanism, and calculate the scores of the freshness objects to obtain a freshness score; the demand network is used to identify physiological and safety-related demand objects for visually impaired users in the semantic graph sequence through a demand attention mechanism, and calculate the scores of the demand objects to obtain a demand score; the comprehensive interest score is a score determined by comprehensively combining the background score, aesthetic score, freshness score and demand score.

[0051] In this embodiment, dynamic panoramic environment video samples are converted into semantic graph sequences through a scene graph generation algorithm. The scene graph generation algorithm can identify key objects in the video and construct a graph structure that describes the objects and their relationships. The spatiotemporal semantic graph sequences in these first-person panoramic videos are used as input data to train the environment recommendation model. In order to train the environment recommendation model, a graph mask self-supervised learning method is adopted. In this process, a portion of the graph nodes (i.e., environmental objects) are randomly selected for masking, that is, the information of these nodes is temporarily hidden. The goal of the model is to predict the features of the masked nodes based on the information of the unmasked nodes. This method enables the model to learn the intrinsic connections between nodes and the semantic structure of the environment, effectively eliminating the subjective aesthetic bias of manual annotation. The model output generation can dynamically highlight the key environmental objects in the scene.

[0052] In unfamiliar environment assistance scenarios, effective information must be provided while avoiding cognitive overload. Visually impaired users pay close attention to aesthetics, novelty, and needs when exploring an environment. However, aesthetic judgments vary significantly from person to person. For example, some may enjoy magnificent scenery, while others prefer history and culture, and still others prefer the sounds of cicadas and birdsong. To this end, the environmental recommendation model is trained using a graph mask self-supervised learning mechanism based on environmental video training samples, using a multimodal attention mechanism to condense information. This effectively eliminates the subjective bias of manual annotation. These environmental video training samples can be tens of thousands of publicly available travel video data. The model captures the semantic symbiotic relationships between thousands of environmental objects. Training the environmental recommendation model reveals unique underlying contextual information during sighted users' travel experiences. For example, in a park visit scenario, the embedding position of "duck" is closer to water-related categories such as "dock," "pond," "fish," and "bridge," demonstrating the contextual association between "duck" and "pond" in a specific scenario.

[0053] The ambient recommendation model consists of four sub-networks: background network, aesthetic network, novelty network and demand network.

[0054] The background network uses the background attention mechanism to identify background objects in the semantic graph sequence, such as the sky and trees, and calculates the scores of these background objects to obtain a background score. This score reflects the stability and prevalence of the object as a background.

[0055] The aesthetic network uses an aesthetic attention mechanism to identify interesting objects, such as sculptures and fountains, that appear repeatedly within a preset time period in a semantic graph sequence. It then calculates the scores of these interesting objects to generate aesthetic scores, which reflect the aesthetic and interest value of the objects to the user.

[0056] The novelty network uses a novelty attention mechanism to identify novel objects that appear for the first time in the semantic graph sequence compared to the previous frame and calculates the scores of these novel objects to obtain a novelty score, which reflects the novelty and uniqueness of the object.

[0057] The demand network uses a demand attention mechanism to identify physiological and safety-related demand objects for visually impaired users in the semantic graph sequence, such as drinking water, using the toilet, blind paths, and handrails. It then calculates the scores of these demand objects to obtain a demand score, which reflects the importance and urgency of the object to the user.

[0058] The environmental recommendation model combines the scores of these four sub-networks to generate a comprehensive interest score, which is used to rank and recommend environmental objects. User terminal device 3 recommends environmental objects to the visually impaired user based on the recommendation list. The user can perceive environmental information through multimodal interaction, including audio guidance and tactile feedback. The tactile feedback is used to update the environmental recommendation model and achieve personalized recommendations.

[0059] As a preferred embodiment, the environmental recommendation model also includes a user interaction adapter; the user interaction adapter is used to dynamically adjust the weights of the background score, aesthetic score, freshness score and demand score based on tactile level feedback using a maximum likelihood estimation algorithm.

[0060] In this embodiment, the user interaction adapter is a key component of the environmental recommendation model. It is responsible for learning user preferences through continuous interaction and dynamically adjusting the weights of environmental object scores based on the user's tactile hierarchical feedback. Specifically, the user interaction adapter uses the Maximum Likelihood Estimation (MLE) algorithm to dynamically adjust the weights of background scores, aesthetic scores, novelty scores, and demand scores. The tactile feedback of visually impaired users is recorded and used as input to the MLE algorithm. For example, if a user touches an environmental object on the exploration interface for a longer time or multiple times, this may indicate that the object has a higher aesthetic or novelty to the user.

[0061] The overall interest score is: , in, is the weight of the aesthetic score, The weight of the freshness score, The weights for scoring the requirements, The weight with which to score the background.

[0062] The user interaction adapter adjusts the weight of the corresponding score based on these feedbacks (like / dislike for each object). For example, if the user shows more interest in the novelty object, the user interaction adapter will increase the weight of the novelty score. , and may reduce the weight of the background score . This dynamic adjustment ensures that the recommendation list can better reflect the user's personal preferences and needs. Objects with the highest comprehensive interest scores (i.e., higher aesthetic scores, novelty scores, demand scores, and lower background scores) will be given higher total scores, thereby obtaining higher priority and ranking in the recommendation list. This MLE iteration enables the tour plan to be continuously updated during the testing phase, and the weight parameters reflecting the user's personalized interests are optimized in real time. This process can be regarded as the user implicitly selecting important parameters that define the priority of the scene graph nodes, while the environmental recommendation model attempts to estimate the true value of these weights.

[0063] In this way, a personalized recommendation list is generated for each user, where the order of environmental objects reflects the user's preferences for different environmental features. This not only improves user satisfaction but also enhances their interactive experience with the environment. The introduction of a user interaction adapter adaptively learns and predicts user behavior patterns, providing more precise and personalized services. This dynamic adjustment mechanism enables the model to continuously optimize over time to adapt to users' evolving needs and preferences.

[0064] The multimodal interactive perception device for dynamic panoramic environment information provided by the present invention is described below. The multimodal interactive perception device for dynamic panoramic environment information described below and the multimodal interactive perception method for dynamic panoramic environment information described above can be referred to in correspondence with each other.

[0065] Please refer to Figure 2 , Figure 2 This is a structural schematic diagram of a multimodal interactive perception device for dynamic panoramic environmental information provided by the present invention.

[0066] The present invention also provides a multimodal interactive perception device for dynamic panoramic environmental information, which is applied to a user terminal device 3, and the device includes: a recommendation list acquisition module 201, which is used to obtain a recommendation list for environmental object perception; an environmental object recommendation module 202, which is used to recommend environmental objects to visually impaired users through multimodal interaction based on the recommendation list, so as to assist visually impaired users in environmental perception from multiple dimensions; multimodal interaction includes audio guidance and tactile hierarchical feedback; the multidimensionality includes exploration, understanding, recall and social interaction dimensions; wherein, the recommendation list is a list of several environmental objects sorted by comprehensive interest scores generated by the environmental recommendation model based on the dynamic panoramic environmental video; the comprehensive interest score is a score that comprehensively considers the aesthetics, novelty and demand factors of the visually impaired user; the environmental recommendation model is used to iteratively update the associated weights of aesthetics, novelty and demand factors based on tactile hierarchical feedback, so as to iteratively update the comprehensive interest score and the recommendation list.

[0067] The multimodal interactive perception system for dynamic panoramic environment information provided by the present invention is described below. The multimodal interactive perception system for dynamic panoramic environment information described below and the multimodal interactive perception method for dynamic panoramic environment information described above can refer to each other.

[0068] Please refer to Figure 3 , Figure 3 A schematic diagram of the structure of a multimodal interactive perception system for dynamic panoramic environmental information provided by the present invention.

[0069] The present invention also provides a multimodal interactive perception system for dynamic panoramic environmental information, comprising: an acquisition device 1, for capturing dynamic panoramic environmental video and transmitting the dynamic panoramic environmental video to a server 2; a server 2, for running an environmental recommendation model based on the dynamic panoramic environmental video to generate a recommendation list; the recommendation list is a list of several environmental objects sorted by comprehensive interest scores; the comprehensive interest score is a score that comprehensively considers the aesthetics, novelty, and demand factors of visually impaired users; the environmental recommendation model is used to iteratively update the associated weights of aesthetics, novelty, and demand factors based on tactile hierarchical feedback to iteratively update the comprehensive interest score and the recommendation list; a user terminal device 3, for executing the above-mentioned multimodal interactive perception method for dynamic panoramic environmental information.

[0070] The system of the present invention can simulate the perspective selection mechanism when humans accompany visually impaired people on a guided tour, breaking down environmental perception needs into three core dimensions: Aesthetic value perception: Capturing elements with aesthetic significance in unfamiliar scenes to help users establish an understanding of the characteristics of the environment.

[0071] Dynamic novelty capture: Identify novel information in the scene caused by temporal and spatial changes (such as temporary performances, special installations, etc.), and enhance the user's perception of the dynamic environment.

[0072] Demand warning signals: Real-time detection of warning information related to basic needs (such as drinking water points, barrier-free facilities, obstacles, etc.) to ensure user safety.

[0073] This paper uses a self-supervised graph-mask semantic graph training scheme. This approach leverages only the intrinsic structural information in unlabeled data. Through a masking mechanism, the algorithm automatically learns how to parse scene semantic relationships, eliminating the need for manual annotation. This training approach not only reduces data acquisition costs but also enables the algorithm to generalize more effectively across environments, providing a continuously optimized interactive experience for visually impaired users.

[0074] Figure 4 An example of a physical structure diagram of an electronic device is shown below. Figure 4 As shown, the electronic device may include: a processor (processor) 401 , a communication interface (Communications Interface) 402 , a memory (memory) 403 and a communication bus 404 , wherein the processor 401 , the communication interface 402 , and the memory 403 communicate with each other via the communication bus 404 . The processor 401 can call the logic instructions in the memory 403 to execute a multimodal interactive perception method of dynamic panoramic environmental information, which is applied to the user terminal device 3. The method includes: obtaining a recommendation list for environmental object perception; based on the recommendation list, recommending environmental objects to the visually impaired user through multimodal interaction to assist the visually impaired user in environmental perception from multiple dimensions; multimodal interaction includes audio guidance and tactile hierarchical feedback; the multidimensionality includes exploration, understanding, recall and social interaction dimensions; wherein the recommendation list is a list of several environmental objects sorted by comprehensive interest scores generated by the environmental recommendation model based on the dynamic panoramic environmental video; the comprehensive interest score is a score that comprehensively considers the aesthetics, novelty and demand factors of the visually impaired user; the environmental recommendation model is used to iteratively update the associated weights of aesthetics, novelty and demand factors based on tactile hierarchical feedback to iteratively update the comprehensive interest score and the recommendation list.

[0075] Furthermore, the logic instructions in the aforementioned memory 403 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes several instructions for causing a computer device (which can be a personal computer, server 2, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0076] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multimodal interactive perception method of dynamic panoramic environmental information provided by the above methods, which is applied to the user terminal device 3. The method includes: obtaining a recommendation list for environmental object perception; based on the recommendation list, recommending environmental objects to visually impaired users through multimodal interaction to assist visually impaired users in environmental perception from multiple dimensions; multimodal interaction includes audio guidance and tactile hierarchical feedback; multidimensionality includes exploration, understanding, recall and social interaction dimensions; wherein the recommendation list is a list of several environmental objects sorted by comprehensive interest scores generated by the environmental recommendation model based on the dynamic panoramic environmental video; the comprehensive interest score is a score that comprehensively considers the aesthetics, novelty and demand factors of the visually impaired user; the environmental recommendation model is used to iteratively update the associated weights of aesthetics, novelty and demand factors based on tactile hierarchical feedback to iteratively update the comprehensive interest score and the recommendation list.

[0077] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a multimodal interactive perception method for dynamic panoramic environmental information provided by the above-mentioned methods, which is applied to a user terminal device 3, and the method includes: obtaining a recommendation list for environmental object perception; based on the recommendation list, recommending environmental objects to visually impaired users through multimodal interaction to assist visually impaired users in environmental perception from multiple dimensions; multimodal interaction includes audio guidance and tactile hierarchical feedback; the multidimensionality includes exploration, understanding, recall and social interaction dimensions; wherein the recommendation list is a list of several environmental objects sorted by comprehensive interest scores generated by the environmental recommendation model based on the dynamic panoramic environmental video; the comprehensive interest score is a score of the aesthetics, novelty and demand factors of the visually impaired user; the environmental recommendation model is used to iteratively update the associated weights of aesthetics, novelty and demand factors based on tactile hierarchical feedback to iteratively update the comprehensive interest score and the recommendation list.

[0078] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0079] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A multimodal interactive perception method for dynamic panoramic environmental information, characterized in that: Applied to a user terminal device, the method includes: Get a list of recommendations based on object awareness. Based on the recommendation list, recommend environmental objects to the visually impaired user through multimodal interaction to assist the visually impaired user in environmental perception from multiple dimensions; the multimodal interaction includes audio guidance and tactile feedback; the multiple dimensions include exploration, understanding, recall, and social interaction; Among them, the recommendation list is a list of several environmental objects sorted by comprehensive interest scores generated by the environmental recommendation model based on the dynamic panoramic environmental video; the comprehensive interest score is a score that comprehensively considers the aesthetics, novelty and demand factors of the visually impaired user; the environmental recommendation model is used to iteratively update the associated weights of aesthetics, novelty and demand factors based on the tactile hierarchical feedback, so as to iteratively update the comprehensive interest score and the recommendation list.

2. The multimodal interactive perception method of dynamic panoramic environment information according to claim 1, characterized in that: When the visually impaired user is exploring the environment, describing information of recommended environmental objects to the visually impaired user through audio based on the recommendation list; the information of the recommended environmental objects includes the name, location and attributes of the recommended environmental objects; Determining an exploration action of the visually impaired user; the exploration action is used to represent preference information of the visually impaired user for the currently recommended environment object; the preference information includes likes and dislikes; The recommendation list is dynamically adjusted based on the exploration action.

3. The multimodal interactive perception method of dynamic panoramic environment information according to claim 1, characterized in that: In the case where the visually impaired user understands the environment, based on the recommendation list, a two-layer hierarchical scene interaction architecture is adopted to form a plurality of the environmental objects in a main layer graph structure and a sub-layer graph structure; The nodes of the main layer graph structure are coarse-grained environment objects, and the edges of the main layer graph structure are the relationships between the coarse-grained objects; the nodes of the sub-layer graph structure are fine-grained environment objects, and the edges of the sub-layer graph structure are the relationships between the fine-grained environment objects; describing information of coarse-grained environmental objects in the main layer graph structure to the visually impaired user through audio based on the recommendation list; Determining an understanding action of the visually impaired user; the understanding action is used to represent the need of the visually impaired user to continue to understand information of finer-grained environmental objects under the currently recommended coarse-grained environmental object; Based on the understanding action, information about the fine-grained environmental objects in the sub-layer graph structure corresponding to the currently recommended coarse-grained environmental object is described to the visually impaired user through audio.

4. The multimodal interactive perception method of dynamic panoramic environment information according to claim 1, characterized in that: In the case where the visually impaired user performs environmental recall, determining a recall action of the visually impaired user; the recall action is used to represent the visually impaired user's recall need for historical environmental objects; Based on the recall action, historical multimodal interactive environment perception information is retrieved, and the information of the historical environment objects is described to the visually impaired user through audio.

5. The multimodal interactive perception method of dynamic panoramic environment information according to claim 1, characterized in that: In the case where the visually impaired user performs environmental social interaction, determining a social interaction action of the visually impaired user; the social interaction action is used to represent a need for the current visually impaired user to share information about his or her own historical environmental objects with another visually impaired user; Based on the social interaction action, historical multimodal interactive environment perception information is retrieved, and the historical multimodal interactive environment perception information is sent to the receiving device of the other visually impaired user, so that the current visually impaired user and the other visually impaired user can share the information of the historical environment object.

6. The multimodal interactive perception method of dynamic panoramic environment information according to any one of claims 1 to 5, characterized in that: Also includes: Converting the dynamic panoramic environment video into a semantic graph sequence through a scene graph generation algorithm, so as to input the semantic graph sequence into the environment recommendation model; The environment recommendation model is based on environment video training samples and is trained using graph mask self-supervised learning and multimodal attention mechanism; The environmental recommendation model includes background network, aesthetic network, novelty network and demand network; The background network is used to identify background objects in the semantic graph sequence through a background attention mechanism, and calculate scores of the background objects to obtain background scores; The aesthetic network is used to identify interesting objects that appear continuously in the semantic graph sequence within a preset time period through an aesthetic attention mechanism, and calculate scores of the interesting objects to obtain aesthetic scores; The freshness network is used to identify freshness objects that appear for the first time in the semantic graph sequence compared to the previous frame through a freshness attention mechanism, and calculate the score of the freshness object to obtain a freshness score; The demand network is used to identify physiological and safety-related demand objects for the visually impaired user in the semantic graph sequence through a demand attention mechanism, and calculate scores of the demand objects to obtain demand scores; The comprehensive interest score is a score determined by comprehensively considering the background score, the aesthetic score, the novelty score, and the demand score.

7. The multimodal interactive perception method of dynamic panoramic environment information according to claim 6, characterized in that: The environmental recommendation model also includes a user interaction adapter; The user interaction adapter is used to dynamically adjust the weights of the background score, the aesthetic score, the freshness score, and the demand score based on the tactile hierarchical feedback using a maximum likelihood estimation algorithm.

8. A multimodal interactive perception device for dynamic panoramic environmental information, characterized in that: Applied to user terminal equipment, the device includes: A recommendation list acquisition module is used to obtain a recommendation list perceived by environmental objects; An environmental object recommendation module is configured to recommend environmental objects to a visually impaired user based on the recommendation list through multimodal interaction, thereby assisting the visually impaired user in environmental perception from multiple dimensions; the multimodal interaction includes audio guidance and tactile feedback; and the multiple dimensions include exploration, understanding, recall, and social interaction. Among them, the recommendation list is a list of several environmental objects sorted by comprehensive interest scores generated by the environmental recommendation model based on the dynamic panoramic environmental video; the comprehensive interest score is a score that comprehensively considers the aesthetics, novelty and demand factors of the visually impaired user; the environmental recommendation model is used to iteratively update the associated weights of aesthetics, novelty and demand factors based on the tactile hierarchical feedback, so as to iteratively update the comprehensive interest score and the recommendation list.

9. A multimodal interactive perception system for dynamic panoramic environmental information, characterized in that: include: An acquisition device is used to capture dynamic panoramic environment video and transmit the dynamic panoramic environment video to a server; The server is configured to generate a recommendation list based on a dynamic panoramic environment video running environment recommendation model; the recommendation list is a list of multiple environment objects sorted by comprehensive interest scores; the comprehensive interest score is a score that integrates aesthetics, novelty, and demand factors of the visually impaired user; the environment recommendation model is configured to iteratively update the associated weights of aesthetics, novelty, and demand factors based on the tactile hierarchical feedback, thereby iteratively updating the comprehensive interest score and the recommendation list; The user terminal device is used to execute the multimodal interactive perception method of dynamic panoramic environment information according to any one of claims 1 to 7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the multimodal interactive perception method of dynamic panoramic environment information as described in any one of claims 1 to 7 is implemented.

Images