A shop description customization generation method for visually impaired users
By using a multimodal information fusion framework and a sequence-to-sequence model to generate concise and clear store descriptions, the problem of visually impaired users having difficulty obtaining information is solved, and efficient, accurate information acquisition and real-time updates are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG UNIV
- Filing Date
- 2025-07-04
- Publication Date
- 2026-07-21
AI Technical Summary
Existing image description technologies generate descriptions that are too lengthy, lack structure, make it difficult to quickly extract key information, and lack concise and clear language, failing to meet the special needs of visually impaired users.
It adopts a multimodal information fusion framework, identifies key store information through convolutional neural networks and attention mechanisms, generates concise and clear natural language descriptions using sequence-to-sequence models, and provides customized store descriptions through voice broadcasting.
The generated descriptions are concise, clear, and structured, enabling quick access to key information and meeting the information acquisition needs of visually impaired users. They also support real-time updates and flexible adjustments, improving information acquisition efficiency.
Smart Images
Figure CN120808016B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence and computer vision technology, and specifically to a method for customizing and generating store descriptions for visually impaired users. Background Technology
[0002] With the rapid development of artificial intelligence technology, computer vision and natural language processing technologies have enabled automatic scene description. Existing image description technologies are mainly based on deep learning models, such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs), which can generate natural language descriptions of image content. However, these technologies have the following problems: the generated descriptions are often too lengthy, containing a large amount of information unnecessary for visually impaired users; the descriptions lack structure, making it difficult to quickly extract key information; the language style is not concise and clear enough, which may cause comprehension difficulties; and there is no optimization for the special needs of visually impaired users. Therefore, in order to address the shortcomings of current related technologies, it is essential to develop a customized method for generating store descriptions for visually impaired users. Summary of the Invention
[0003] This invention aims to overcome the shortcomings of current technologies and, based on the advantages of artificial intelligence and computer vision technologies, provides a method for customizing and generating store descriptions for visually impaired users. The method employs well-known mathematical models and algorithms (such as convolutional neural networks and attention mechanisms), and its improvements mainly lie in the design of the multimodal information fusion framework and application adaptation to accessibility needs.
[0004] To address the aforementioned technical issues, the method for customizing store descriptions for visually impaired users includes the following steps:
[0005] S110 defines the key elements framework for store description, including elements such as store name, business category, main products or services, current promotional activities, and store environment overview;
[0006] S120 uses a multimodal model to identify and extract key information about the store.
[0007] S130 organizes natural language output according to a predetermined structure, using a concise and clear language style;
[0008] S140 generates a customized store description that meets the needs of visually impaired users.
[0009] Preferably, the key element framework for defining the store description described in step S110 specifically includes:
[0010] S1101, Determine the key elements of the store description, including store name, business category, main products or services, current promotional activities, store environment overview, spatial layout information and special facilities description;
[0011] S1102 ensures that the framework can meet the needs of visually impaired users for rapid access to information.
[0012] Preferably, the multimodal model described in step S120 is based on a deep learning architecture and includes:
[0013] Cross-modal attention mechanism: By calculating the image feature vector V∈R d×n With text feature vector Q∈R d×m Similarity matrix Achieve alignment and fusion of image and text information;
[0014] Feature-weighted aggregation: using a learnable parameter α i The weighted average of multi-source information is calculated using the following formula: Where F i This represents the eigenvector of the i-th mode;
[0015] Natural language generation: based on sequence-to-sequence models (such as LSTM or Transformer decoders), using probability distribution P(y t ∣y <t ,F fused Generate target description text, where y t This represents the t-th output word.
[0016] Furthermore, step S120, which involves identifying and extracting key information about the store using a multimodal model, specifically includes:
[0017] S1201 uses computer vision technology to identify store exterior features;
[0018] S1202, Extract store logo and signboard information;
[0019] S1203, identifies promotional posters and event information;
[0020] S1204, Analyze the store's spatial layout;
[0021] S1205, Inspecting special facilities (such as facilities for people with disabilities).
[0022] Preferably, the step S130 of organizing the natural language output according to a predetermined structure specifically includes:
[0023] S1301, classify the extracted information according to the element framework;
[0024] S1302, ranking information by importance using the PageRank algorithm, its mathematical expression is: Where R(V) i ) represents node Vi Importance score, d is the damping coefficient (usually taken as 0.85), N is the total number of nodes, M(V i ) indicates pointing to V i The set of nodes, L(V) j ) represents node V j The number of outgoing edges;
[0025] S1303, filters redundant information;
[0026] S1304, Supplement necessary contextual information;
[0027] S1305 employs a templated language structure, uses concise and clear expressions, ensures that information is presented in order of importance, and controls the length and complexity of descriptions.
[0028] Preferably, step S140, generating a customized store description that meets the needs of visually impaired users, specifically includes:
[0029] S1401, automatically triggers store description generation based on user location information;
[0030] S1402 provides store descriptions to visually impaired users via voice broadcast;
[0031] S1403 supports real-time updates of store information, maintaining the accuracy of descriptions;
[0032] S1404, Adjust the level of information detail according to user needs.
[0033] This invention defines a key element framework for store descriptions, extracts store information using a multimodal model, and organizes natural language output according to a predetermined structure to generate customized descriptions that meet the needs of visually impaired users. The method includes constructing an element framework, training a multimodal model to recognize store features, designing a natural language generation module, and outputting the description via voice broadcast. The generated descriptions employ a concise and clear language style, ensuring that information is structured and ordered by importance, and supporting real-time updates and flexible adjustments. This method can be widely applied in fields such as assisted navigation and intelligent guides, providing visually impaired users with an efficient and accurate way to obtain store information, overcoming the lack of specificity in traditional image description technologies.
[0034] The beneficial effects of this invention are that the multimodal model technology employed possesses powerful information extraction capabilities, accurately identifying and extracting key information from store images while maintaining a high level of accuracy even under complex environmental conditions. Compared to traditional image description methods, this invention significantly improves the efficiency of information acquisition for visually impaired users through a predefined element framework and concise language style. Furthermore, the method supports real-time updates and flexible adjustments, adapting to different user needs and changing scenarios, providing visually impaired users with efficient, accurate, and practical store description services. This invention is widely applicable to fields such as assisted navigation and intelligent navigation. Attached Figure Description
[0035] Figure 1 This is an overall flowchart of the method of the present invention. Detailed Implementation
[0036] The technical solution of the present invention will be further described below with reference to the accompanying drawings.
[0037] This embodiment relates to a method for generating customized store descriptions for visually impaired users. This method utilizes multimodal model technology to accurately extract key information from store images and generate customized descriptions that meet the needs of visually impaired users. The multimodal model is based on well-known deep learning architectures (such as ResNet-50, Transformer, etc.) and achieves key functions through the following mathematical methods:
[0038] Cross-modal attention mechanism: By calculating the image feature vector V∈R d×n With text feature vector Q∈R d×m Similarity matrix Achieve alignment and fusion of image and text information; feature weighted aggregation: using a learnable parameter α i The weighted average of multi-source information is calculated using the following formula: Where F i This represents the eigenvector of the i-th mode;
[0039] Natural language generation: based on sequence-to-sequence models (such as LSTM or Transformer decoders), using probability distribution P(y t ∣y <t ,F fused Generate target description text, where y t This represents the t-th output word.
[0040] Specifically, the method includes:
[0041] S110 defines the key elements framework for store descriptions;
[0042] Specifically, identify the key elements of the store description, including store name, business category, main products or services, current promotions, store environment overview, spatial layout information, and description of special facilities; ensure that the framework can meet the needs of visually impaired users for quick access to information.
[0043] S120 uses a multimodal model to identify and extract key information about the store.
[0044] Specifically, computer vision technology is used to identify store exterior features; extract store logos and signage information; identify promotional posters and event information; analyze store interior layout; and detect special facilities (such as accessible facilities).
[0045] S130 organizes natural language output according to a predetermined structure, using a concise and clear language style;
[0046] Specifically, the extracted information is categorized according to the element framework; the information is ranked by importance; and redundant information is filtered out. The "importance ranking" is achieved using the PageRank algorithm, whose mathematical expression is: Where R(V) i ) represents node V i Importance score, d is the damping coefficient (usually taken as 0.85), N is the total number of nodes, M(V i ) indicates pointing to V i The set of nodes, L(V) j ) represents node V j The number of outgoing edges; supplementing necessary contextual information; adopting a templated language structure, using concise and clear expressions, ensuring that information is presented in order of importance, and controlling the length and complexity of the description.
[0047] S140 generates a customized store description that meets the needs of visually impaired users.
[0048] Specifically, it automatically triggers the generation of store descriptions based on user location information; provides store descriptions to visually impaired users through voice broadcast; supports real-time updates of store information to maintain the accuracy of descriptions; and adjusts the level of detail of information according to user needs.
[0049] The embodiments described in this specification are merely examples of implementations of the inventive concept. The scope of protection of this invention should not be considered as limited to the specific forms stated in the embodiments. The scope of protection of this invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.
Claims
1. A method for customizing and generating store descriptions for visually impaired users, characterized in that, Includes the following steps: S110 defines the key elements framework for store description, including store name, business category, main products or services, current promotions, store environment overview, spatial layout information, and description of special facilities. S120 uses a multimodal model to identify and extract key information about the store. The aforementioned multimodal model is based on a deep learning architecture and includes: Cross-modal attention mechanism: by calculating image feature vectors With text feature vectors Similarity matrix This enables the alignment and fusion of image and text information. Feature-weighted aggregation: using learnable parameters The weighted average of multi-source information is calculated using the following formula: ,in This represents the eigenvector of the i-th mode; Natural Language Generation: Based on sequence-to-sequence models, using probability distributions Generate target description text, where This represents the t-th output word; S130, organize the natural language output according to a predetermined structure, classify the extracted information according to the element framework; rank the information by importance; filter redundant information; specifically including: S1301, classify the extracted information according to the element framework; S1302, ranking information by importance using the PageRank algorithm, its mathematical expression is: ;in Represents a node Importance score, d is the damping coefficient, N is the total number of nodes. Indicates pointing to The set of nodes, Represents a node The number of outgoing edges; S1303, filters redundant information; S1304, Supplementing contextual information; S1305 uses a templated language structure to present information in order of importance and control the length and complexity of descriptions; S140 generates a customized store description that meets the needs of visually impaired users.
2. The method for customizing and generating store descriptions for visually impaired users as described in claim 1, characterized in that, The key element framework for defining the store description described in step S110 specifically includes: S1101, Determine the key elements of the store description; S1102 ensures that the framework can meet the needs of visually impaired users for rapid access to information.
3. The method for customizing and generating store descriptions for visually impaired users as described in claim 1, characterized in that, Step S120, which describes identifying and extracting key information about the store using a multimodal model, specifically includes: S1201 uses computer vision technology to identify store exterior features; S1202, Extract store logo and signboard information; S1203, identifies promotional posters and event information; S1204, Analyze the store's spatial layout; S1205, Inspection of special facilities.
4. The method for customizing and generating store descriptions for visually impaired users as described in claim 1, characterized in that, Step S140, which involves generating a customized store description that meets the needs of visually impaired users, specifically includes: S1401, automatically triggers store description generation based on user location information; S1402 provides store descriptions to visually impaired users via voice broadcast; S1403 supports real-time updates of store information, maintaining the accuracy of descriptions; S1404, Adjust the level of information detail according to user needs.