Visually impaired user-oriented shop description customization generation method

Abstract

A visual impaired user-oriented shop description customization generation method comprises the steps of defining a key element framework of shop description, including elements such as shop names, operation categories, main commodities or services, current promotion activities, in-shop environment general situations and the like; identifying and extracting various key information of the shop through a multi-modal model; natural language output is organized according to a preset structure, and a simple and choking language style is adopted; and generating customized shop description meeting the requirements of the visually impaired user. The description generated by the invention adopts a simple and clear language style, ensures information structuring and sorting according to importance, supports real-time updating and flexible adjustment, and provides an efficient and accurate shop information acquisition mode for visually impaired users.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and computer vision, in particular to a shop description customization generation method for visually impaired users. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, computer vision and natural language processing technology have been able to realize automatic description of the scene. Existing image description technology is mainly based on deep learning models such as convolutional neural network (CNN) and recurrent neural network (RNN), etc., which can generate natural language description of image content. However, these technologies have the following problems: the generated description content is often too long, containing a lot of unnecessary information for visually impaired users; the description lacks structure, making it difficult to quickly obtain key information; the language style is not concise and clear, which may cause understanding difficulties; and there is no optimization for the special needs of visually impaired users. Therefore, in order to solve the deficiencies of the related technology, it is necessary to develop a shop description customization generation method for visually impaired users. SUMMARY

[0003] The present application overcomes the shortcomings of the current technology, based on the advantages of artificial intelligence and computer vision technology, and provides a shop description customization generation method for visually impaired users. The method uses known mathematical models and algorithms (such as convolutional neural network, attention mechanism, etc.), and the improvement mainly lies in the design of the multi-modal information fusion framework and the application adaptation for barrier-free needs.

[0004] In order to solve the above technical problems, the shop description customization generation method for visually impaired users comprises the following steps:

[0005] S110, defining the key element framework of shop description, including shop name, business category, main goods or services, current promotion activities, and shop environment profile;

[0006] S120, identifying and extracting various key information of the shop through a multi-modal model;

[0007] S130, organizing natural language output according to a predetermined structure, and adopting a concise and concise language style;

[0008] S140, generating a customized shop description that meets the needs of visually impaired users.

[0009] Preferably, the step S110 of defining the key element framework of shop description specifically comprises:

[0010] S1101, determining the key elements of shop description, including shop name, business category, main goods or services, current promotion activities, shop environment profile, spatial layout information, and special facility description.

[0011] S1102, ensuring that the framework can meet the needs of visually impaired users for quick access to information.

[0012] Preferably, the multi-modal model of step S120 is based on a deep learning framework, including:

[0013] Cross-modal attention mechanism: by calculating the similarity matrix of image feature vector V∈R d×n and text feature vector Q∈R d×m ; Align and fuse image and text information;

[0014] Feature weighting aggregation: using learnable parameters α i to weight multi-source information, the calculation formula is where F i represents the feature vector of the i-th modality;

[0015] Natural language generation: based on sequence-to-sequence models (such as LSTM or Transformer decoder), generate target description text through probability distribution P(y t | y <t , F fused ), where y t represents the t-th output word.

[0016] Further, the multi-modal model of step S120 identifies and extracts various key information of the store, specifically including:

[0017] S1201, identifying store appearance features through computer vision technology;

[0018] S1202, extracting store identification and sign information;

[0019] S1203, identifying promotional posters and activity information;

[0020] S1204, analyzing store space layout;

[0021] S1205, detecting special facilities (such as barrier-free facilities).

[0022] Preferably, the natural language output of step S130 is organized according to a predetermined structure, specifically including:

[0023] S1301, classifying the extracted information according to the element framework;

[0024] S1302, sorting information by importance, achieved by PageRank algorithm, the mathematical expression is: where R(V i ) represents the node Vi importance score, d is damping coefficient (usually 0.85), N is total number of nodes, M(V i ) represents node set pointing to V i , L(V j ) represents the number of outgoing edges of node V j ;

[0025] S1303, filtering redundant information;

[0026] S1304, supplementing necessary context information;

[0027] S1305, using templated language structure, using simple and clear expression, ensuring information in order of importance, controlling description length and complexity.

[0028] Preferably, the generation of customized store description meeting the needs of visually impaired users in step S140 specifically includes:

[0029] S1401, automatically triggering store description generation according to user location information;

[0030] S1402, providing store description to visually impaired users through voice broadcast;

[0031] S1403, supporting real-time updating of store information to maintain description accuracy;

[0032] S1404, adjusting information detail level according to user needs.

[0033] The present application defines the key element framework of store description, extracts store information using multi-modal model, organizes natural language output according to predetermined structure, and generates customized description meeting the needs of visually impaired users. The method includes constructing the element framework, training the multi-modal model to identify store features, designing the natural language generation module and outputting the description through voice broadcast. The generated description adopts simple and clear language style, ensures structured information and importance sorting, supports real-time updating and flexible adjustment. The present application can be widely applied in the fields of auxiliary navigation and intelligent guide, providing efficient and accurate store information acquisition method for visually impaired users, and making up for the lack of targeted defects in traditional image description technology.

[0034] The beneficial effects of the present application are that the multi-modal model technology adopted by the present application has strong information extraction capability, can accurately identify and extract key information from store images, and can maintain high accuracy even in complex environmental conditions; compared with traditional image description methods, the present application greatly improves the efficiency of information acquisition of visually impaired users through a pre-defined element framework and simple language style. In addition, the method of the present application supports real-time updating and flexible adjustment, can adapt to different user needs and scene changes, and provides efficient, accurate and practical store description services for visually impaired users. The method of the present application is widely applicable to the fields of auxiliary navigation and intelligent tour guide. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 The overall flowchart of the method of the present application. DETAILED DESCRIPTION

[0036] The technical solutions of the present application will be further described below in combination with the drawings.

[0037] The present embodiment relates to a store description customization generation method for visually impaired users, which utilizes multi-modal model technology to accurately extract key information from store images and generate customized descriptions that meet the needs of visually impaired users. The multi-modal model is based on known deep learning architectures (such as ResNet-50, Transformer, etc.) and implements key functions through the following mathematical methods:

[0038] Cross-modal attention mechanism: by calculating the similarity matrix of image feature vector V d×n and text feature vector Q d×m to realize the alignment and fusion of image and text information; feature weighting aggregation: using learnable parameters i to weight the multi-source information, the calculation formula is where F i represents the feature vector of the i-th modality;

[0039] Natural language generation: based on sequence-to-sequence models (such as LSTM or Transformer decoder), the target description text is generated through the probability distribution P(y t | y <t , F fused ), where y t represents the t-th output word.

[0040] Specifically, the method comprises:

[0041] S110, defining a key element framework of store description;

[0042] ​Specifically, the key elements of the store description are determined, including the store name, business category, main goods or services, current promotional activities, store environment profile, space layout information, and special facility description; and the framework is ensured to meet the needs of visually impaired users for quick access to information.

[0043] S120, identifying and extracting various key information of the store through a multi-modal model;

[0044] Specifically, the store appearance features are identified through computer vision technology; the store logo and sign information are extracted; promotional posters and activity information are identified; the store space layout is analyzed; and special facilities (such as barrier-free facilities) are detected.

[0045] S130, organizing natural language output according to a predetermined structure and adopting a concise and concise language style;

[0046] Specifically, the extracted information is classified according to the element framework; the information is sorted by importance; and redundant information is filtered. The "importance sorting" is achieved through the PageRank algorithm, which is mathematically expressed as: where R(V i ) represents the importance score of node V i , d is the damping coefficient (usually 0.85), N is the total number of nodes, M(V i ) represents the set of nodes pointing to V i , and L(V j ) represents the number of outgoing edges of node V j ; supplement necessary context information; adopt a templated language structure, use a concise expression, ensure that the information is presented in order of importance, and control the length and complexity of the description.

[0047] S140, generating a customized store description that meets the needs of visually impaired users.

[0048] Specifically, the store description generation is automatically triggered according to user location information; the store description is provided to visually impaired users through voice broadcast; real-time updating of store information is supported to maintain the accuracy of the description; and the level of detail of the information is adjusted according to user needs.

[0049] The content described in the embodiments of the present specification is only a list of implementation forms of the inventive concept, and the protection scope of the present application should not be considered as limited to the specific forms stated in the embodiments, and the protection scope of the present application also extends to equivalent technical means that can be thought of by those skilled in the art according to the inventive concept.

Claims

1. A method for customizing and generating store descriptions for visually impaired users, characterized in that: The following steps are involved: S110, define the key elements framework of store description, including store name, business category, main products or services, current promotions, store environment overview, etc.; S120, identifying and extracting key store information through a multimodal model; S130, organizing the natural language output according to a predetermined structure, classifying the extracted information according to the element framework; and ranking the information by importance; Filter out redundant information; S140: Generate a customized store description that meets the needs of visually impaired users.

2. The method for customizing and generating store descriptions for visually impaired users according to claim 1, characterized in that: The key elements framework for defining the store description in step S110 specifically includes: S1101, determine the key elements of the store description, including the store name, business category, main products or services, current promotions, store environment overview, space layout information and special facilities description; S1102, ensure that the framework can meet the needs of visually impaired users for rapid access to information.

3. The method for customizing and generating store descriptions for visually impaired users according to claim 1, wherein: The multimodal model described in step S120 is based on a deep learning framework and includes: Cross-modal attention mechanism: by calculating the image feature vector V∈R d×n With the text feature vector Q∈R d×m The similarity matrix Achieve alignment and fusion of image and text information; Feature weighted aggregation: using learnable parameter α i Weighting multi-source information, the calculation formula is: Among them F i represents the eigenvector of the i-th mode; Natural language generation: Based on the sequence-to-sequence model, through the probability distribution P(y t ∣y <t ,F fused ) Generate target description text, where y t represents the t-th output word.

4. The method for customizing and generating store descriptions for visually impaired users according to claim 3, wherein: The identification and extraction of key store information using the multimodal model in step S120 specifically includes: S1201, identifying store appearance features using computer vision technology; S1202, extracting store logo and sign information; S1203, identifying promotional posters and event information; S1204, analyze the store space layout; S1205, testing special facilities.

5. The method for customizing and generating store descriptions for visually impaired users according to claim 1, wherein: Organizing the natural language output according to the predetermined structure in step S130 specifically includes: S1301, classifying the extracted information according to the element framework; S1302, sorting the importance of information is achieved through the PageRank algorithm, which is mathematically expressed as: Where R(V i ) represents node V i The importance score of , d is the damping coefficient, N is the total number of nodes, M(V i ) indicates pointing to V i The node set, L(V j ) represents node V j The number of outgoing edges; S1303, filtering redundant information; S1304, supplement necessary context information; S1305 adopts a templated language structure and uses concise and clear expressions to ensure that information is presented in order of importance and to control the length and complexity of the description.

6. The method for customizing and generating store descriptions for visually impaired users according to claim 1, wherein: The step S140 of generating a customized store description that meets the needs of visually impaired users specifically includes: S1401, automatically triggering store description generation based on user location information; S1402, providing a store description to the visually impaired user through voice broadcast; S1403, supports real-time update of store information to maintain description accuracy; S1404: Adjust the level of information detail according to user needs.

Images