Barrier-free intelligent service method and device based on multi-modal fusion, and electronic equipment

Abstract

The invention discloses a barrier-free intelligent service scheme based on multi-modal fusion, and belongs to the technical field of intelligent equipment, and the method comprises the steps: determining a service type selected by a user according to the first input of the user, and switching to a corresponding service interface; determining a target service request selected by the user in the service interface according to second input of the user; searching target information in a background database according to the target service request; generating first text information and a first braille dot matrix of voice broadcast according to the target information; and calling a visual display module to visually display the target information, calling a voice broadcast module to broadcast the first text information by voice, and displaying the first braille dot matrix on a touch display module. According to the scheme provided by the invention, comprehensive and intelligent information query and in-station navigation service can be provided for special crowds.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent devices, and in particular to a barrier-free intelligent service method and device, and electronic equipment based on multimodal fusion. Background Art

[0002] With the intelligent development of urban public services, service terminals such as electronic display screens have been gradually equipped in large public scenes such as passenger terminal halls at transportation hubs. Although these service terminals can meet the needs of most users, they still have great limitations for special groups such as the blind, the deaf, and the elderly. For example, the blind cannot rely on visual perception of information, the deaf cannot use auditory interaction, and the elderly often encounter difficulties when using service terminals to interact with them due to the decline of hearing and vision.

[0003] Therefore, there is an urgent need for technical personnel in this field to provide an accessible intelligent service terminal to provide special groups with a more intelligent and comprehensive accessible service experience when they are waiting for the train in the station hall. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a barrier-free intelligent service method and device, and electronic device based on multimodal fusion, which can solve the problem in the prior art that service terminals cannot provide barrier-free services to special groups of people.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0006] An embodiment of the present invention provides a barrier-free intelligent service method based on multimodal fusion, wherein the method includes:

[0007] Determine the service type selected by the user according to the first input of the user, and switch to the corresponding service interface; wherein the service type includes: information query service, site navigation service;

[0008] Determining the target service request selected by the user in the service interface according to the second input of the user;

[0009] Searching for target information in a background database according to the target service request;

[0010] Generating first text information and a first Braille dot matrix for voice broadcast according to the target information;

[0011] The visual display module is called to visually display the target information, the voice broadcast module is called to voice broadcast the first text information, and the first Braille dot matrix is ​​displayed on the tactile display module.

[0012] Optionally, when the service type is an information query service, the step of generating the first text information and the first Braille dot matrix for voice broadcast according to the target information includes:

[0013] In the case where the service type is an information query service, generating first text information and second text information according to the target information, wherein the first text information includes detailed content of the target information, and the second text information includes a summary content of the target information;

[0014] The second text information is converted into a first Braille dot matrix.

[0015] Optionally, when the service type is a station navigation service, the step of generating the first text information and the first Braille dot matrix for voice broadcast according to the target information includes:

[0016] In the case where the service type is an in-site navigation service, a preset multimodal comparison learning model is used to search for a target text description template with the highest matching degree with the target information from preset text description templates, wherein the target information is image information;

[0017] Determining dynamic filling information of the target text description template through real-time environmental perception information;

[0018] Filling the dynamic filling information into the target text description template to generate first text information;

[0019] The first text information is converted into a first Braille dot matrix.

[0020] Optionally, when the service type is a site navigation service, the step of using a preset multimodal comparison learning model to search for a target text description template having the highest matching degree with the target information from preset text description templates includes:

[0021] In the case where the service type is an in-site navigation service, for each preset text description template, the target information is parsed by an image encoder included in the preset multimodal comparison learning model to obtain a first parsing result;

[0022] Parsing the preset text description template by means of a text encoder included in the preset multimodal comparison learning model to obtain a second parsing result;

[0023] Calculating the similarity between the first parsing result and the second parsing result;

[0024] The preset text description template corresponding to the highest similarity is determined as the target text description template with the highest matching degree with the target information.

[0025] Optionally, the step of determining the service type selected by the user according to the first input of the user and switching to the corresponding service interface includes:

[0026] Outputting the service type to the user in a preset combination manner, wherein the preset combination manner includes a combination of at least two of the following: visually displaying the service type selection control, voice broadcasting the service type selection control, and displaying a second Braille dot matrix corresponding to the service type selection control through a tactile display module;

[0027] Receive a first input from a user, wherein the first input includes at least one of the following: a touch operation on a visually displayed service type selection control, a voice input of a selected service type, and an operation of inputting a service type selection in Braille through a Braille keyboard;

[0028] The service type selected by the user is determined according to the user's first input, and the corresponding service interface is switched.

[0029] The embodiment of the present invention further provides a barrier-free intelligent service device based on multimodal fusion, wherein the device comprises:

[0030] A first determination module is used to determine the service type selected by the user according to the first input of the user, and switch to the corresponding service interface; wherein the service type includes: information query service, site navigation service;

[0031] A second determination module, used to determine the target service request selected by the user in the service interface according to the second input of the user;

[0032] A search module, used to search for target information in a background database according to the target service request;

[0033] A generating module, used for generating first text information and a first Braille dot matrix for voice broadcast according to the target information;

[0034] The output module is used to call the visual display module to visually display the target information, call the voice broadcast module to voice broadcast the first text information, and display the first Braille dot matrix on the tactile display module.

[0035] Optionally, the generating module includes:

[0036] A first submodule is used to generate first text information and second text information according to the target information when the service type is an information query service, wherein the first text information includes detailed content of the target information and the second text information includes a summary content of the target information;

[0037] The second submodule is used to convert the second text information into a first Braille dot matrix.

[0038] Optionally, the generating module includes:

[0039] A third submodule is used for, when the service type is an in-site navigation service, using a preset multimodal comparison learning model to search for a target text description template with the highest matching degree with the target information from preset text description templates, wherein the target information is image information;

[0040] A fourth submodule is used to determine dynamic filling information of the target text description template through real-time environmental perception information;

[0041] A fifth submodule, used for filling the dynamic filling information into the target text description template to generate first text information;

[0042] The sixth submodule is used to convert the first text information into a first Braille dot matrix.

[0043] Optionally, the third submodule includes:

[0044] The first unit is configured to parse the target information by using an image encoder included in the preset multimodal comparison learning model for each preset text description template when the service type is an in-site navigation service, so as to obtain a first parsing result;

[0045] The second unit is used to parse the preset text description template through the text encoder included in the preset multimodal comparison learning model to obtain a second parsing result;

[0046] A third unit, configured to calculate the similarity between the first parsing result and the second parsing result;

[0047] The fourth unit is used to determine the preset text description template corresponding to the highest similarity as the target text description template with the highest matching degree with the target information.

[0048] Optionally, the first determining module includes:

[0049] A seventh submodule is used to output the service type to the user in a preset combination mode, wherein the preset combination mode includes a combination of at least two of the following: visually displaying the service type selection control, voice broadcasting the service type selection control, and displaying a second Braille dot matrix corresponding to the service type selection control through a tactile display module;

[0050] An eighth submodule is configured to receive a first input from a user, wherein the first input includes at least one of the following: a touch operation on a visually displayed service type selection control, a voice input of a selected service type, and an operation of selecting a service type in Braille by inputting a Braille keyboard;

[0051] The ninth submodule is used to determine the service type selected by the user according to the first input of the user, and switch to the corresponding service interface.

[0052] An embodiment of the present invention provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of any one of the above-mentioned barrier-free intelligent service methods based on multimodal fusion.

[0053] An embodiment of the present invention provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of any one of the above-mentioned barrier-free intelligent service methods based on multimodal fusion are implemented.

[0054] The barrier-free intelligent service solution based on multimodal fusion provided by the embodiment of the present invention determines the service type selected by the user according to the first input of the user, and switches to the corresponding service interface; determines the target service request selected by the user in the service interface according to the second input of the user; searches for target information in the background database according to the target service request; generates the first text information and the first Braille dot matrix for voice broadcast according to the target information; calls the visual display module to visually display the target information, calls the voice broadcast module to voice broadcast the first text information, and displays the first Braille dot matrix on the tactile display module. The barrier-free intelligent service solution based on multimodal fusion provided by the embodiment of the present invention is based on the audio-visual and tactile multimodal fusion technology, and can provide comprehensive and intelligent information query and in-site navigation services for special groups such as the visually impaired, the hearing impaired and the elderly. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 is a flowchart showing a method for providing barrier-free intelligent services based on multimodal fusion according to an embodiment of the present application;

[0056] Figure 2 is a structural block diagram of a barrier-free intelligent service terminal based on multimodal fusion according to an embodiment of the present application;

[0057] Figure 3 is a schematic diagram showing an interactive module included in the intelligent service terminal of an embodiment of the present application;

[0058] Figure 4 It is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0060] The barrier-free intelligent service solution based on multimodal fusion provided by the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0061] As attached Figure 1 As shown, the barrier-free intelligent service method based on multimodal fusion in the embodiment of the present application includes the following steps:

[0062] Step 101: Determine the service type selected by the user according to the user's first input, and switch to the corresponding service interface.

[0063] Among them, service types include but are not limited to: information query services and site navigation services.

[0064] The barrier-free intelligent service method based on multimodal fusion provided in the embodiment of the present application is applied to an intelligent service terminal provided in a public place. A computer program is stored on the intelligent service terminal. When the computer program is executed by the processor of the intelligent service terminal, the barrier-free intelligent service method based on multimodal fusion described in the present application is executed. Figure 2 This is a block diagram of an exemplary barrier-free intelligent service terminal based on multimodal fusion. The services that can be provided are divided into two types: information query service and station navigation service. The information query service further includes: train information, ticket gate information, loss information, train delay information and station announcement information, etc. The station navigation service further includes: station facility guidance, transfer guidance and station traffic reminder, etc.

[0065] An optional method of determining the service type selected by the user according to the user's first input and switching to the corresponding service interface may include the following sub-steps:

[0066] Sub-step 1: Output the service type to the user using a preset combination method.

[0067] The preset combination includes at least two of the following combinations: visual display of the service type selection control, voice broadcast of the service type selection control, and display of the second Braille dot matrix corresponding to the service type selection control through the tactile display module. This combination ensures that all types of special groups can obtain service type selection information, making it easier for users to select service types.

[0068] Sub-step 2: Receive a first input from the user.

[0069] The first input includes at least one of the following: a touch operation on a visually displayed service type selection control, a voice input of a selected service type, and an operation of inputting a service type selection in Braille through a Braille keyboard.

[0070] Sub-step 3: Determine the service type selected by the user according to the user's first input, and switch to the corresponding service interface.

[0071] In the actual implementation process, when the user selects each service type and its sub-services, the visual display module will first display a series of clearly visible service options in the form of icons and text, which is convenient for users to quickly identify. At the same time, the voice broadcast module will synchronously broadcast the detailed information of each option, and the tactile display module will generate a Braille dot matrix corresponding to the service name. When the user inputs the service selection, three types of input are provided: touch screen input, voice input and tactile input. Among them, during voice input, voice recognition technology is used to identify the query content and service options spoken by the user, and the intelligent service terminal will provide voice feedback after each user input. Tactile input uses the Braille keyboard equipped with the terminal. Visually impaired people can input by touching the keyboard. At the same time, when the user is entering information, the intelligent service terminal will use slight vibrations to prompt whether the input is valid to help the user confirm the input.

[0072] Step 102: Determine the target service request selected by the user in the service interface according to the second input of the user.

[0073] The second input may be in the same form as the first input and may include at least one of the following: a touch operation on a visually displayed sub-service control, a sub-service selected by voice input, and a sub-service operation selected by inputting a Braille keyboard.

[0074] For special groups of people with visual impairments, the selected sub-service can be input through a Braille keyboard or through voice input; for special groups of people with hearing impairments, the sub-service can be selected by touching the visually displayed sub-service controls.

[0075] Step 103: Searching for target information in a background database according to the target service request.

[0076] Step 104: Generate first text information and a first Braille dot matrix for voice broadcast according to the target information.

[0077] If the service type selected by the user through the first input is different, the method of generating the first text information and the first Braille dot matrix for voice broadcast based on the found target information is different. The system is preset with the first text information and the first Braille dot matrix generation strategy corresponding to these two different service types. When the service type is information query service, the found target information is usually text information; when the service type is site navigation service, the found target information is usually an image or a map.

[0078] In an optional embodiment, when the service type is an information query service, the method of generating the first text information and the first Braille dot matrix for voice broadcast according to the target information may be as follows:

[0079] When the service type is information query service, first text information and second text information are generated according to the target information, wherein the first text information includes detailed content of the target information and the second text information includes summary content of the target information; and the second text information is converted into a first Braille dot matrix.

[0080] The second text information is a simplified summary text information of the complex and detailed target information (also called service information). This type of information is converted into Braille ASCII code by the Braille conversion module with reference to the Braille ASCII code comparison table, and then generates Braille dot matrix. Finally, the Braille dot matrix will be published on the tactile display module.

[0081] In an optional embodiment, when the service type is a station navigation service, the method of generating the first text information and the first Braille dot matrix for voice broadcast according to the target information may be as follows:

[0082] When the service type is an in-site navigation service, a preset multimodal comparison learning model is used to search for a target text description template with the highest matching degree with the target information from preset text description templates, wherein the target information is image information; dynamic fill information of the target text description template is determined through real-time environmental perception information; the dynamic fill information is filled into the target text description template to generate a first text information; and the first text information is converted into a first Braille dot matrix.

[0083] The first text information is converted into a Braille ASCII code by a Braille conversion module with reference to a Braille ASCII code comparison table, and then a first Braille dot matrix is ​​generated.

[0084] This method of optionally converting image and map data into text information and Braille dot matrices can convert intuitive information viewed through both eyes into Braille dot matrices, making it easier for special people with visual impairments to obtain navigation information contained in images or maps.

[0085] More specifically, the method of using a preset multimodal comparison learning model to search for a target text description template with the highest matching degree with the target information from preset text description templates may include the following sub-steps:

[0086] Sub-step 1: when the service type is a site navigation service, for each preset text description template, the target information is parsed by using an image encoder included in a preset multimodal comparison learning model to obtain a first parsing result;

[0087] Sub-step 2: parsing the preset text description template by using a text encoder included in a preset multimodal comparison learning model to obtain a second parsing result;

[0088] This step is to parse the text information embedded in the preset text description template.

[0089] The preset multimodal contrastive learning model trains an image encoder and a text encoder through a contrastive learning method so that the semantic distance of matching image-text pairs is as close as possible, while the distance of mismatched pairs is as far as possible. The image encoder uses a convolutional neural network to extract the features of the image and outputs a vector representation of a fixed dimension. The text encoder converts the text into a feature vector that shares the same dimension with the image vector. For a batch of image-text pairs, the goal learned by the model is to minimize the distance between the vector of the image and its corresponding text, while maximizing the distance between the image and other unrelated texts. In this way, the preset multimodal contrastive learning model can learn to align images and texts in the same semantic space. By matching the target information and the preset text description templates through the preset multimodal contrastive learning model, the similarity corresponding to each preset text description template can be determined respectively.

[0090] Sub-step 3: Calculate the similarity between the first parsing result and the second parsing result.

[0091] Sub-step 4: Determine the preset text description template corresponding to the highest similarity as the target text description template with the highest matching degree with the target information.

[0092] Step 105: calling the visual display module to visually display the target information, calling the voice broadcast module to voice broadcast the first text information, and displaying the first Braille dot matrix on the tactile display module.

[0093] Figure 3 It is a schematic diagram of an interactive module included in an intelligent service terminal. In this step, the information requested by the service request input by the user is output through the three-modal fusion method of the visual display module, the voice broadcast module and the tactile display module, which can provide comprehensive and intelligent information query and in-station navigation services for special groups. The barrier-free intelligent service solution based on multimodal fusion provided by the embodiment of the present invention determines the service type selected by the user according to the first input of the user, and switches to the corresponding service interface; determines the target service request selected by the user in the service interface according to the second input of the user; searches for target information in the background database according to the target service request; generates the first text information and the first Braille dot matrix for voice broadcast according to the target information; calls the visual display module to visualize the target information, calls the voice broadcast module to voice broadcast the first text information, and displays the first Braille dot matrix in the tactile display module. The barrier-free intelligent service method based on multimodal fusion provided by the embodiment of the present invention, based on the audio-visual and tactile multimodal fusion technology, can provide comprehensive and intelligent information query and in-station navigation services for special groups such as visually impaired, hearing-impaired and the elderly.

[0094] Combine the following Figure 2The barrier-free intelligent service solution based on multimodal fusion provided in the embodiment of the present application is explained with a specific example.

[0095] The embodiment of the present application provides an audio-visual (i.e., multi-modal) integrated barrier-free intelligent service terminal, which can be used in public places to provide services to users.

[0096] The service terminal can release service information to users through vision, hearing and touch simultaneously. Figure 2 This is a block diagram of the barrier-free intelligent service terminal based on multimodal fusion. The service terminal system mainly includes two parts: information query service (i.e. the first type of service below) and station navigation service (i.e. the second type of service below). Among them, the information query service includes train information, ticket gate information, loss information, train delay information and station announcement information. The station navigation service includes station facility guidance, transfer guidance and station traffic reminder.

[0097] In the first type of service, the system ultimately provides users with information mainly based on text data content. Its main workflow is as follows: First, the user selects the service information (i.e., sub-service) that he wants to query through the human-computer interaction terminal. Then, after the system obtains the service request, it will search the service information in the background database according to the service requested by the user to obtain the target information, which is text data information. Subsequently, the queried target information in the database is published. When the target information is published, it is mainly published to three modules, namely, the visual display module, the voice broadcast module, and the tactile display module. First, the queried target information is published to the visual display module in the form of a tabular picture, and then the target information is generated by the AI ​​(Artificial Intelligence) large model to generate two text descriptions. One is the first text information that describes the service content in detail, which is sent to the voice broadcast module for playback; the other is the second text information that simplifies the complex and detailed service information. The second text information is converted into the Braille ASCⅡ code by the Braille conversion module with reference to the Braille ASCⅡ code comparison table, and then the first Braille dot matrix is ​​generated. Finally, the first Braille dot matrix will be published in the tactile display module.

[0098] In detail, when the user selects each service and its sub-services, the visual display module will first display a series of clearly visible service options in the form of icons and text, which is convenient for users to quickly identify. At the same time, the voice broadcast module will synchronously broadcast the detailed information of each option, and the tactile display module will generate a Braille dot matrix corresponding to the service name. When the user inputs the service selection, three types of input are provided: touch screen input, voice input and tactile input. Among them, voice input uses voice recognition technology to identify the query content and service options spoken by the user, and the device will provide voice feedback after each user input. Tactile input uses the Braille keyboard equipped with the terminal. Visually impaired people can input by touching the keyboard. At the same time, when the user is entering information, the device will prompt whether the input is valid through slight vibration to help the user confirm the input.

[0099] In the second category of services, the in-site navigation service is service information that is highly related to maps, images, etc. This part of information is difficult to publish to the voice broadcast module and the tactile display module through the method used in the first category of services. At the same time, information such as maps and images is very difficult for the visually impaired to understand and perceive. Therefore, this application adopts a visual-text-tactile contrast learning model to perform contrast learning between multiple modalities, so that they can find matching representations in the same embedding space. At the same time, the matched text modal information is converted into Braille ASCⅡ code with reference to the Braille ASCⅡ code comparison table, and then the Braille dot matrix is ​​generated and the tactile modal information is generated.

[0100] Specifically, the multimodal contrastive learning model can train an image encoder and a text encoder through a contrastive learning method so that the semantic distance of matching image-text pairs is as close as possible, while the distance of unmatched pairs is as far as possible. The image encoder uses a convolutional neural network to extract the features of the image and outputs a vector representation of a fixed dimension. The text encoder uses a Transformer architecture to convert the text into a feature vector, sharing the same dimension with the image vector. For a batch of image-text pairs, the goal learned by the model is to minimize the distance between the vector of the image and its corresponding text, while maximizing the distance between the image and other irrelevant texts. In this way, the multimodal contrastive learning model can learn to align images and texts in the same semantic space. For the present invention, the system prepares a series of preset text description templates (such as "the ticket gate is xx meters in the xx direction", "the restroom is xx meters on the xx side, please xx") for common navigation service scenarios in advance. The multimodal contrastive learning model embeds the input image into all preset text description templates for comparison. For each image embedding, the multimodal contrastive learning model calculates the similarity between it and the text information embedded in each text description template, and then selects the text description template with the highest similarity as the matching result. Then, the dynamic filling data in the preset text description template is calculated through real-time environmental perception information to determine the final text description information, and then this text description is played to the user through the voice broadcast module. At the same time, the text description is further converted into Braille dots and finally published to the tactile display module. The schematic diagram of the interactive module included in the intelligent service terminal is shown in the figure below. Figure 3 shown.

[0101] For example, a visually impaired user chooses to navigate to the nearest restroom through an interactive terminal. The visual display module will publish the location information and navigation route from the current location to the nearest restroom. Then, the multimodal contrastive learning model inputs the navigation image information into the image encoder, extracts image features and generates image embedding. The text encoder retrieves a series of text description templates related to restroom navigation, such as "The restroom is located at [direction] [distance] meters, please [action] go". The multimodal contrastive learning model compares the generated image embedding with the preset text description template, calculates the similarity between each image embedding and the text embedding in the preset text description template, and finds the text description template that best matches the current image. The background calculates the dynamic filling data [direction] as the right front, [distance] as 30 meters, and [action] as straight through real-time environmental perception information. The final complete language description is determined as "The restroom is 30 meters in front of the user's right, please go straight", and the final complete language description is published to the voice broadcast and tactile display modules.

[0102] The barrier-free intelligent service terminal based on multimodal fusion provided in this specific example takes into account the interactive needs of people with disabilities, especially the visually impaired, and integrates a tactile module. It can provide services to visually impaired users through Braille keyboard input, and the output method of combining Braille display with voice broadcast. This design can help visually impaired users obtain services such as querying station information and station navigation. In addition, the intelligent service terminal adopts a multimodal contrast learning model to compare and learn complex image information with text descriptions, so that images and texts are embedded in the same semantic space for matching. The model can convert the navigation image information in the station into a concise text description, ensuring that visually impaired people can receive navigation information through voice broadcast or tactile modules.

[0103] Figure 4 A structural block diagram of a barrier-free intelligent service system based on multimodal fusion to implement an embodiment of the present application.

[0104] The barrier-free intelligent service system based on multimodal fusion provided in the embodiment of the present application includes the following functional modules:

[0105] The first determination module 401 is used to determine the service type selected by the user according to the first input of the user, and switch to the corresponding service interface; wherein the service type includes: information query service, site navigation service;

[0106] A second determination module 402, configured to determine the target service request selected by the user in the service interface according to the second input of the user;

[0107] A search module 403 is used to search for target information in a background database according to the target service request;

[0108] A generating module 404, configured to generate first text information and a first Braille dot matrix for voice broadcast according to the target information;

[0109] The output module 405 is used to call the visual display module to visually display the target information, call the voice broadcast module to voice broadcast the first text information, and display the first Braille dot matrix on the tactile display module.

[0110] Optionally, the generating module includes:

[0111] A first submodule is used to generate first text information and second text information according to the target information when the service type is an information query service, wherein the first text information includes detailed content of the target information and the second text information includes a summary content of the target information;

[0112] The second submodule is used to convert the second text information into a first Braille dot matrix.

[0113] Optionally, the generating module includes:

[0114] A third submodule is used for, when the service type is an in-site navigation service, using a preset multimodal comparison learning model to search for a target text description template with the highest matching degree with the target information from preset text description templates, wherein the target information is image information;

[0115] A fourth submodule is used to determine dynamic filling information of the target text description template through real-time environmental perception information;

[0116] A fifth submodule, used for filling the dynamic filling information into the target text description template to generate first text information;

[0117] The sixth submodule is used to convert the first text information into a first Braille dot matrix.

[0118] Optionally, the third submodule includes:

[0119] The first unit is configured to parse the target information by using an image encoder included in the preset multimodal comparison learning model for each preset text description template when the service type is an in-site navigation service, so as to obtain a first parsing result;

[0120] The second unit is used to parse the preset text description template through the text encoder included in the preset multimodal comparison learning model to obtain a second parsing result;

[0121] A third unit, configured to calculate the similarity between the first parsing result and the second parsing result;

[0122] The fourth unit is used to determine the preset text description template corresponding to the highest similarity as the target text description template with the highest matching degree with the target information.

[0123] Optionally, the first determining module includes:

[0124] A seventh submodule is used to output the service type to the user in a preset combination mode, wherein the preset combination mode includes a combination of at least two of the following: visually displaying the service type selection control, voice broadcasting the service type selection control, and displaying a second Braille dot matrix corresponding to the service type selection control through a tactile display module;

[0125] An eighth submodule is configured to receive a first input from a user, wherein the first input includes at least one of the following: a touch operation on a visually displayed service type selection control, a voice input of a selected service type, and an operation of selecting a service type in Braille by inputting a Braille keyboard;

[0126] The ninth submodule is used to determine the service type selected by the user according to the first input of the user, and switch to the corresponding service interface.

[0127] The barrier-free intelligent service device based on multimodal fusion provided in the embodiment of the present application determines the service type selected by the user according to the first input of the user, and switches to the corresponding service interface; determines the target service request selected by the user in the service interface according to the second input of the user; searches for target information in the background database according to the target service request; generates the first text information and the first Braille dot matrix for voice broadcast according to the target information; calls the visual display module to visually display the target information, calls the voice broadcast module to voice broadcast the first text information, and displays the first Braille dot matrix on the tactile display module. The barrier-free intelligent service device based on audio-visual fusion, based on audio-visual and tactile multimodal fusion technology, can provide comprehensive and intelligent information query and in-site navigation services for special groups such as the visually impaired, the hearing impaired and the elderly.

[0128] In the present application embodiment Figure 4 The barrier-free intelligent service device based on multimodal fusion shown may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiments of the present application.

[0129] The embodiments of the present application provide Figure 4 The barrier-free intelligent service device based on multimodal fusion can achieve Figure 1 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0130] Optionally, an embodiment of the present application also provides an electronic device, including a processor, a memory, and a program or instruction stored in the memory and executable on the processor. When the program or instruction is executed by the processor, the various processes performed by the above-mentioned barrier-free intelligent service device based on multimodal fusion are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0131] It should be noted that the electronic device in the embodiment of the present application includes the server described above.

[0132] The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0133] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0134] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A barrier-free intelligent service method based on multimodal fusion, characterized in that: The method comprises: Determine the service type selected by the user according to the first input of the user, and switch to the corresponding service interface; wherein the service type includes: information query service, site navigation service; Determining the target service request selected by the user in the service interface according to the second input of the user; Searching for target information in a background database according to the target service request; Generating first text information and a first Braille dot matrix for voice broadcast according to the target information; The visual display module is called to visually display the target information, the voice broadcast module is called to voice broadcast the first text information, and the first Braille dot matrix is ​​displayed on the tactile display module.

2. The method according to claim 1, characterized in that In the case where the service type is an information query service, the step of generating the first text information and the first Braille dot matrix for voice broadcast according to the target information includes: In the case where the service type is an information query service, generating first text information and second text information according to the target information, wherein the first text information includes detailed content of the target information, and the second text information includes a summary content of the target information; The second text information is converted into a first Braille dot matrix.

3. The method according to claim 1, characterized in that In the case where the service type is an in-station navigation service, the step of generating the first text information and the first Braille dot matrix for voice broadcast according to the target information comprises: In the case where the service type is an in-site navigation service, a preset multimodal comparison learning model is used to search for a target text description template with the highest matching degree with the target information from preset text description templates, wherein the target information is image information; Determining dynamic filling information of the target text description template through real-time environmental perception information; Filling the dynamic filling information into the target text description template to generate first text information; The first text information is converted into a first Braille dot matrix.

4. The method according to claim 3, characterized in that In the case where the service type is an in-site navigation service, the step of using a preset multimodal comparison learning model to search for a target text description template with the highest matching degree with the target information from preset text description templates includes: In the case where the service type is an in-site navigation service, for each preset text description template, the target information is parsed by an image encoder included in the preset multimodal comparison learning model to obtain a first parsing result; Parsing the preset text description template by means of a text encoder included in the preset multimodal comparison learning model to obtain a second parsing result; Calculating the similarity between the first parsing result and the second parsing result; The preset text description template corresponding to the highest similarity is determined as the target text description template with the highest matching degree with the target information.

5. The method according to claim 1, characterized in that The step of determining the service type selected by the user according to the user's first input and switching to the corresponding service interface includes: Outputting the service type to the user in a preset combination manner, wherein the preset combination manner includes a combination of at least two of the following: visually displaying the service type selection control, voice broadcasting the service type selection control, and displaying a second Braille dot matrix corresponding to the service type selection control through a tactile display module; Receive a first input from a user, wherein the first input includes at least one of the following: a touch operation on a visually displayed service type selection control, a voice input of a selected service type, and an operation of inputting a service type selection in Braille through a Braille keyboard; The service type selected by the user is determined according to the user's first input, and the corresponding service interface is switched.

6. A barrier-free intelligent service device based on multimodal fusion, characterized in that: The device comprises: A first determination module is used to determine the service type selected by the user according to the first input of the user, and switch to the corresponding service interface; wherein the service type includes: information query service, site navigation service; A second determination module, used to determine the target service request selected by the user in the service interface according to the second input of the user; A search module, used to search for target information in a background database according to the target service request; A generating module, used for generating first text information and a first Braille dot matrix for voice broadcast according to the target information; The output module is used to call the visual display module to visually display the target information, call the voice broadcast module to voice broadcast the first text information, and display the first Braille dot matrix on the tactile display module.

7. The device according to claim 6, characterized in that The generation module comprises: A first submodule is used to generate first text information and second text information according to the target information when the service type is an information query service, wherein the first text information includes detailed content of the target information and the second text information includes a summary content of the target information; The second submodule is used to convert the second text information into a first Braille dot matrix.

8. The device according to claim 6, characterized in that The generation module comprises: A third submodule is used for, when the service type is an in-site navigation service, using a preset multimodal comparison learning model to search for a target text description template with the highest matching degree with the target information from preset text description templates, wherein the target information is image information; A fourth submodule is used to determine dynamic filling information of the target text description template through real-time environmental perception information; A fifth submodule, used for filling the dynamic filling information into the target text description template to generate first text information; The sixth submodule is used to convert the first text information into a first Braille dot matrix.

9. The device according to claim 8, characterized in that The third submodule includes: The first unit is configured to parse the target information by using an image encoder included in the preset multimodal comparison learning model for each preset text description template when the service type is an in-site navigation service, so as to obtain a first parsing result; The second unit is used to parse the preset text description template through the text encoder included in the preset multimodal comparison learning model to obtain a second parsing result; A third unit, configured to calculate the similarity between the first parsing result and the second parsing result; The fourth unit is used to determine the preset text description template corresponding to the highest similarity as the target text description template with the highest matching degree with the target information.

10. An electronic device, characterized in that: The electronic device includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction is executed by the processor to perform the steps of any one of the barrier-free intelligent service methods based on multimodal fusion in claims 1-5.

Images