Method and system for assisting visually-impaired / hearing-impaired person to browse multimedia without barrier

Abstract

The invention provides a method for assisting visually-impaired / hearing-impaired people in barrier-free multimedia browsing. The method comprises the following steps: acquiring a multimedia file to be processed; processing a pre-acquired visual impairment auxiliary text file into a visual impairment auxiliary sound file, and encoding the visual impairment auxiliary sound file into auxiliary digital audio in the same format as the multimedia file to be processed; and / or, processing a pre-acquired hearing impairment auxiliary text file into a text frame synchronized with the content of the multimedia file to be processed; and synchronously adding the auxiliary digital audio into an idle sound channel of the to-be-processed multimedia file, and / or synchronously adding the text frame into an SEI frame or an auxiliary frame of the to-be-processed multimedia file, so as to form a new multimedia file consistent with the format of the to-be-processed multimedia file. According to the method and the device, the user can selectively listen to the picture audio or watch the picture text, so that the personalized requirements of more visually impaired people and hearing impaired people in the multimedia era are met.

Description

Technical Field

[0001] The present invention relates to the field of multimedia browsing technology, and more specifically, to a method and system for assisting visually impaired / hearing-impaired people to browse multimedia without obstacles. Background Art

[0002] As we all know, visually impaired people can only hear sounds, but cannot understand videos and images through visual experience. At present, the way for visually impaired people to enjoy videos is usually to invite them to cinemas, auditoriums, conference rooms and other places to manually narrate the video images on the spot, giving them a sense of the images and solving the problem that they cannot watch the video images. The above method is costly, labor-intensive, and has a small coverage. In addition, visually impaired people cannot watch richer multimedia content and cannot benefit more visually impaired people.

[0003] For the hearing-impaired, although they can currently obtain sound signals by wearing hearing aids, hearing aids will amplify all sounds at the same time. Sometimes the background noise will be very loud, causing effective information and key content to be masked, and they cannot listen or watch for a long time, making it impossible for the hearing-impaired to comfortably listen to news, music, movies, and other programs.

[0004] Therefore, there is an urgent need for a solution that can facilitate visually impaired / hearing impaired people to browse multimedia. Summary of the invention

[0005] In view of the above problems, an object of the present invention is to provide a method and system for assisting visually impaired / hearing impaired persons to browse multimedia without obstacles, so as to solve the problem that visually impaired / hearing impaired persons are unable to browse multimedia independently and conveniently.

[0006] The method provided by the present invention for assisting visually impaired / hearing impaired people to browse multimedia without obstacles comprises: Get the multimedia files to be processed; Processing the pre-acquired visually impaired auxiliary text file into a visually impaired auxiliary sound file, and encoding the visually impaired auxiliary sound file into auxiliary digital audio in the same format as the multimedia file to be processed; and / or processing the pre-acquired hearing-impaired auxiliary text file into a text frame synchronized with the content of the multimedia file to be processed; The auxiliary digital audio is synchronously added to the idle channel of the multimedia file to be processed, and / or the text frame is synchronously added to the SEI frame or the auxiliary frame of the multimedia file to be processed to form a new multimedia file consistent with the format of the multimedia file to be processed.

[0007] An optional solution is that the visually impaired auxiliary sound file is obtained by the following method: Pre-prepare a sound file corresponding to the multimedia file to be processed; or perform real-time dubbing according to the multimedia file to be processed.

[0008] An optional solution is that the hearing-impaired auxiliary text file is obtained by the following method: Perform AI / manual recognition processing on the multimedia file to be processed to generate text describing the video screen of the multimedia file to be processed; store the text in text form as a hearing-impaired auxiliary text file.

[0009] An optional solution is to make the text into a text file in text form and store it as a hearing-impaired assistance text file, including: storing the text in the SEI frame of each H.264 / 265 of the multimedia file to be processed, or in other video-encoded subsidiary frames of the multimedia file to be processed, and synchronizing it with the storage of the text in the text form in the corresponding video screen to form a new audio and video file.

[0010] An optional solution is that the audio encoding format of the multimedia file to be processed is AAC or OPUS.

[0011] An optional solution is that after the new multimedia file is formed, the new multimedia file is provided to the user in the form of online playback, cinema playback or offline download playback.

[0012] An optional solution is that when the new multimedia file is provided to the user for use in the form of online playback, cinema playback or offline download playback, in addition to playing the original normal audio of the new multimedia file, a voice / text prompt is provided to the user that there are multi-channel audio options available, so that the user can choose to play the audio of one or several channels separately, or choose to play it mixed with the original normal audio.

[0013] On the other hand, the present invention further provides a system for assisting visually impaired / hearing impaired people to browse multimedia without obstacles, which is used to browse multimedia using the method for assisting visually impaired / hearing impaired people to browse multimedia without obstacles as described above, comprising: An original file acquisition unit, used for acquiring a multimedia file to be processed; The auxiliary file acquisition unit is used to process the pre-acquired visually impaired auxiliary text file into a visually impaired auxiliary sound file, and encode the visually impaired auxiliary sound file into an auxiliary digital audio with the same format as the multimedia file to be processed; and / or process the pre-acquired hearing-impaired auxiliary text file into a text frame synchronized with the content of the multimedia file to be processed; The auxiliary file adding unit is used to synchronously add the auxiliary digital audio to the idle channel of the multimedia file to be processed, and / or synchronously add the text frame to the SEI frame or the auxiliary frame of the multimedia file to be processed, so as to form a new multimedia file consistent with the format of the multimedia file to be processed.

[0014] In a third aspect, the present invention provides an electronic device, characterized in that the device comprises: a processor, a memory and a system bus; the processor and the memory are connected via the system bus; the memory is used to store one or more programs, and the one or more programs include instructions, which, when executed by the processor, enable the processor to execute the steps in the aforementioned method of assisting the visually impaired / hearing-impaired to browse multimedia without obstacles.

[0015] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the aforementioned method of assisting the visually impaired / hearing-impaired to browse multimedia without barrier.

[0016] It can be seen from the above technical scheme that the method and system provided by the present invention for assisting the visually impaired / hearing-impaired to browse multimedia without obstacles, by processing specific visually impaired auxiliary sound files and hearing-impaired auxiliary text files into auxiliary digital audio with the same format as the multimedia file to be processed and text frames synchronized with the content of the multimedia file to be processed; and synchronously adding the auxiliary digital audio to the idle channel of the multimedia file to be processed, and / or, synchronously adding the text frame to the SEI frame or auxiliary frame of the multimedia file to be processed to form a new multimedia file consistent with the format of the multimedia file to be processed, thereby meeting the needs of the visually impaired / hearing-impaired to enjoy movies in cinemas, auditoriums, conference rooms and the like, and can also cover many visually impaired / hearing-impaired groups who enjoy film and television programs at home through the Internet. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] By referring to the following description in conjunction with the accompanying drawings, and with a more comprehensive understanding of the present invention, other objects and results of the present invention will become more apparent and easy to understand. In the accompanying drawings: Figure 1 A schematic flow chart of a method for assisting visually impaired / hearing impaired people to browse multimedia without obstacles according to an embodiment of the present invention; Figure 2 A logical structural block diagram of a system for assisting visually impaired / hearing impaired people to browse multimedia without obstacles according to an embodiment of the present invention; Figure 3 A schematic diagram of the principle of a method and system for assisting visually impaired / hearing impaired people to browse multimedia without obstacles according to an embodiment of the present invention; Figure 4FIG. 4 is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention.

[0018] The same reference numerals throughout the drawings indicate similar or corresponding features or functions. DETAILED DESCRIPTION

[0019] In the following description, for the purpose of illustration, in order to provide a comprehensive understanding of one or more embodiments, many specific details are set forth. However, it is apparent that these embodiments may also be implemented without these specific details. In other examples, for ease of describing one or more embodiments, known structures and devices are shown in the form of block diagrams.

[0020] The present invention can be subjected to various changes and can have various embodiments, and specific embodiments are illustrated and described in the accompanying drawings. The accompanying drawings show preferred embodiments of the present invention, which are provided to make the contents disclosed herein comprehensive and complete, and these embodiments fully convey the scope of the present invention to those skilled in the art. The present invention is not limited to this specific embodiment, and all changes, equivalents and substitutes that fall within the concept and technical scope of the present invention are covered and should be understood to be included.

[0021] The terms used in this application are only used to illustrate specific embodiments and are not intended to limit the present invention. Unless the context clearly provides otherwise, singular expressions include plural expressions. In this application, it should be understood that terms such as "including" or "having" are used to specify the presence of features, numbers, steps, operations, constituent elements, parts or combinations thereof recorded in the specification, and do not preclude the presence or additional possibility of one or more other features, numbers, steps, operations, constituent elements, parts or combinations thereof.

[0022] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as those generally understood by those skilled in the art. Terms defined in commonly used dictionaries should be interpreted as having the same meaning as in the context of the relevant technology, and if not clearly defined in this application, they should not be interpreted as ideal or overly formal meanings.

[0023] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0024] Existing multimedia playback methods cannot meet the multimedia appreciation and use needs of hearing-impaired / visually-impaired people. In view of this problem, the present invention provides a method and system for assisting visually-impaired / hearing-impaired people to browse multimedia without obstacles, by selecting one channel from multiple channels of multimedia content such as news, film and television programs or music to load audio that describes the plot and status of the picture in synchronization with the picture, or to load a channel of pure audio effective information and key content without background sound; and adding a text frame synchronized with the content of the multimedia file in the SEI frame or auxiliary frame of the multimedia file, so that the user end can allow the user to choose to listen to the picture description audio or pure audio effective information, key content or watch the picture description text, so as to meet the personalized needs of more visually-impaired and hearing-impaired people in the multimedia era.

[0025] In order to better illustrate the technical solution of the present invention, some technical terms involved in the present invention are briefly explained below.

[0026] SEI (Supplemental Enhancement Information), the full name of which is Supplemental Enhancement Information, is one of the features of the H.265 / H.264 standard, providing a method for adding additional information to the video bitstream. SEI frames can transmit metadata related to the video content, such as time code, subtitles, and shot switching information. SEI frames also support the transmission of user-defined data.

[0027] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0028] It should be noted that the following description of the exemplary embodiments is merely illustrative and is not intended to limit the present invention and its application or use. Technologies and devices known to ordinary technicians in the relevant field may not be discussed in detail, but where appropriate, the technologies and devices should be considered as part of the specification.

[0029] In order to illustrate the method and system provided by the present invention for assisting the visually impaired / hearing impaired to browse multimedia without obstacles, Figure 1 , Figure 2 , Figure 3 The embodiments of the present invention are exemplified from three aspects: method flow, system architecture and principle.

[0030] like Figure 1 and Figure 3 As shown together, the method provided by this embodiment for assisting visually impaired / hearing impaired people to browse multimedia without obstacles includes: S110: Acquire a multimedia file to be processed; S120: Processing the pre-acquired visually impaired auxiliary text file into a visually impaired auxiliary sound file, and encoding it into auxiliary digital audio in the same format as the multimedia file to be processed; and / or processing the pre-acquired hearing-impaired auxiliary text file into a text frame synchronized with the content of the multimedia file to be processed; S130: Synchronously adding the auxiliary digital audio to an idle channel of the multimedia file to be processed, and / or synchronously adding the text frame to an SEI frame or an auxiliary frame of the multimedia file to be processed, so as to form a new multimedia file consistent with the format of the multimedia file to be processed.

[0031] Among them, as an example, the following method can be used to obtain a sound file assisting the visually impaired: first, AI or manual recognition processing is performed on the multimedia file to be processed to generate text that describes the video screen of the multimedia file to be processed; then the text of the text is made into a sound file assisting the visually impaired. Pre-prepare a sound file corresponding to the description of the multimedia file to be processed; or, perform real-time dubbing based on the multimedia file to be processed. The following method can be used to obtain a text file assisting the hearing impaired: first, AI or manual recognition processing is performed on the multimedia file to be processed to generate text that describes the video screen of the multimedia file to be processed; then the text is stored in the form of text as a text file assisting the hearing impaired.

[0032] Most of the film and television programs currently available for users to watch are stored in MP4, m3u8 or TS format, and their audio encoding formats are mostly AAC or OPUS. AAC or OPUS both support multiple audio channels, AAC supports 48 channels and OPUS supports 255 channels. Therefore, in a more specific embodiment of the present invention, the multimedia content such as the produced movies, film and television programs, audio programs, etc. are imported into the audio editing software, and the audio editing software is used to encode the sound files describing the state and plot of the picture, or other pre-made audio sound files, or real-time dubbing into AAC or OPUS digital audio, and synchronously added to the movie, film and television program, audio AAC or OPUS One or more idle channels are formed to form a new audio and video file containing several more channels than the original audio and video file; the idle channel can be a specified or default idle channel, or the idle channel can be determined according to the characteristics of the channel or the use priority. Here, it is necessary to make the format of the new audio and video file consistent with the original audio and video file format so as to better merge the two files.

[0033] In another specific embodiment of the present invention, the audio and video files can also be processed by artificial intelligence (AI) technology and manual proofreading to identify the video screen, and the generated text describing the video screen is stored in the form of text in each H.264 / 265 SEI frame, or in other video encoding subsidiary frames of the multimedia file to be processed, and synchronized with the video screen to form a new audio and video file. The method of adding channels is similar to the method of adding text, and can exist in a new audio and video file at the same time.

[0034] The new audio and video files formed in the above embodiment can be provided to users in at least the following three ways: online playback, cinema playback, and offline download playback.

[0035] Among them, the online playback method refers to placing new audio and video files on the cloud platform for users to watch on demand. The terminal devices used by users can be mobile phones + APP + headsets, pads + APP + headsets, computers + applications + headsets, etc. When users play audio and video files + dubbing on demand through APP, they can use streaming media transmission protocols such as RTMP, RTSP, WEBRTC, and push / pull streams to users in real time through IP networks to watch or listen. Users can also use headsets to search for new audio and video programs on the cloud platform through the App or application on the terminal device. The searched new audio and video programs are broadcast to users through the headset voice for users to choose. Users can use headsets to select new audio and video programs they want to watch and play them with voice commands. When the APP is playing new audio and video programs, in addition to playing the original normal audio, it can also voice / text prompt users that there are multi-channel audio to choose from. Users can choose to play the audio of one or several channels alone, or choose to play it mixed with the original normal audio. Users can also control the volume with voice, and can also exit the APP or application with voice commands after playback is completed. There are also corresponding buttons in the APP or application to complete operations such as selecting a new video file to play, selecting one or more newly added audio channels to play, and choosing to mix the sound with the original video.

[0036] When playing audio and video files + screen description text on demand through APP, users can use headsets to search for new audio and video programs on the cloud platform through the APP or application on the terminal device, and the new audio and video programs searched are broadcast to users through the headset voice for users to choose. Users can use voice commands to select new audio and video programs they want to watch through the headset user for playback. When the APP plays new audio and video programs, in addition to playing the original normal audio, it will also voice / text prompt users that there are "screen description texts" to choose from. Among them, a more preferred solution is that the APP provides a specific type of voice selection similar to the driving navigation method. Users can choose their favorite male, female or specific person to broadcast the "screen description text". Users can choose to play the audio of the "screen description text" alone, or choose to mix it with the original normal audio. Users can also use voice to control the volume. After the playback is completed, they can also use voice commands to exit the APP or application. In addition, corresponding buttons can be set in the APP or application for users to manually complete operations such as selecting new video files to play and choosing to mix with the sound of the original video.

[0037] In the cinema playback mode, the new audio and video file refers to a digital movie audio and video file with one or more channels added to the digital movie audio and video file, referred to as the barrier-free digital movie. The digital playback equipment in the cinema is mainly the server and playback software. When the barrier-free digital movie is played normally in the cinema, the original digital audio is decoded and transmitted to the cinema's audio equipment through cables for normal sound playback. At the same time, the playback software is added with the function of extracting the newly added one or more channels of audio, and after decoding the audio of this one or more channels, it is pushed to the cinema audience's Bluetooth / Wi-Fi headphones through the cinema's Bluetooth / WI-FI wireless network, and the sound is played and provided to the visually impaired and people in need. The buttons on the user's Bluetooth / Wi-Fi headphones can adjust the volume. In this way, normal people and visually impaired people (or people in need) can watch (listen) barrier-free digital movies at the same time in the cinema. Of course, the sound or narration describing the picture can be mixed with the sound of the original normal movie picture, and transmitted to the cinema's audio equipment through cables for playback, so as to play the barrier-free digital movie separately for the visually impaired or other people in need.

[0038] Offline playback mode means downloading new audio and video files to the user's terminal device. The terminal device used by the user can be a mobile phone + APP + headset, Pad + APP + headset, computer + application + headset. When the APP plays audio and video files on demand + dubbing, the user's operation method is similar to the online APP on-demand playback of audio and video files + dubbing mode, except that the new audio and video files are stored locally, and there is no need to push / pull streams. The new audio and video files can be played directly.

[0039] When the APP plays audio and video files + screen description text on demand, the user's APP can convert text to speech by connecting to the Internet, or use the AI ​​function of mobile phones / PADs / PCs to directly complete the text-to-speech conversion locally. The user's operation method is similar to the mode when the online APP plays audio and video files + screen description text on demand, so it will not be described in detail here.

[0040] In this embodiment, a specific mark is set on the new audio / video file (cover) to indicate that the new audio / video file can be used for barrier-free browsing by visually / hearing-impaired people, thereby helping users to identify and use it.

[0041] As described above, the method provided by the present invention for assisting the visually impaired / hearing impaired to browse multimedia without obstacles is achieved by processing specific visually impaired auxiliary sound files and hearing impaired auxiliary text files into auxiliary digital audio with the same format as the multimedia file to be processed and text frames synchronized with the content of the multimedia file to be processed; and synchronously adding the auxiliary digital audio and / or text frames to the multimedia file to be processed to form a new multimedia file with the same format as the multimedia file to be processed, thereby meeting the needs of the visually impaired / hearing impaired to enjoy movies in cinemas, auditoriums, conference rooms and the like, and can also cover many visually impaired groups who enjoy film and television programs at home through the Internet, as well as hearing impaired groups who listen to audio.

[0042] In detail, the present invention has the following advantages: 1. Use existing audio and video files for editing and production, no need to shoot separately, low production cost; 2. There are a large number of audio and video files to choose from, allowing the visually impaired and hearing-impaired to enjoy a large amount of multimedia content without obstacles; 3. Audio and video files can be used multiple times after being produced once; 4. The newly added audio channel does not affect the normal use of audio and video files; 5. It can benefit and cover users in need, which is a great blessing for the visually impaired and hearing-impaired people; 6. It is possible to add channels and adapt to different languages ​​for use by ethnic minorities. It can also be exported to other countries for use, thus achieving barrier-free international communication.

[0043] like Figure 2 As shown, the present invention also provides a system 200 for assisting the visually impaired / hearing impaired to browse multimedia without barrier, which is used to browse multimedia using the method for assisting the visually impaired / hearing impaired to browse multimedia without barrier as described above, and includes an original file acquisition unit 210, an auxiliary file acquisition unit 220 and an auxiliary file adding unit 230.

[0044] The original file acquisition unit 210 is used to acquire the multimedia file to be processed; The auxiliary file acquisition unit 220 is used to process the pre-acquired visually impaired auxiliary text file into a visually impaired auxiliary sound file, and encode the visually impaired auxiliary sound file into an auxiliary digital audio in the same format as the multimedia file to be processed; and / or process the pre-acquired hearing-impaired auxiliary text file into a text frame synchronized with the content of the multimedia file to be processed; The auxiliary file adding unit 230 is used to synchronously add the auxiliary digital audio to the idle channel of the multimedia file to be processed, and / or synchronously add the text frame to the SEI frame or the auxiliary frame of the multimedia file to be processed, so as to form a new multimedia file consistent with the format of the multimedia file to be processed.

[0045] The above-mentioned system for assisting the visually impaired / hearing impaired to browse multimedia without barrier is a system corresponding to the above-mentioned method for assisting the visually impaired / hearing impaired to browse multimedia without barrier. Its more specific implementation method can refer to the specific embodiment of the above-mentioned method for assisting the visually impaired / hearing impaired to browse multimedia without barrier, and will not be described in detail here.

[0046] In addition, the present invention provides an electronic device, including a processor, a memory and a system bus; the processor and the memory are connected via the system bus; the memory is used to store one or more programs, and the one or more programs include instructions, which, when executed by the processor, enable the processor to execute the steps in the aforementioned method of assisting the visually impaired / hearing-impaired to browse multimedia without barrier, so as to realize the aforementioned method of assisting the visually impaired / hearing-impaired to browse multimedia without barrier.

[0047] The logical structure of the above electronic equipment is as follows Figure 4 As shown. The memory 11 is used to store computer programs; the processor 12 is used to implement the steps of the method for assisting visually impaired / hearing impaired people to browse multimedia without obstacles as described in any of the above method embodiments when executing the computer program; the bus 13 is used to interconnect the memory 11, the processor 12, the network interface 14 and the user interface 15.

[0048] Among them, the memory 11 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the device, such as a hard disk of the device. In other embodiments, the memory 11 can also be an external storage device of the device, such as a plug-in hard disk equipped on the device, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Further, the memory 11 can also include both an internal storage unit of the device and an external storage device. The memory 11 can not only be used to store application software and various types of data installed in the device, such as program codes for executing fault prediction methods, etc., but can also be used to temporarily store data that has been output or is to be output. The processor 12 can be a central processing unit (CPU) in some embodiments. In some embodiments, the processor 12 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor or other data processing chip, which is used to run the program code stored in the memory 11 or process data, such as executing the code of the social recommendation method based on homomorphic encryption. The bus 13 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0049] Further, optionally, the network interface 14 may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices.

[0050] Optionally, the user interface 15 may be a display, an input unit (such as a keyboard), or a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface.

[0051] The emotional image editing program based on the multi-agent framework stored in the memory 11 of the electronic device is a combination of multiple instructions. When the processor 12 is running, the following steps can be implemented: S110: Acquire a multimedia file to be processed; S120: Processing the pre-acquired visually impaired auxiliary text file into a visually impaired auxiliary sound file, and encoding the visually impaired auxiliary sound file into auxiliary digital audio in the same format as the multimedia file to be processed; and / or processing the pre-acquired hearing-impaired auxiliary text file into a text frame synchronized with the content of the multimedia file to be processed; S130: Synchronously adding the auxiliary digital audio to an idle channel of the multimedia file to be processed, and / or synchronously adding the text frame to an SEI frame or an auxiliary frame of the multimedia file to be processed, so as to form a new multimedia file consistent with the format of the multimedia file to be processed.

[0052] Specifically, the specific implementation method of the processor 12 for the above instructions can refer to Figure 1 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0053] Those skilled in the art will appreciate that the structure shown in the figure does not constitute a limitation on the electronic device, and may include fewer or more components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0054] For example, although not shown, the electronic device may also include a power source (such as a battery) for supplying power to various components. Preferably, the power source may be logically connected to the at least one processor 12 through a power management device, so that the power management device can realize functions such as charging management, discharging management, and power consumption management. The power source may also include any components such as one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, and power status indicators. The electronic device may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0055] It should be understood that the above embodiment is for illustration only, and the scope of the patent application is not limited to this structure.

[0056] Furthermore, if the module / unit integrated in the electronic device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable storage medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM).

[0057] As described above with reference to the accompanying drawings, the method and system for assisting the visually impaired / hearing-impaired to browse multimedia without obstacles proposed by the present invention are described by way of example. However, those skilled in the art should understand that various improvements can be made to the method and system for assisting the visually impaired / hearing-impaired to browse multimedia without obstacles proposed by the present invention without departing from the content of the present invention. Therefore, the protection scope of the present invention should be determined by the content of the attached claims.

Claims

1. A method for assisting visually impaired / hearing impaired people to browse multimedia without obstacles, characterized in that: include: Get the multimedia files to be processed; Processing the pre-acquired visually impaired auxiliary text file into a visually impaired auxiliary sound file, and encoding the visually impaired auxiliary sound file into auxiliary digital audio in the same format as the multimedia file to be processed; and / or processing the pre-acquired hearing-impaired auxiliary text file into a text frame synchronized with the content of the multimedia file to be processed; The auxiliary digital audio is synchronously added to the idle channel of the multimedia file to be processed, and / or the text frame is synchronously added to the SEI frame or the auxiliary frame of the multimedia file to be processed to form a new multimedia file consistent with the format of the multimedia file to be processed.

2. The method for assisting the visually impaired / hearing impaired to browse multimedia without obstacles as claimed in claim 1, characterized in that: The visually impaired auxiliary sound file is obtained by the following method: prefabricate a sound file corresponding to and describing the multimedia file to be processed; or, Perform real-time dubbing according to the multimedia file to be processed.

3. The method for assisting the visually impaired / hearing impaired to browse multimedia without obstacles as claimed in claim 2, characterized in that: The hearing-impaired assistance text file is obtained by the following method: Performing AI / manual recognition processing on the multimedia file to be processed to generate text describing the video screen of the multimedia file to be processed; The text is stored in a text format as a hearing-impaired assistance text file.

4. The method for assisting the visually impaired / hearing impaired to browse multimedia without obstacles as claimed in claim 3, characterized in that: The text is stored in a text format as a hearing-impaired auxiliary text file, including: The text is stored in the SEI frame of each H.264 / 265 of the multimedia file to be processed, or in the subsidiary frame of other video encoding of the multimedia file to be processed, and is synchronized with the storage of the text in the corresponding video picture to form a new audio and video file.

5. The method for assisting the visually impaired / hearing impaired to browse multimedia without obstacles as claimed in claim 4, characterized in that: The audio encoding format of the multimedia file to be processed is AAC or OPUS.

6. The method for assisting the visually impaired or hearing impaired to browse multimedia without obstacles as claimed in claim 1, characterized in that: After forming the new multimedia file, the method further comprises: The new multimedia file is provided to the user in the form of online playback, cinema playback or offline download playback.

7. The method for assisting the visually impaired or hearing impaired to browse multimedia without obstacles as claimed in claim 6, characterized in that: When the new multimedia file is provided to the user in the form of online playback, cinema playback or offline download playback, In addition to playing the original normal audio of the new multimedia file, the user is also prompted by voice / text that there are multi-channel audios to choose from, so that the user can choose to play the audio of one or more channels alone, or choose to play it mixed with the original normal audio.

8. A system for assisting visually impaired / hearing impaired people to browse multimedia without obstacles, characterized in that: The method for assisting the visually impaired or hearing-impaired to browse multimedia without obstacles as claimed in any one of claims 1 to 7 is used to browse multimedia, comprising: The original file acquisition unit is used to acquire the multimedia file to be processed; An auxiliary file acquisition unit, used to process the pre-acquired visually impaired auxiliary text file into a visually impaired auxiliary sound file, and encode the visually impaired auxiliary sound file into an auxiliary digital audio in the same format as the multimedia file to be processed; and / or, process the pre-acquired hearing-impaired auxiliary text file into a text frame synchronized with the content of the multimedia file to be processed; The auxiliary file adding unit is used to synchronously add the auxiliary digital audio to the idle channel of the multimedia file to be processed, and / or synchronously add the text frame to the SEI frame or the auxiliary frame of the multimedia file to be processed, so as to form a new multimedia file consistent with the format of the multimedia file to be processed.

9. An electronic device, characterized in that: The device includes: a processor, a memory and a system bus; the processor and the memory are connected via the system bus; the memory is used to store one or more programs, and the one or more programs include instructions, which, when executed by the processor, enable the processor to execute the steps in the method of assisting the visually impaired / hearing-impaired to browse multimedia without obstacles as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for assisting the visually impaired / hearing impaired to browse multimedia without obstacles as described in any one of claims 1 to 7 is implemented.

Images