Environmental perception interaction system for visual impairment patient

Abstract

The invention discloses an environment perception interaction system for visual disorder patients, and belongs to the field of artificial intelligence. The awakening module is used for awakening and receiving user voice, when an awakening keyword is detected, the user is informed that the system is awakened, and the system is switched into a voice interaction mode; the voice interaction module is used for starting the system, and when the user voice is detected, converting the user voice into a machine text required by the user and temporarily storing the machine text; the voice processing module is also used for receiving the broadcast type machine text output by the artificial intelligence module and converting and synthesizing the broadcast type machine text into human voice; the voice broadcasting module is also used for broadcasting the synthesized human voice to the user; the visual processing module is used for acquiring a picture of a current scene, performing target detection, and outputting the picture and article name and position information to the artificial intelligence module; and the artificial intelligence module is used for processing the received text, photo, article name and article position information into broadcast type machine text based on blind person space cognitive logic and cognitive map normal form and outputting the broadcast type machine text to the voice interaction module.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence and relates to an environment perception interaction system for visually impaired patients. Background Art

[0002] Eyes are an important organ for people to perceive the world. When healthy people enter an unfamiliar environment, they observe the surrounding objects to judge the nature of the environment, establish environmental cognition or spatial representation, and then plan what they can do next. However, the fear and uneasiness felt by blind people after entering an unfamiliar environment are largely attributed to the inability to understand the current environment through vision. Therefore, establishing spatial representation of unfamiliar environments for blind people is an important means to help them get out of the shadows and return to society. Summary of the invention

[0003] The purpose of the present invention is to solve the above-mentioned problems existing in the prior art and to provide an environment perception interaction system for visually impaired patients.

[0004] The present invention can describe the real-time scene in words, help blind users build cognitive maps in their minds, and enable blind users to efficiently build an overall cognition of their current environment even without being able to see, help blind users overcome their fear of unfamiliar environments, and improve their spatial representation ability. At the same time, it can initially help blind users perceive the position of their prosthetic hands by mobilizing their own sound and positioning abilities, which is of great significance for subsequent navigation and training in grasping target objects.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] An environment perception interaction system for visually impaired patients, comprising a wake-up module, a voice interaction module, a visual processing module and an artificial intelligence module, wherein the voice interaction module comprises a cloud-based voice recognition unit and a broadcasting unit;

[0007] The wake-up module is connected to the voice interaction module in communication, the voice interaction module is connected to the artificial intelligence module in bidirectional communication, the visual processing module is connected to the artificial intelligence module in communication, the cloud voice recognition unit of the voice interaction module is connected to the wake-up module and the artificial intelligence module in communication, and the broadcast unit is connected to the artificial intelligence module in communication; wherein:

[0008] A wake-up module is used to wake up and receive user voice at any time. In the wake-up phase, when a wake-up keyword is detected in the user's voice, the user is informed that the environment perception interaction system has been awakened and can start interaction, and the environment perception interaction system is automatically switched to voice interaction mode;

[0009] A voice interaction module is used to start when the environment perception interaction system is switched to the voice interaction mode.

[0010] Also used for, when user voice is detected, converting the user voice into user demand machine text, and temporarily storing the user demand machine text for further processing or response;

[0011] Also used for receiving the broadcast type machine text output by the artificial intelligence module, and converting and synthesizing the broadcast type machine text into human speech;

[0012] Also used to broadcast synthesized human voice to users;

[0013] A visual processing module, used to obtain a photo of the current scene through a depth camera and perform target detection, obtain the name and location information of the object in the photo, and output the photo, the name and location information of the object to the artificial intelligence module at the same time;

[0014] The artificial intelligence module is used to simultaneously receive the user's required machine text, photos, item names and item location information, and process them into broadcast-type machine text for output to the voice interaction module.

[0015] Furthermore, the artificial intelligence module is supported by a specialized ChatGPT, and the specialized ChatGPT is a ChatGPT that has been specialized in advance through prompt engineering.

[0016] Furthermore, the wake-up module, voice interaction module, and artificial intelligence module are all installed on a hardware platform.

[0017] Furthermore, the depth camera of the visual processing module is worn on the chest of the user, and the lens of the depth camera faces the front of the user, and the visual processing module is communicated with the artificial intelligence module 4 through a hardware platform.

[0018] Furthermore, the hardware platform is communicatively connected to the depth camera via a USB 3.0 interface.

[0019] Furthermore, the hardware platform is communicatively connected with the driver-free coil-type wired headset with a built-in audio decoding chip via a USB 2.0 interface.

[0020] Furthermore, the broadcast unit is a speech synthesis function control board installed on the back of the artificial hand, the speech synthesis function control board includes a main control chip and a function chip, the main control chip and the function chip are connected in communication, and the main control chip is connected in serial communication with the hardware platform;

[0021] The main control chip of the broadcast unit is used to receive the broadcast machine text output by the artificial intelligence module and send data or command information to the functional chip in the command frame format;

[0022] The functional chip of the broadcast unit is used for speech synthesis and pre-made playback, and processes the broadcast machine text in the command frame format sent by the main control chip into human speech for broadcasting.

[0023] Furthermore, the object location information includes a quantitative expression of the distance.

[0024] The beneficial effects of the present invention compared to the prior art are: the system can be directly delivered to blind users for use, without the need for prior learning and training, that is, it helps blind users to establish spatial representation within 1 minute of entering an unfamiliar environment in a "zero sample" mode. Compared to allowing the blind to explore an unfamiliar environment completely independently or having sighted people support and guide the blind to explore, the system can help blind users establish spatial representation at a lower cost and lower time cost, thereby improving their exploration efficiency. At the same time, the environmental description generated by the system is less subjective because it comes from a large language model, and the system's hardware and software platform is very versatile, making it easy to integrate into other devices and instruments, thereby improving the intelligence of the equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a working effect diagram of the environment perception interaction system for visually impaired patients of the present invention;

[0026] Figure 2 It is a flowchart of the overall process of the environment perception interaction system for visually impaired patients of the present invention;

[0027] Figure 3 It is a flow chart of the recognition channel of the voice interaction module;

[0028] Figure 4 It is a flowchart of the visual processing module;

[0029] Figure 5 It is a flowchart of the speech synthesis process of the speech interaction module;

[0030] Figure 6 This is a synthetic simulation verification diagram of the voice interaction module;

[0031] Figure 7 It is a signal connection relationship block diagram of the environment perception interaction system for visually impaired patients of the present invention.

[0032] The names and reference numerals of the components involved in the above drawings are as follows:

[0033] Wake-up module 1, voice interaction module 2, cloud voice recognition unit 21, broadcast unit 22, main control chip 221, function chip 222, visual processing module 3, artificial intelligence module 4. DETAILED DESCRIPTION

[0034] Specific implementation method 1: Figure 1-Figure 7 As shown, this embodiment discloses an environment perception interaction system for visually impaired patients, including a wake-up module 1, a voice interaction module 2, a visual processing module 3 and an artificial intelligence module 4, wherein the voice interaction module 2 includes a cloud voice recognition unit 21 and a broadcast unit 22;

[0035] The wake-up module 1 is connected in communication with the voice interaction module 2, the voice interaction module 2 is connected in bidirectional communication with the artificial intelligence module 4, the visual processing module 3 is connected in communication with the artificial intelligence module 4, the cloud voice recognition unit 21 of the voice interaction module 2 is connected in communication with the wake-up module 1 and the artificial intelligence module 4, and the broadcasting unit 22 is connected in communication with the artificial intelligence module 4; wherein:

[0036] A wake-up module 1 is used to wake up and receive user voice at any time. In the wake-up phase, when a wake-up keyword is detected in the user's voice, the user is informed that the environment perception interaction system has been awakened and can start interaction, and the environment perception interaction system is automatically switched to voice interaction mode;

[0037] The voice interaction module 2 is used to start when the environment perception interaction system is switched to the voice interaction mode.

[0038] It is also used to, when user voice is detected, convert the user voice into user demand machine text, and temporarily store the user demand machine text for further processing or response (completed by the cloud voice recognition unit 21);

[0039] It is also used to receive the broadcast type machine text output by the artificial intelligence module 4, and convert and synthesize the broadcast type machine text into human speech (completed by the cloud-based speech recognition unit 21);

[0040] Also used to announce the synthesized human voice to the user (accomplished by the announcement unit 22);

[0041] The visual processing module 3 is used to obtain a photo of the current scene through a depth camera and perform target detection, obtain the name and location information of the object in the photo, and output the photo and the name and location information of the object to the artificial intelligence module 4 at the same time;

[0042] The artificial intelligence module 4 is used to simultaneously receive the user's required machine text, photos, item names and item location information, and process them into broadcast-type machine text and output them to the voice interaction module 2.

[0043] Furthermore, the artificial intelligence module 4 is supported by a specialized ChatGPT, and the specialized ChatGPT is a ChatGPT that has been specialized in advance through prompt engineering.

[0044] Furthermore, the wake-up module 1, the voice interaction module 2, and the artificial intelligence module 4 are all installed on a hardware platform.

[0045] Furthermore, the depth camera of the visual processing module 3 is worn on the chest of the user, and the lens of the depth camera faces the front of the user, and the visual processing module 3 is communicated with the artificial intelligence module 4 through a hardware platform.

[0046] Furthermore, the hardware platform is communicatively connected to the depth camera via a USB 3.0 interface.

[0047] Furthermore, the hardware platform is communicatively connected with the driver-free coil-type wired headset with a built-in audio decoding chip via a USB 2.0 interface.

[0048] Furthermore, the broadcast unit 22 is a speech synthesis function control board installed on the back of the artificial hand, and the speech synthesis function control board includes a main control chip 221 and a function chip 222, the main control chip 221 and the function chip 222 are connected in communication, and the main control chip 221 is connected in serial communication with the hardware platform (the hardware platform on which the visual processing module 3 and the artificial intelligence module 4 depend for operation);

[0049] The main control chip 221 of the broadcast unit 22 is used to receive the broadcast type machine text output by the artificial intelligence module 4, and send data or command information to the function chip 222 in the command frame format;

[0050] The function chip 222 of the broadcast unit 22 is used for speech synthesis and pre-made playback (supporting mixed Chinese and English text synthesis and four common text encoding formats), and processes the broadcast machine text in the command frame format sent by the main control chip 221 into human speech for broadcasting.

[0051] Furthermore, the object location information includes a quantitative expression of the distance.

[0052] like Figure 1-Figure 7 As shown, the working process of this system is:

[0053] 1. The hardware platform of this system uses the NVIDIA Jetson Orin NX high-performance edge computing platform, which has an AI performance of 100TOPS, equipped with a 1024-core NVIDIA Ampere architecture GPU with 32 Tensor Cores, and an 8-core -A78AE v8.2 64-bit CPU, which can provide excellent computing power to support various complex artificial intelligence (AI) applications. It communicates with the external device Intel realsense D435i depth camera through the USB3.0 interface, and communicates with the driver-free coil-type wired headset with built-in audio decoding chip through the USB2.0 interface, leaving TypeC, USB3.0 and other interfaces to support the possibility of connecting with other devices.

[0054] 2. After the system is turned on, it can be awakened at any time and receive the user's needs for environmental perception. After it is turned on, the system will always be in a state of waiting to be awakened and detect the user's voice. After detecting the awakening keyword, the system will emit a prompt sound to feedback to the user that the system has been awakened and can start interaction, and automatically set the system itself to voice interaction mode.

[0055] 3. When the system is in voice interaction mode, it first recognizes the user's voice, converts it into text, and temporarily stores it as the user's needs. The user's needs here are essentially to express the need for environmental description, which may be accompanied by some personal emphasis, such as whether there are other people in the environment, whether the environment is serious or comfortable, etc. Therefore, the system will save the user's needs instead of simply processing them into a "yes" or "no" judgment for subsequent program execution.

[0056] Fourth, the uplink channel of voice interaction module 2 converts human voice into machine text, and calls Baidu voice recognition interface online. The near-field Mandarin Chinese recognition accuracy rate reaches 98% in specific scenarios, which can greatly ensure the business availability of recognition results; Baidu short voice recognition adopts the latest decoding technology, and the API interface recognition speed is increased by more than 5 times, and the time consumed is only one-tenth of the audio duration, which improves the voice interaction experience. If there is no semantic detection result for 5 consecutive seconds, the system infers that the user's expression is completed, ends the voice recognition in the interactive mode, turns on the depth camera, and enters the visual processing mode.

[0057] 5. The downlink channel of the voice interaction module 2 is the part that converts machine text into human voice, and is a voice synthesis function control board placed on the back of the prosthetic hand. The voice synthesis function control board uses STM32CBT6 as the main control chip 221 and SYN6288 as the function chip 222 (main function chip). The main control chip 221 and the function chip 222 communicate with the hardware platform through serial ports. The function chip 222 has both voice synthesis and pre-made playback functions, and the sound effect is natural, the speech speed is controllable, the tone is clear and mellow, and the comprehensibility is better. At the same time, the function chip 222 supports mixed Chinese and English text synthesis and four common text encoding formats. It can autonomously identify special readings in life such as date, time, surname, etc. and broadcast them according to people's habits. The text synthesis accuracy reaches 96%, which is enough to meet the needs of this project. After the STM32CBT6 main control chip 221 receives the environment description text sent by the artificial intelligence module 4 in the serial communication mode, it sends data or command information to the function chip 222 in the command frame format, and the function chip 222 performs corresponding operations according to the command frame sent. Functional chip 222 supports editing all command frames in the format of "frame header FD+data area length+data area". All commands and data that can be effectively received by the SYN6288 function chip 222 need to be packaged and edited in the form of "frame" before transmission. If "frame" encapsulation is not performed or the format specified in the data manual is not encapsulated, the function chip 222 will cause an error in operation, and garbled characters will be broadcast instead of the target text. After the STM32CBT6 main control chip 221 receives the text information of GPT, it first constructs frame configuration information such as frame header, data area length and command word, and then performs XOR check on it; after the frame check is completed, the text information is constructed in the "GB2312" format, and the text information is XOR checked; after the text information check is completed, it is sent to the broadcast unit 22 through the serial port for broadcasting. Whenever the voice interaction module 2 receives the environmental description information transmitted by the artificial intelligence module 4, it will enter the relevant interrupt first, perform voice synthesis, and broadcast the environmental information to the user in a timely manner.

[0058] 6. When the system enters the visual processing mode, it first turns on the realsense depth camera and takes continuous photos of the current scene. Since the depth camera is worn on the user's chest when the system is used, the system is more stable and closer to the user's naked eye perspective than when it is worn on the top of the head or on the wrist. After obtaining the image, the system will perform target detection on it to obtain the names of the objects in the current scene and their location information. One of the characteristics of this system is that the visual processing module 3 is used to obtain the location information in a coded format. The location information contains a quantitative expression of the distance, which is very important for users to establish spatial cognition.

[0059] 7. The artificial intelligence module 4 of this system is supported by an expert ChatGPT. It is impossible to obtain specific numbers using only the artificial intelligence module 4, and it is impossible for users to understand only the location information of the visual processing module 3. Therefore, this system transmits the image results and digital results generated by the visual processing module 3 to the artificial intelligence module 4, and uses its powerful text analysis and generation capabilities to generate a natural language format environment description that users can understand. According to the user's "environmental perception" needs, this system obtains the current scene picture, combines the target recognition results, and passes the text information and picture information to the expert ChatGPT of the artificial intelligence module 4, requiring the generation of environmental perception text according to the "cognitive map" paradigm.

[0060] ChatGPT is a pre-trained language model developed by OpenAI. It is based on the Transformer architecture and is trained with a large amount of text data to generate coherent and natural conversations. The Transformer architecture does not rely on recurrent or convolutional layers, but is completely based on the attention mechanism to process sequence data. Therefore, compared with traditional natural language processing architectures, it has the advantages of parallel processing, long-distance dependency capture, and high scalability. Although ChatGPT has powerful generation capabilities, the environment-aware text generated by the current general model is inefficient for the establishment of user space representation and often contains a large number of descriptions that are not helpful to users. Therefore, this system pre-expertizes it through prompt engineering so that the generated environment-aware text can describe the environment as efficiently as possible. Expert ChatGPT generates text for the establishment of environment perception, assisting users in the next step of path planning and task arrangement, rather than other descriptions that are not related to the path, such as floor color, specific patterns of wall paintings, etc. In the process of becoming an expert, this system incorporates the idea of ​​"cognitive map". The resulting text can describe the current scene in accordance with the "scene representation" which is a common way of thinking used by acquired blind people. It helps users establish spatial representation at a lower cost (compared to using volunteers to guide and explain to the blind many times).

[0061] 8. After obtaining the text generated by the expert ChatGPT, this system returns to the voice interaction module 2 for speech synthesis, and conveys the scene description to the user in the form of voice broadcast, helping the user to establish environmental awareness of unfamiliar environments.

[0062] When the environment perception interaction system for visually impaired patients of the present invention is used, the depth camera is worn on the user's chest, the depth camera is ensured to be level, headphones are worn, a prosthetic hand is worn, and "Run" is clicked after powering on, and the system can automatically run and wait for the user to wake up.

[0063] The above are only preferred specific implementation modes of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical solutions and inventive concepts of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. An environmental perception interaction system for visually impaired patients, Features: It comprises a wake-up module (1), a voice interaction module (2), a visual processing module (3) and an artificial intelligence module (4), wherein the voice interaction module (2) comprises a cloud-based voice recognition unit (21) and a broadcasting unit (22); The wake-up module (1) is connected to the voice interaction module (2) in communication, the voice interaction module (2) is connected to the artificial intelligence module (4) in bidirectional communication, the visual processing module (3) is connected to the artificial intelligence module (4) in communication, the cloud voice recognition unit (21) of the voice interaction module (2) is connected to the wake-up module (1) and the artificial intelligence module (4), and the broadcasting unit (22) is connected to the artificial intelligence module (4); wherein: A wake-up module (1) is used to wake up and receive user voice at any time. In the wake-up phase, when a wake-up keyword is detected in the user's voice, the user is informed that the environment perception interaction system has been awakened and can start interaction, and the environment perception interaction system is automatically switched to a voice interaction mode; A voice interaction module (2) is used to start when the environment perception interaction system is switched to a voice interaction mode, Also used for, when user voice is detected, converting the user voice into user demand machine text, and temporarily storing the user demand machine text for further processing or response; It is also used to receive the broadcast type machine text output by the artificial intelligence module (4), and convert and synthesize the broadcast type machine text into human speech; Also used to broadcast synthesized human voice to users; A visual processing module (3) is used to obtain a photo of the current scene through a depth camera and perform target detection, obtain the name and location information of the object in the photo, and output the photo, the name and location information of the object to the artificial intelligence module (4) at the same time; The artificial intelligence module (4) is used to simultaneously receive the user's required machine text, photos, and item names and item location information, and process them into broadcast-type machine text for output to the voice interaction module (2).

2. The environment perception interaction system for visually impaired patients according to claim 1, Features: The artificial intelligence module (4) is supported by a specialized ChatGPT, and the specialized ChatGPT is a ChatGPT that has been specialized in advance through prompt engineering.

3. The environment perception interaction system for visually impaired patients according to claim 1, Features: The wake-up module (1), the voice interaction module (2), and the artificial intelligence module (4) are all installed on a hardware platform.

4. The environment perception interaction system for visually impaired patients according to claim 1, Features: The depth camera of the visual processing module (3) is worn on the chest of the user, and the lens of the depth camera faces the front of the user. The visual processing module (3) is connected to the artificial intelligence module (4) through a hardware platform.

5. The environment perception interaction system for visually impaired patients according to claim 4, Features: The hardware platform is connected to the depth camera through a USB 3.0 interface.

6. The environment perception interaction system for visually impaired patients according to claim 5, Features: The hardware platform is communicatively connected with the driver-free coil-type wired headset with a built-in audio decoding chip via a USB 2.0 interface.

7. The environment perception interaction system for visually impaired patients according to claim 6, Features: The broadcast unit (22) is a speech synthesis function control board installed on the back of the artificial hand, the speech synthesis function control board comprises a main control chip (221) and a function chip (222), the main control chip (221) and the function chip (222) are communicatively connected, and the main control chip (221) is communicatively connected to the hardware platform serial port; wherein: The main control chip (221) of the broadcast unit (22) is used to receive the broadcast type machine text output by the artificial intelligence module (4), and send data or command information to the function chip (222) in the command frame format; The function chip (222) of the broadcast unit (22) is used for speech synthesis and pre-made playback, and processes the broadcast-type machine text in the command frame format sent by the main control chip (221) into human speech for broadcasting.

8. The environment perception interaction system for visually impaired patients according to claim 1, Features: The object location information includes a quantitative expression of distance.

Images