Character and traffic light recognition system based on AR (Augmented Reality) technology and working method

Abstract

The invention discloses a character and traffic light recognition system based on an AR technology and a working method, and particularly relates to the technical field of deep learning target detection algorithm application and AR glasses application. The system comprises an image acquisition module (1), a central processing module (2) and an image display module (3), wherein the image acquisition module composed of the AR equipment camera acquires real-time video streams, and the image acquisition module is connected with the central processing module to process the video streams. According to the invention, a deep learning target detection model for character recognition and traffic light recognition is implanted into the target detection module in the central processing module, so that detection and recognition of characters and traffic lights in a real-time video stream are realized. And the output of the central processing module is connected with the image display module, and video stream information processed by the central processing module is transmitted to the image display module, namely, an AR equipment display terminal for display, so that the old and other groups with visual impairment are helped to identify a target detection object, namely characters or traffic lights.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning target detection algorithm application and AR glasses application technology. Specifically, it relates to an AR technology-based text and traffic light recognition system that helps the elderly and other people with visual impairments to recognize specific objects. Background Art

[0002] AR glasses, or augmented reality glasses, are wearable devices that integrate virtual information with the real world. They overlay computer-generated images on the user's field of view through optical components, allowing users to see virtual objects in the real world. These virtual objects can be text, images, videos, 3D models, etc. They can interact and overlap with objects in the real world to bring users a rich, three-dimensional augmented reality experience. In recent years, with the continuous development of AR and chip technology, the functions of AR glasses have been continuously improved, the computing power of AR devices has been continuously improved, the experience of AR glasses has gradually improved, and AR glasses have been used in more and more fields.

[0003] Deep learning is a machine learning technology that simulates the learning process of the human brain through neural networks. It trains the model with a large amount of data so that it can automatically learn and extract features from the data for prediction and classification. Deep learning has achieved remarkable results in computer vision, natural language processing, speech recognition and other fields. Object detection algorithm is an important task in the field of computer vision, which aims to identify and locate the target objects in the image from the image or video. Object detection algorithms based on deep learning, such as R-CNN, Fast / Faster R-CNN, YOLO, SSD, etc., automatically learn image features through neural networks to achieve object detection. This type of algorithm has good performance in processing complex scenes, small targets and real-time detection. In recent years, object detection algorithms based on deep learning have been widely studied and applied, and the performance of deep learning object detection algorithms on various public data sets has been continuously improved.

[0004] Elderly people with visual impairments need to be exposed to a large amount of text information and various signs in natural life scenes. They have difficulty observing and identifying smaller text nearby or distant signs (such as traffic lights). The usual solution is to correct it by wearing different glasses, but this method is not smart and flexible enough when processing certain specific information such as text in a natural environment or traffic light signs in the distance. This application is the first attempt to help elderly people with visual impairments recognize the text and traffic light signs mentioned above by deploying a deep learning model in AR glasses and designing corresponding applications. Summary of the invention

[0005] Technical problem: In view of the above-mentioned defects or deficiencies in the prior art, the present invention aims to provide a text and traffic light recognition system and working method based on AR technology, combining deep learning target detection algorithm and AR technology, by deploying the deep learning model on AR glasses and designing corresponding human-computer interaction programs to help the elderly and other groups with visual impairments recognize text information and traffic signs in life.

[0006] Technical solution: To achieve the above-mentioned purpose, the text and traffic light recognition system based on AR technology of the present invention includes an image acquisition module, a central processing module, and an image display module, wherein the image acquisition module is composed of an AR device camera to collect real-time video stream, and the video stream information output by the image acquisition module is connected to the central processing module, and the video stream information processed by the central processing module is transmitted to the image display module composed of an AR device display terminal for display.

[0007] The text and traffic light recognition system based on AR technology according to claim 1 is characterized in that the image acquisition module includes a built-in camera of the AR device, a mobile camera or a mobile phone.

[0008] The central processing module includes an image preprocessing module, a target detection module, and a human-computer interaction program module connected in sequence; wherein the image preprocessing module includes a video decoding module, an image cropping module, and an image filtering connected in sequence; the target detection module includes a model switching module, image detection, and a deep learning target detection model, wherein the deep learning target detection model includes a text detection model and a traffic light detection model; the human-computer interaction program module includes a device stability detection module, a detection information receiving module, an image magnification module, a brightness adjustment module, and a contrast adjustment module.

[0009] The image display module includes an AR device display terminal composed of a micro display, an optical waveguide and an optical lens.

[0010] The working method of the text and traffic light recognition system based on AR technology of the present invention is as follows:

[0011] Step 1: The user wears the AR device to collect real-time video information:

[0012] When the image acquisition module works normally, the AR device collects real-time video information through the camera, and by wearing the AR device correctly, the image collected by the camera is consistent with the wearer's visual field;

[0013] Step 2: Perform image preprocessing operations such as decoding, cropping, and filtering on the collected real-time video stream:

[0014] The real-time video stream collected by the image acquisition module is transmitted to the image preprocessing module in the central processing module; the video decoding module in the image preprocessing module decodes each frame of the image from the video stream and transmits it to the image cropping module to crop each frame of the image, and the image after the cropping operation is transmitted to the image filtering module to obtain processed image information, and the median filtering and Gaussian filtering algorithms are used in this module to reduce image noise and improve image quality;

[0015] Step 3: Detect the target object on the preprocessed image to obtain the target object’s category information and location information in the image:

[0016] The image preprocessing module is connected to the target detection module to transmit the processed image information to the target detection module; a deep learning target detection model is implanted in the target detection module to realize the detection and recognition of the corresponding target detection objects - text and traffic lights. The deep learning target detection model implanted in this part includes a text detection model and a traffic light detection model, so that the text detection model or traffic light detection can be switched through the model switching module; the target detection object is detected in the image detection module and the category information of the target object and the location information of the target object in the image are obtained, and the category information and location information are transmitted to the detection information receiving module in the human-computer interaction program module;

[0017] Step 4: Implement human-computer interaction operations for image magnification and image clarity through human-computer interaction programs:

[0018] The target detection module is connected to the human-computer interaction program module, and transmits the category information of the target detection object, the position information of the target object in the image, and the image information to the detection information receiving module in the human-computer interaction program module; the human-computer interaction program module determines whether to enlarge the image and perform image sharpening operations according to the category information of the target detection object and the position information of the target object in the image; wherein the image sharpening operation includes an image brightness adjustment operation and an image contrast adjustment operation; the image enlargement operation is implemented by the image enlargement module, and the image brightness adjustment and sharpness adjustment are implemented by the brightness adjustment module and the contrast adjustment module respectively;

[0019] Step 5: The image processed in step 4 is displayed by the AR device:

[0020] The human-computer interaction program module is connected to the image display module, and the image information processed in step 4 is transmitted to the display terminal of the AR device for display; the user observes the enlarged image content of the target detection object on the AR device.

[0021] The workflow of the human-computer interaction program module is:

[0022] Step 1: After the device is started, the device stability detection module in the human-computer interaction program module works; the program detects whether the mobile device, i.e., the AR device, is in a stable state. When the AR device is in a stable state, the program considers that the user has worn the AR device and is in a calm state; when the AR device is not in a stable state, the program considers that the user has not worn the AR device, and the program will continue with step 1 until the device is stable;

[0023] Step 2: After the program determines in the above step 1 that the user has worn the AR device, the detection information receiving module in the human-computer interaction program module works; the program determines whether there is a target detection object - text and traffic lights - in the area of ​​interest based on the target object category and position information received by the detection information receiving module; wherein the area of ​​interest is a rectangular area defined by the central position of the human-computer interaction program interface; after the user wears the AR device, the rectangular boundary box of the area of ​​interest will be displayed in the center of the user's field of view, and the rectangular boundary box displayed by the AR device will overlap with the field of view observed by the human eye. When the user wears the AR device, the content in the human eye field of view within the rectangular box will be detected; if there is a target detection object in the area of ​​interest, proceed to the following step 3, if no target detection object is detected, the program interface in this step does not change;

[0024] Step 3: In this step, the image magnification module, brightness adjustment module, and contrast adjustment module in the human-computer interaction program module work; if there is a target object to be detected in the region of interest in step 2, the program will magnify, adjust the brightness, and adjust the contrast of the content in the region of interest; the image generated after the above series of operations is displayed on the AR device display terminal, and the image quality will be improved to a certain extent; once the magnification, brightness adjustment, and contrast adjustment operations are performed, the program will perform the judgment in the following step 4;

[0025] Step 4. In this step, the detection information receiving module in the human-computer interaction program module works; after the operation in step 3 occurs, the program continues to determine whether there is a target detection object in the area of ​​interest for a certain period of time; if there is no target detection object in the area of ​​interest, the program will proceed to step 1; if there is a target detection object in the area of ​​interest, the program will continue to repeat the operation in step 3 until there is no target detection object in the area of ​​interest, and return to execute step 1.

[0026] Beneficial effects: Compared with the prior art, the advantages of the present invention are as follows:

[0027] 1. Implant the deep learning target detection model into the AR device, and use the deep learning model to identify the target detection objects - text information and traffic lights, to help filter out irrelevant information, and only operate on specific information. In addition, different deep learning target detection models can be switched within the program to achieve detection and identification of different target detection objects. For the elderly and other visually impaired groups, this method is more flexible and intelligent than the traditional method of wearing glasses to correct vision.

[0028] 2. Collect real-time video information through the camera of the AR device, process the video information in real time, and optimize the display quality of the image in real time. Use AR applications to process virtual information and display it on the AR device. Users can complete information collection, processing, display and other operations by wearing an AR device, which reduces the complexity of the system, makes the device light and convenient for the wearer to use. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention and do not constitute a limitation of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of this specification and the accompanying drawings, or directly or indirectly applied in other related technical fields, are also included in the protection scope of the present invention. In the accompanying drawings:

[0030] Figure 1 The system structure diagram disclosed in the present invention is shown.

[0031] Figure 2 A flowchart showing the human-computer interaction program module disclosed in the present invention,

[0032] Figure 3 A schematic diagram showing the working of the device disclosed in the present invention;

[0033] The accompanying drawings are marked as: image acquisition module 1, central processing module 2, image display module 3, image preprocessing module 21, target detection module 22, human-computer interaction program module 23, video decoding module 211, image cropping module 212, image filtering module 213, model switching module 221, image detection module 222, deep learning target detection model 223, text detection model 2231, traffic light detection model 2232, equipment stability detection module 231, detection information receiving module 232, image magnification module 233, brightness adjustment module 234, contrast adjustment module 235. DETAILED DESCRIPTION

[0034] The following will be combined with the accompanying drawings in the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] The image acquisition module 1 uses a monocular or binocular camera with wired or wireless transmission;

[0036] The central processing module 2 is a computer or edge computing module using a central processing unit (CPU), a video processing unit (GPU), a memory, a data bus, and an external device interface;

[0037] The image display module 3 uses a liquid crystal, OLED or MircoLED display.

[0038] Example 1

[0039] This example provides a text and traffic light recognition system based on AR glasses and its working method. The text and traffic light recognition system based on AR technology includes three parts: an image acquisition module 1, a central processing module 2, and an image display module 3. Among them, the image acquisition module 1 composed of an AR device camera acquires a real-time video stream, the image acquisition module 1 is connected to the central processing module 2, the output of the central processing module 2 is connected to the image display module 3, and the video stream information processed by the central processing module 2 is transmitted to the image display module 3 composed of the AR device display terminal for display. The image acquisition module 1 is composed of a built-in camera of the AR device, which includes an AR device camera but is not limited to using an AR device camera, and other mobile devices such as mobile phones, mobile cameras, etc. can be used. The central processing module 2 includes: an image preprocessing module 21, a target detection module 22, and a human-computer interaction program module 23. Among them, the image preprocessing module 21 includes: video decoding module 211, image cropping module 212, image filtering module 213; the target detection module 22 includes model switching module 221, image detection module 222, deep learning target detection model 223; the human-computer interaction program module 23 includes: device stability detection module 231, detection information receiving module 232, image magnification module 233, brightness adjustment module 234, contrast adjustment module 235; the deep learning target detection model 223 includes text detection model 2231, traffic light detection model 2232. The image display module 3 is composed of an AR device display terminal, including devices such as a micro display, an optical waveguide, and an optical lens.

[0040] The working method of the text and traffic light recognition system based on AR glasses is as follows:

[0041] Step 1: The wearer wears the AR device to collect real-time video information:

[0042] The image acquisition module 1 is composed of a built-in camera in the AR device. When the device is working normally, the AR device collects real-time video information through the camera. By wearing the AR device correctly, the image collected by the camera is consistent with the wearer's human eye field content.

[0043] Step 2: Perform image preprocessing operations such as decoding, cropping, filtering, etc. on the collected real-time video stream:

[0044] The image acquisition module 1 is connected to the image preprocessing module 21 of the central processing module 2, and the acquired real-time video stream is transmitted to the image preprocessing module 21. The video decoding module 211 decodes each frame of the image from the video stream and transmits it to the image cropping module 212 to crop each frame of the image. The image after the cropping operation is transmitted to the image filtering module 213, in which the median filtering and Gaussian filtering algorithms are used to reduce image noise and improve image quality.

[0045] Step 3: Detect the target object on the preprocessed image to obtain the target object’s category information and location information in the image:

[0046] The image preprocessing module 21 is connected to the target detection module 22, and transmits the processed image information to the target detection module 22. The deep learning target detection model 223 implanted in the target detection module 22 can realize the detection and recognition of the corresponding target detection objects - text and traffic lights. The deep learning target detection model 223 implanted in this part includes a text detection model 2231 and a traffic light detection model 2232, and the text detection model 2231 or the traffic light detection model 2232 can be switched through the model switching module 221. The target detection object is detected in the image detection module 222, and the category information of the target object and the location information of the target object in the image are obtained, and these category information and location information are transmitted to the detection information receiving module 232 in the human-computer interaction program module 23.

[0047] Step 4: Implement human-computer interaction operations such as image magnification and image clarity through the human-computer interaction program:

[0048] The target detection module connection 22 is connected to the human-computer interaction program module 23 to transmit the category information of the target detection object, the position information of the target object in the image, and the image information to the detection information receiving module 232 in the human-computer interaction module. The human-computer interaction program module 23 determines whether to enlarge the image and perform image sharpening operations based on the category information of the target detection object and the position information of the target object in the image. Since the image quality deteriorates to a certain extent after the image is enlarged, the image sharpening operation is a necessary operation after the image enlargement operation. Among them, the image sharpening operation includes the image brightness adjustment operation and the image contrast adjustment operation. The image enlargement operation is implemented by the image enlargement module 233, and the image brightness adjustment and sharpness adjustment are implemented by the brightness adjustment module 234 and the contrast adjustment module 235 respectively.

[0049] Step 5: The image processed in step 4 is displayed by the AR device:

[0050] The human-computer interaction program module 23 is connected to the image display module 3, and the image information processed in step 4 is transmitted to the display terminal of the AR device for display. The wearer observes the enlarged image content of the target detection object on the AR device.

[0051] In summary, the present invention realizes the real-time acquisition of video streams through the built-in camera of the AR device, and after the input video stream is cropped and other operations are performed, the image information is detected and identified by the implanted deep learning target recognition model. A human-computer interaction program is designed to realize functions such as image amplification, brightness and contrast adjustment, improve the quality of the image, amplify the content of the target detection object, and facilitate the wearer to identify the target detection object. Finally, the system outputs the processed image through the AR device display terminal, realizes the acquisition, processing, and output of image information by the AR device, and ensures the portability of the device. By implanting the deep learning model into the AR device, the recognition of specific target detection objects is realized, and most of the irrelevant information in the daily life application environment is filtered out. Compared with traditional vision correction methods, it has its unique flexibility and convenience.

[0052] Example 2

[0053] This embodiment provides the workflow of the human-computer interaction program module 3:

[0054] Step 1: After the device is started, the device stability detection module 231 in the human-computer interaction program module 23 works. The program detects whether the mobile device, i.e., the AR device, is in a stable state. When the AR device is in a stable state, the program assumes that the wearer has worn the AR device and is in a calm state. When the AR device is not in a stable state, the program assumes that the wearer has not worn the AR device, and the program will continue with step 1 until the device is stable, and the program will proceed to step 2.

[0055] In step 2 and step 1, after the program determines that the wearer has put on the AR device, the detection information receiving module 232 in the human-computer interaction program module 23 works. The program determines whether there is a target detection object - text and traffic lights - in the area of ​​interest based on the target object category and position information received by the detection information receiving module 232. Among them, the area of ​​interest is a rectangular area defined by the central position of the human-computer interaction program interface. After the wearer wears the AR device, the rectangular bounding box of the area of ​​interest will be displayed in the center of the wearer's field of view. The rectangular box displayed by the AR device will overlap with the field of view observed by the human eye. When the wearer wears the AR device, the content in the human eye field of view within the rectangular bounding box will be detected. If there is a target detection object in the area of ​​interest, proceed to step 3. If the target detection object is not detected, the program interface does not change in this step.

[0056] Step 3. In this step, the image magnification module 233, the brightness adjustment module 234, and the contrast adjustment module 235 in the human-computer interaction program module 23 work. If there is a target object to be detected in the region of interest in step 2, the program will magnify, adjust the brightness, and adjust the contrast of the content in the region of interest. The image generated after the above series of operations is displayed on the AR device display terminal, and the image quality will be improved to a certain extent. Once the magnification, brightness adjustment, and contrast adjustment operations are performed, the program will perform the judgment in step 4.

[0057] Step 4: In this step, the detection information receiving module 232 in the human-computer interaction program module 23 works. After the operation in step 3 occurs, the program continues to determine whether there is a target detection object in the region of interest for a certain period of time. If there is no longer a target detection object in the region of interest, the program will proceed to step 1; if there is a target detection object in the region of interest, the program will continue to repeat the operation in step 3 until there is no longer a target detection object in the region of interest, and then return to step 1.

[0058] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A text and traffic light recognition system based on AR technology, characterized by: The invention comprises an image acquisition module (1), a central processing module (2), and an image display module (3), wherein the image acquisition module (1) is composed of an AR device camera to acquire a real-time video stream, the video stream information output by the image acquisition module (1) is connected to the central processing module (2), and the video stream information processed by the central processing module (2) is transmitted to the image display module (3) composed of an AR device display terminal for display.

2. The text and traffic light recognition system based on AR technology according to claim 1, characterized in that: The image acquisition module (1) comprises a camera built into an AR device, a mobile camera or a mobile phone.

3. The text and traffic light recognition system based on AR technology according to claim 1, characterized in that: The central processing module (2) comprises an image preprocessing module (21), a target detection module (22), and a human-computer interaction program module (23) which are connected in sequence; wherein the image preprocessing module (21) comprises a video decoding module (211), an image cropping module (212), and an image filtering module (213) which are connected in sequence; the target detection module (22) comprises a model switching module (221), an image detection module (222), and a deep learning target detection model (223); wherein the deep learning target detection model (223) comprises a text detection model (2231) and a traffic light detection model (2232); and the human-computer interaction program module (23) comprises a device stability detection module (231), a detection information receiving module (232), an image magnification module (233), a brightness adjustment module (234), and a contrast adjustment module (235).

4. The text and traffic light recognition system based on AR technology according to claim 1, characterized in that: The image display module (3) comprises an AR device display terminal consisting of a micro display, an optical waveguide and an optical lens.

5. A method for operating the text and traffic light recognition system based on AR technology as claimed in claim 1, 2, 3 or 4, characterized in that The method is as follows: Step 1: The user wears the AR device to collect real-time video information: When the image acquisition module (1) works normally, the AR device acquires real-time video information through the camera, and by correctly wearing the AR device, the image acquired by the camera is consistent with the content of the wearer's human eye field; Step 2: Perform image preprocessing operations such as decoding, cropping, and filtering on the collected real-time video stream: The real-time video stream collected by the image acquisition module (1) is transmitted to the image preprocessing module (21) in the central processing module (2); the video decoding module (211) in the image preprocessing module (21) decodes each frame of image from the video stream and transmits it to the image cropping module (212) to crop each frame of image, and the image after the cropping operation is transmitted to the image filtering module (213) to obtain processed image information, and the median filtering and Gaussian filtering algorithms are used in this module to reduce image noise and improve image quality; Step 3: Detect the target object on the preprocessed image to obtain the target object’s category information and location information in the image: The image preprocessing module (21) is connected to the target detection module (22) to transmit the processed image information to the target detection module (22); a deep learning target detection model (223) is implanted in the target detection module (22) to realize the detection and recognition of the corresponding target detection objects - text and traffic lights, and the deep learning target detection model (223) implanted in this part includes a text detection model (2231) and a traffic light detection model (2232), so that the text detection model (2231) or the traffic light detection model (2232) can be switched through the model switching module (221); the target detection object is detected in the image detection module (222) and the category information of the target object and the location information of the target object in the image are obtained, and the category information and location information are transmitted to the detection information receiving module (232) in the human-computer interaction program module (23); Step 4: Implement human-computer interaction operations for image magnification and image clarity through human-computer interaction programs: The target detection module (22) is connected to the human-computer interaction program module (23) to transmit the category information of the target detection object, the position information of the target object in the image and the image information to the detection information receiving module (232) in the human-computer interaction program module (23); the human-computer interaction program module (23) determines whether to perform an image magnification and image clarity operation on the image according to the category information of the target detection object and the position information of the target object in the image; wherein the image clarity operation includes an image brightness adjustment operation and an image contrast adjustment operation; the image magnification operation is implemented by the image magnification module (233), and the image brightness adjustment and clarity adjustment are implemented by the brightness adjustment module (234) and the contrast adjustment module (235) respectively; Step 5: The image processed in step 4 is displayed by the AR device: The human-computer interaction program module (23) is connected to the image display module (3), and the image information processed in step 4 is transmitted to the display terminal of the AR device for display; the user observes the enlarged image content of the target detection object on the AR device.

6. The working method of the text and traffic light recognition system based on AR technology according to claim 5, characterized in that: The workflow of the human-computer interaction program module (23) is as follows: Step 1: After the device is started, the device stability detection module (231) in the human-computer interaction program module (23) works; the program detects whether the mobile device, i.e., the AR device, is in a stable state. When the AR device is in a stable state, the program considers that the user has worn the AR device and is in a calm state; when the AR device is not in a stable state, the program considers that the user has not worn the AR device, and the program will continue to perform step 1 until the device is stable. Step 2: After the program determines in the above step 1 that the user has worn the AR device, the detection information receiving module (232) in the human-computer interaction program module (23) operates; the program determines whether there is a target detection object - text and traffic lights - in the area of ​​interest based on the target object category and position information received by the detection information receiving module (232); wherein the area of ​​interest is a rectangular area defined by the central position of the human-computer interaction program interface; after the user wears the AR device, the rectangular boundary box of the area of ​​interest will be displayed in the center of the user's field of vision, and the rectangular boundary box displayed by the AR device will overlap with the field of vision observed by the human eye. When the user wears the AR device, the content in the human eye field of vision within the rectangular box will be detected; if there is a target detection object in the area of ​​interest, the following step 3 is performed; if no target detection object is detected, the program interface in this step does not change; Step 3: In this step, the image magnification module (233), the brightness adjustment module (234), and the contrast adjustment module (235) in the human-computer interaction program module (23) work; if the target object to be detected exists in the region of interest in step 2, the program will magnify, adjust the brightness, and adjust the contrast of the content in the region of interest; the image generated after the above series of operations is displayed on the AR device display terminal, and the image quality will be improved to a certain extent; once the magnification, brightness adjustment, and contrast adjustment operations are performed, the program will perform the judgment in the following step 4; Step 4. In this step, the detection information receiving module (232) in the human-computer interaction program module (23) works; after the operation in step 3 occurs, the program continues to determine whether there is a target detection object in the region of interest for a certain period of time; if there is no target detection object in the region of interest, the program will proceed to step 1; if there is a target detection object in the region of interest, the program will continue to repeat the operation in step 3 until there is no target detection object in the region of interest, and return to execute step 1.

Images