Tactile image recognition method and system based on YOLOv8 and voice broadcast
Through the tactile image recognition method based on YOLOv8, combined with voice broadcast technology, the problem of difficulty in recognizing tactile images for congenitally blind people is solved, their image understanding and learning efficiency are improved, and their cognitive abilities are enhanced.
Patent Information
- Application Number
- CN202411732979.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-09-29
- Filing Date
- 2024-11-29
- Publication Date
- 2025-01-21
AI Technical Summary
Congenital blind people have difficulty accurately identifying tactile images, resulting in obstacles in understanding and learning image information.
A tactile image recognition method based on YOLOv8 and voice broadcast is adopted. By acquiring and preprocessing the tactile image data set, the YOLOv8 model is trained to detect the position of the fingertips of the blind person in real time, and a voice broadcast is performed when the recognition results coincide.
It improves the blind people's understanding and learning efficiency of tactile images, allowing them to better understand and learn different forms of physical concepts, thereby improving their cognitive level.
Smart Images
Figure CN119339046A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and more specifically, to a tactile image recognition method and system based on YOLOv8 and voice broadcast. Background Art
[0002] Information is the basis for people to acquire knowledge and communicate with each other. However, for the blind and other visually impaired people, due to their impaired vision, they can only obtain external information through other sensory abilities, especially touch and hearing. When faced with text information, they can still understand the information through passive voice reading or active reading of Braille. When faced with image information, they cannot directly obtain the image content, especially congenitally blind people. Their cognition of things is usually memorized and understood through the descriptions of others. When there are many abstract descriptions, it will cause great obstacles to their memory and understanding. This problem has not been well solved.
[0003] With the development and progress of the information age, traditional text information will gradually decrease, and more Internet multimedia information will be available. The Internet generally uses graphical interfaces for interaction, including illustrations, videos, etc., which greatly increases the need for the blind and other visually impaired people to understand graphic images. Braille displays and tactile displays are now available on the market to help the visually impaired obtain text and graphic information more conveniently and quickly. However, the blind cannot browse the complete image like sighted people, and it is difficult for them to perceive the structural form of things in the image. In the process of using tactile or contact displays, they can only rely on their own memory and imagination, especially when encountering complex images, which are easy to confuse and difficult to understand, resulting in limitations in the learning and cognition of the blind.
[0004] Due to the particularity of congenitally blind people, they have never had a visual concept of the physical world. Since they can remember, they have basically understood the world through the descriptions of others and reading Braille. With the development of computer vision technology, it is necessary to design a method and system for congenitally blind people that can automatically identify tactile image structures and voice broadcast them, which is of great significance to their lives and work. It is also of positive significance to let them understand and learn the forms of things around them from an early age and integrate into the living world of sighted people to establish their worldview. Summary of the invention
[0005] In order to overcome the defect in the prior art that it is difficult for congenitally blind people to accurately identify tactile images, the present invention provides a tactile image recognition method and system based on YOLOv8 and voice broadcast, which can automatically identify the tactile image area where the fingers of congenitally blind people stay, and use voice broadcast to remind them, thereby improving the blind people's understanding of tactile images and learning efficiency, allowing blind users to further understand and learn physical concepts of different forms, and improve their cognitive level to a certain extent.
[0006] In order to solve the above technical problems, the technical solution of the present invention is as follows:
[0007] A tactile image recognition method based on YOLOv8 and voice broadcast, including the following methods:
[0008] S1: Obtain tactile images fused with visual information, take several photos of different tactile images, construct a tactile image dataset and perform preprocessing;
[0009] S2: Build the YOLOv8 model based on machine learning algorithm;
[0010] Using the preprocessed tactile image dataset to train the YOLOv8 model to obtain a trained YOLOv8 model;
[0011] S3: Using the trained YOLOv8 model to detect the tactile image to be recognized by the blind person in real time, and obtaining the recognition result of each specific position in the tactile image to be recognized;
[0012] S4: Detect and track the position of the fingertips of the blind person's fingers in real time. When the position of the fingertips of the blind person's fingers coincides with any specific position in the tactile image to be identified, the recognition result of the specific position is outputted by voice broadcast to complete the recognition of the tactile image.
[0013] Preferably, in step S1, obtaining a tactile image fused with visual information includes:
[0014] Visual graphics construction: Analyze the features of physical entities and simplify them to obtain simplified features; decompose the physical entities according to the simplified features; convert all decomposed structures into basic geometric figures that can be understood by the blind; select specific angles that can express all simplified features, reorganize all basic geometric figures at the specific angles to obtain visual graphics;
[0015] Tactile image construction: distributing touch points on all basic geometric figure contours of the visual figure to obtain the tactile image integrated with the visual information.
[0016] Preferably, in step S1, constructing a tactile image dataset includes:
[0017] Under different conditions, several photos of different tactile images are taken to construct a tactile image dataset;
[0018] The conditions include: any one of resolution, lighting conditions, shooting angle and shooting distance.
[0019] Preferably, in step S1, the preprocessing includes:
[0020] For each photo of the tactile image, use the Make Sense annotation tool to perform online data annotation in the YOLO format to obtain an annotated tactile image dataset;
[0021] Adjust all photos in the annotated tactile image dataset to a uniform size, remove photos with incorrect annotations and resolutions below a preset threshold, and obtain a tactile image dataset after data removal;
[0022] Performing data enhancement on the tactile image dataset after data elimination to complete the preprocessing;
[0023] The data enhancement includes: any one of random flipping, random cropping, color jittering and Mosaic data enhancement.
[0024] Preferably, in step S2, a transfer learning strategy is adopted to train the YOLOv8 model: the YOLOv8 model is initialized using weights pre-trained on the COCO dataset, and the parameters of the YOLOv8 model are fine-tuned based on a grid search method during the training process.
[0025] Preferably, in step S2, the total loss function is set Used to supervise the YOLOv8 model for training;
[0026] The total loss function include:
[0027]
[0028] Wherein, λ1 and λ2 are the first and second weight coefficients respectively, λ1+λ2=1; is the classification loss function; is the bounding box regression loss function
[0029] The classification loss function Specifically, it is the BCE binary cross entropy loss, and the formula is as follows:
[0030]
[0031] Among them, y i is the label of sample i, the positive class is 1, the negative class is 0, and N is the number of samples; pi The probability of predicting sample i as a positive class;
[0032] The bounding box regression loss function Including WIoU loss + DFL loss, the formula is as follows:
[0033]
[0034] W g =w+w gt -W i
[0035] H g =h+h gt -H i
[0036]
[0037] Among them, λ3 and λ4 are the third and fourth weight coefficients respectively, λ3+λ4=1; (x, y) and (x gt ,y gt ) are the horizontal and vertical coordinates of the center points of the predicted box and the true box respectively; w and h are the width and height of the predicted box respectively; w gt 、h gt are the width and height of the real frame respectively; w i , H i are the lengths of the overlapping parts of the width and height of the predicted box and the real box respectively; are the predicted value and the near-predicted value output by the YOLOv8 model; y′, y′ i and y′ i+1 They are the actual value of the label, the integral value of the label, and the integral value of the adjacent labels;
[0038] When the total loss function When the minimum value is obtained, the trained YOLOv8 model is obtained.
[0039] Preferably, in step S2, after obtaining the trained YOLOv8 model, the following steps are further included:
[0040] Model conversion: Convert the trained YOLOv8 model to TensorFlow Lite format;
[0041] Quantization: Dynamically quantize the YOLOv8 model in TensorFlow Lite format;
[0042] Mobile deployment: The dynamically quantized YOLOv8 model is deployed to a mobile device for operation; the mobile device is specifically a mobile terminal device based on the Android operating system.
[0043] Preferably, in step S4, the Hands module of MediaPipe is used to detect and track the key points of the blind person's hands in real time, and to identify the positions of the blind person's fingertips in real time;
[0044] When the position of the fingertip of the blind person coincides with any specific position in the tactile image to be identified, the recognition result of the specific position is outputted by voice broadcast.
[0045] Preferably, in step S4, the recognition result is parsed into a text description, and the text description is converted into speech using Google Text-to-Speech technology and outputted as a broadcast.
[0046] The present invention also provides a tactile image recognition system based on YOLOv8 and voice broadcast, which uses the above-mentioned tactile image recognition method based on YOLOv8 and voice broadcast, including:
[0047] Data preprocessing unit: used to take a number of photos of different tactile images, construct a tactile image dataset and perform preprocessing;
[0048] Model training unit: used to build the YOLOv8 model based on the machine learning algorithm;
[0049] and for training the YOLOv8 model using the preprocessed tactile image dataset to obtain a trained YOLOv8 model;
[0050] A tactile image recognition unit is used to detect the tactile image to be recognized by the blind person in real time using the trained YOLOv8 model, and obtain the recognition result of each specific position in the tactile image to be recognized;
[0051] Voice broadcast unit: used to detect and track the position of the blind person's fingertips in real time. When the position of the blind person's fingertips coincides with any specific position in the tactile image to be identified, the recognition result of the specific position is output through voice broadcast to complete the recognition of the tactile image.
[0052] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:
[0053] The present invention provides a tactile image recognition method and system based on YOLOv8 and voice broadcast, which first collects and preprocesses a tactile image data set; then builds a YOLOv8 model based on a machine learning algorithm; uses the preprocessed tactile image data set to train the YOLOv8 model to obtain a trained YOLOv8 model; then uses the trained YOLOv8 model to detect a tactile image to be recognized by a blind person in real time, and obtains recognition results of each specific position in the tactile image to be recognized; finally, detects and tracks the position of the fingertip of the blind person's finger in real time, and when the position of the fingertip of the blind person's finger coincides with any specific position in the tactile image to be recognized, outputs the recognition result of the specific position through voice broadcast, and completes the recognition of the tactile image;
[0054] Through the trained YOLOv8 model, the present invention can automatically identify the tactile image area where the fingers of congenitally blind people rest, and make real-time voice broadcast reminders based on the recognition results, thereby improving the blind people's understanding of tactile images and learning efficiency, allowing blind users to further understand and learn physical concepts of different forms, and improve their cognitive level to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] Figure 1 This is a flow chart of a tactile image recognition method based on YOLOv8 and voice broadcast provided in Example 1.
[0056] Figure 2 This is a schematic diagram of the “bee” visual graphic provided in Example 1.
[0057] Figure 3 This is a schematic diagram of the “bee” contact information provided in Example 1.
[0058] Figure 4 This is a schematic diagram of the “bee” tactile image provided in Example 1.
[0059] Figure 5 This is a diagram of the YOLOv8 model structure provided in Example 1.
[0060] Figure 6 This is a schematic diagram of the “snail” tactile image provided in Example 2.
[0061] Figure 7 This is a diagram of the training and testing results provided in Example 2.
[0062] Figure 8 This is a schematic diagram of the confusion matrix provided in Example 2.
[0063] Fig. 9 This is an architecture diagram of tactile image recognition based on dual threads on a mobile terminal provided in Example 2.
[0064] Fig.10 This is a schematic diagram of the “snail” tactile image recognition provided in Example 2.
[0065] Fig.11 This is a structural diagram of a tactile image recognition system based on YOLOv8 and voice broadcast provided in Example 3. DETAILED DESCRIPTION
[0066] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;
[0067] In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product;
[0068] It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.
[0069] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.
[0070] Example 1
[0071] like Figure 1 As shown, this embodiment provides a tactile image recognition method based on YOLOv8 and voice broadcast, including the following methods:
[0072] S1: Obtain tactile images fused with visual information, take several photos of different tactile images, construct a tactile image dataset and perform preprocessing;
[0073] S2: Build the YOLOv8 model based on machine learning algorithm;
[0074] Using the preprocessed tactile image dataset to train the YOLOv8 model to obtain a trained YOLOv8 model;
[0075] S3: Using the trained YOLOv8 model to detect the tactile image to be recognized by the blind person in real time, and obtaining the recognition result of each specific position in the tactile image to be recognized;
[0076] S4: detecting and tracking the position of the fingertips of the blind person's fingers in real time, and when the position of the fingertips of the blind person's fingers coincides with any specific position in the tactile image to be identified, outputting the recognition result of the specific position by voice broadcast, thereby completing the recognition of the tactile image;
[0077] In the step S1, obtaining a tactile image fused with visual information includes:
[0078] Visual graphics construction: Analyze the features of physical entities and simplify them to obtain simplified features; decompose the physical entities according to the simplified features; convert all decomposed structures into basic geometric figures that can be understood by the blind; select specific angles that can express all simplified features, reorganize all basic geometric figures at the specific angles to obtain visual graphics;
[0079] Tactile image construction: distributing touch points on all basic geometric figure contours of the visual figure to obtain the tactile image integrated with the visual information.
[0080] In the specific implementation process, firstly, a tactile image fused with visual information is obtained, a number of photos of different tactile images are taken, a tactile image dataset is constructed and preprocessed;
[0081] There are differences in image perception between blind people and ordinary people, so specific touch images need to be set; the specific differences are as follows:
[0082] 1) Unlike the quick response of ordinary people's visual recognition, the time of tactile image recognition is affected by the size and difficulty of the graphics. Graphics that are too large or too small and complex consume a long time, while graphics with moderate size and simple touch recognition speed are fast;
[0083] 2) Unlike the process of visual recognition of ordinary people, which is to first perceive the whole and then the parts, tactile cognition has the cognitive characteristics of cognition from the parts to the whole. Visually impaired people usually have the habit of touching with their left hand fixed and the right hand along the line, repeatedly several times, to find the relative position relationship of the point or block surface structure. Only after all the local features are touched can they establish the perception of the complete object;
[0084] 3) Unlike the fine processing of ordinary people's vision, the cognitive resolution of touch is much lower than that of vision. Visually impaired children need to determine information through points one by one, and the minimum recognition threshold between two points is 2.5mm;
[0085] 4) Unlike ordinary people who have a lot of visual experience in life, the lack of image and graphic recognition experience leads to their lack of tactile recognition ability for complex graphics. For example, the lack of depth clue experience makes them unable to understand the perspective relationship on the plane. For example, the more basic graphic tactile experience a visually impaired person has, the higher the finger sensitivity;
[0086] In this embodiment, a visual graphic is first designed, and then a touch image with a clear structure is constructed based on the visual graphic;
[0087] In visual graphic design, the focus is on component structure and object feature design, and feature analysis-angle selection-structure splitting-geometry-reorganization design is a standardized design process; taking the design of bees as an example, the specific steps are as follows:
[0088] The first step is to analyze the characteristics of bees that people remember, such as stingers, stripes, transparent wings, compound eyes, etc.
[0089] Step 2: Choose a suitable angle that can show these features;
[0090] Step 3: Decompose the object and disassemble the bee into head, body, wings, feet and tail;
[0091] Step 4: Convert the components into basic geometric shapes. Since geometric shapes with more than certain edges or features are difficult for blind people to recognize, the design focus of this step is to incorporate the object components into generalized and simplified graphics that can be recognized by blind people, and combine them with tactile features.
[0092] Step 5: On the basis of maintaining the geometric structure, add simplified features of the bee, such as large, round and compound eyes, roughly triangular wings with a certain line structure, an elliptical body with horizontal line features, and a thin and long needle at the tail;
[0093] The visual graphic of "bee" in this embodiment is as follows Figure 2 As shown;
[0094] For the design of tactile images, the distribution of touch information is designed according to the key information points of the visual graphics, and the touch points of different sizes and spacing are used to express the detailed features, such as Figure 3 As shown in the figure; for example, the turning points are all pointed with points, the line segments all have clear starting points and end points, the contact spacing at the junction of the main and secondary contours is distinguished, the contacts in the main pattern area are filled, and the contacts at the tip of the graphic are merged; the size of the points and the use of dot lines are standardized; for example, the sizes of the points used are small points with a diameter of 1.0mm, midpoints with a diameter of 1.5mm, and large points with a diameter of 3.0mm; the outer contour of the graphic is uniformly expressed by dot lines composed of midpoints; the inner contour of the graphic is uniformly expressed by dot lines composed of small points; the interval between different components is not less than 5mm, etc.; and finally Figure 3 The contact information in Figure 2 The visual graphics in the image are integrated to form Figure 4 The tactile image shown contains both visual information and contact information, and can be recognized by a machine vision system and can also be perceived by a blind person through touch;
[0095] Then, the visually impaired person is asked to perform touch recognition and tactile correction, and the tactile image is considered to meet the requirements if the object can be recognized within a certain period of time;
[0096] After constructing a specific tactile image, take several tactile image photos under different angles, resolutions, etc., construct a tactile image dataset and perform preprocessing;
[0097] Then build the YOLOv8 model based on the machine learning algorithm;
[0098] YOLOv8 is an advanced target detection algorithm that can achieve low-latency and high-precision target detection. Its network structure is as follows Figure 5 As shown in the figure, the YOLOv8 model consists of three parts: Backbone, Neck and Head:
[0099] Backbone is mainly responsible for extracting high-level semantic features from the input image. The C2f structure combines the advantages of the residual structure and the ELAN structure, adjusts the number of channels and residual connections, achieves lightweight design and rich gradient flow information, and thus improves the feature extraction capability.
[0100] Neck is used to further integrate and adjust the features extracted by Backbone, which is conducive to fusing features at different levels and thus improving the network's ability to perceive targets. At the same time, it uses top-down and bottom-up path aggregation strategies to significantly improve the performance of target detection through multi-scale feature fusion and enhanced feature representation.
[0101] The head part includes a bounding box regression layer and a classification layer, which are responsible for predicting the target position and category respectively. The decoupled head structure is used to separate the classification prediction and the bounding box regression prediction, and detect them in parallel at different scales. At the same time, it is transformed from Anchor-Based to Anchor-Free to reduce the number of parameters and computational complexity, and enhance the generalization ability of the algorithm.
[0102] Then, the preprocessed tactile image dataset is used to train the YOLOv8 model to obtain the trained YOLOv8 model;
[0103] Then, the trained YOLOv8 model is used to detect the tactile images to be recognized by the blind in real time, and the recognition results of each position in the tactile images to be recognized are obtained;
[0104] Finally, the position of the fingertip of the blind person's finger is detected and tracked in real time. When the position of the fingertip of the blind person's finger coincides with any specific marked position in the tactile image to be identified, the recognition result of the specific position is outputted by voice broadcast, thereby completing the recognition of the tactile image;
[0105] Through the trained YOLOv8 model, this method can automatically identify the tactile image area where the fingers of congenitally blind people rest, and make real-time voice broadcast reminders based on the recognition results, thereby improving the blind people's understanding of tactile images and learning efficiency, allowing blind users to further understand and learn physical concepts of different forms and improve their cognitive level to a certain extent.
[0106] Example 2
[0107] This embodiment provides a tactile image recognition method based on YOLOv8 and voice broadcast, including the following methods:
[0108] S1: Obtain tactile images fused with visual information, take several photos of different tactile images, construct a tactile image dataset and perform preprocessing;
[0109] S2: Build the YOLOv8 model based on machine learning algorithm;
[0110] Using the preprocessed tactile image dataset to train the YOLOv8 model to obtain a trained YOLOv8 model;
[0111] S3: Using the trained YOLOv8 model to detect the tactile image to be recognized by the blind person in real time, and obtaining the recognition result of each specific position in the tactile image to be recognized;
[0112] S4: detecting and tracking the position of the fingertips of the blind person's fingers in real time, and when the position of the fingertips of the blind person's fingers coincides with any specific position in the tactile image to be identified, outputting the recognition result of the specific position by voice broadcast, thereby completing the recognition of the tactile image;
[0113] In the step S1, obtaining a tactile image fused with visual information includes:
[0114] Visual graphics construction: Analyze the features of physical entities and simplify them to obtain simplified features; decompose the physical entities according to the simplified features; convert all decomposed structures into basic geometric figures that can be understood by the blind; select specific angles that can express all simplified features, reorganize all basic geometric figures at the specific angles to obtain visual graphics;
[0115] Tactile image construction: distributing touch points on all basic geometric figure contours of the visual figure to obtain the tactile image integrated with the visual information;
[0116] In the step S1, constructing a tactile image dataset includes:
[0117] Under different conditions, several photos of different tactile images are taken to construct a tactile image dataset;
[0118] The conditions include: any one of resolution, lighting conditions, shooting angle and shooting distance;
[0119] In step S1, the preprocessing includes:
[0120] For each photo of the tactile image, use the Make Sense annotation tool to perform online data annotation in the YOLO format to obtain an annotated tactile image dataset;
[0121] Adjust all photos in the annotated tactile image dataset to a uniform size, remove photos with incorrect annotations and resolutions below a preset threshold, and obtain a tactile image dataset after data removal;
[0122] Performing data enhancement on the tactile image dataset after data elimination to complete the preprocessing;
[0123] The data enhancement includes: any one of random flipping, random cropping, color jittering and Mosaic data enhancement;
[0124] In step S2, the YOLOv8 model is trained using a transfer learning strategy: the YOLOv8 model is initialized using weights pre-trained on the COCO dataset, and the parameters of the YOLOv8 model are fine-tuned based on a grid search method during the training process;
[0125] In step S2, the total loss function is set Used to supervise the YOLOv8 model for training;
[0126] The total loss function include:
[0127]
[0128] Wherein, λ1 and λ2 are the first and second weight coefficients respectively, λ1+λ2=1; is the classification loss function; is the bounding box regression loss function
[0129] The classification loss function Specifically, it is the BCE binary cross entropy loss, and the formula is as follows:
[0130]
[0131] Among them, y i is the label of sample i, the positive class is 1, the negative class is 0, and N is the number of samples; p i The probability of predicting sample i as a positive class;
[0132] The bounding box regression loss function Including WIoU loss + DFL loss, the formula is as follows:
[0133]
[0134] W g =w+w gt -W i
[0135] H g =h+hgt -H i
[0136]
[0137] Among them, λ3 and λ4 are the third and fourth weight coefficients respectively, λ3+λ4=1; (x, y) and (x gt ,y gt ) are the horizontal and vertical coordinates of the center points of the predicted box and the true box respectively; w and h are the width and height of the predicted box respectively; w gt 、h gt are the width and height of the real frame respectively; w i , H i are the lengths of the overlapping parts of the width and height of the predicted box and the real box respectively; are the predicted value and the near-predicted value output by the YOLOv8 model; y′, y′ i and y′ i+1 They are the actual value of the label, the integral value of the label, and the integral value of the adjacent labels;
[0138] When the total loss function When the minimum value is obtained, the trained YOLOv8 model is obtained;
[0139] In the step S2, after obtaining the trained YOLOv8 model, the following steps are further included:
[0140] Model conversion: Convert the trained YOLOv8 model to TensorFlow Lite format;
[0141] Quantization: Dynamically quantize the YOLOv8 model in TensorFlow Lite format;
[0142] Mobile terminal deployment: deploy the dynamically quantized YOLOv8 model to a mobile device for operation; the mobile device is specifically a mobile terminal device based on the Android operating system;
[0143] In step S4, the Hands module of MediaPipe is used to detect and track the key points of the hand of the blind person in real time, and the trained YOLOv8 model is used to dynamically identify the graphic category and graphic components in the input image, and the position information returned by the running result is marked and saved; if yes, the running of the YOLOv8 model is stopped, and it is determined whether the fingertip is located in the saved prediction box, and if yes, the voice broadcast is automatically triggered;
[0144] In step S4, the recognition result is parsed into a text description, and the text description is converted into speech using Google Text-to-Speech technology and outputted as a broadcast.
[0145] In the specific implementation process, firstly, a tactile image fused with visual information is obtained, a number of photos of different tactile images are taken, a tactile image dataset is constructed and preprocessed;
[0146] This embodiment constructs 80 complex tactile images. When preparing data, it is necessary to ensure the quality and richness of the data to improve the accuracy and robustness of the model. Data collection is the basic link of YOLOv8 model training. In order to ensure the robustness and generalization ability of the model, this embodiment uses common smartphones with different resolutions to collect images. During the collection process, real usage scenarios are simulated, different lighting conditions are created, and target images are taken from multiple common angles to cover various possible postures of the target. In addition, shooting targets at different distances within a certain range is also considered. This comprehensive data collection strategy can help the model learn the performance of the target in various real scenarios, improve the accuracy and adaptability of detection, and construct a tactile image dataset.
[0147] This is followed by preprocessing, which includes:
[0148] For each photo of the tactile image, the Make Sense annotation tool is used to perform online data annotation in the YOLO format. Data annotation is a key step in converting the original image into structured data that can be used for training. At this stage, the online annotation tool Make Sense allows the annotator to draw a bounding box on the image and assign a corresponding category label to each target. Data annotation focuses on structural explanations based on the graphic characteristics of the blind image and the cognitive characteristics of the blind, and emphasizes the basic structure and typical feature points of the object.
[0149] All photos in the annotated tactile image dataset were resized to a uniform size (640×640). When resizing, the aspect ratio of the image was maintained to avoid deformation of the target. The dataset was carefully checked to remove images with errors or low quality, such as severely blurred or incorrectly annotated images.
[0150] The tactile image dataset after data removal is enhanced by data, such as random horizontal flipping, random cropping, color jittering, etc., to generate more diverse training samples by transforming the original image; in addition, Mosaic data enhancement is introduced to splice 4 different training images into a new image, which greatly increases the context information and small target instances of the image. Through the combined use of these data enhancement methods, YOLOv8 can learn more robust and universal feature representations on a limited training dataset, effectively alleviate the overfitting problem, and improve the detection performance of the model in various complex scenarios;
[0151] Commonly used target detection algorithms include RCNN, SSD and YOLO. This method uses the YOLOv8 algorithm and adopts single forward propagation to achieve real-time detection. It significantly improves the detection accuracy while maintaining high speed, especially in the detection of small targets, and achieves a good balance between accuracy and speed.
[0152] In this embodiment, YOLOv8 also uses technologies such as adaptive anchor box selection and multi-scale training to further improve the model's ability to detect objects of different sizes;
[0153] Then, the preprocessed tactile image dataset is used to train the YOLOv8 model to obtain the trained YOLOv8 model;
[0154] In this embodiment, the data of the target data set is imported into the established algorithm model for training; when training the model, it is necessary to use appropriate optimization algorithms and hyperparameters to improve the accuracy and convergence speed of the model;
[0155] The model building of YOLOv8 relies on the ultralytics library established by its development team. After importing the library, fine-tuning and setting the corresponding parameters, training can begin. The determination of hyperparameters mainly starts from the default values, and grid search is performed on possible hyperparameter combinations.
[0156] Using grid search to tune YOLOv8 parameters can systematically and automatically explore the model's hyperparameter space; this method helps find the best parameter combination for a specific data set, reducing the workload of manual trial and error; in addition, it may also discover some parameter combinations that may be overlooked by manual parameter tuning, potentially improving model performance; when using grid search to tune YOLOv8 hyperparameters, use the Ray Tune library to define the search space to specify the value range and type of each hyperparameter, and then put these hyperparameters into the YOLOv8 model for training, by traversing the discrete value combinations of hyperparameters and selecting the combination with the best performance;
[0157] The advantages of transfer learning are fully utilized in the model training process to accelerate model convergence and improve final performance. The training uses the weights pre-trained on a large-scale target detection dataset (COCO dataset) as initialization. These pre-trained weights have learned rich low-level visual features and high-level semantic information. On this basis, the model is adaptively adjusted according to the data distribution of specific tasks through fine-tuning. In the fine-tuning process, a hierarchical learning rate strategy is adopted, and different learning rates are used for parameters of different layers, so that the shallow feature extraction capability is retained, while the deep semantic representation can better adapt to new tasks. Through this transfer learning method, YOLOv8 can quickly achieve better detection results with less training data and iterations, which is especially suitable for small sample learning scenarios.
[0158] In addition, this embodiment replaces the original CIoU loss function in the YOLOv8 model with a more advanced WIoU loss function; WIoU can dynamically adjust the loss contribution of different samples by introducing an adaptive weight factor, paying special attention to those samples that are difficult to optimize; this improvement is expected to significantly improve the detection accuracy of the model, especially the recognition ability of small targets and partially occluded targets; the adaptive characteristics of WIoU may not only accelerate the convergence speed of the model, but also improve the training stability and model generalization ability; although WIoU may bring a slight increase in computational complexity, its potential performance improvement may far outweigh this cost;
[0159] After the model is trained, it needs to be tested and evaluated. The evaluation indicators include accuracy and recall, precision (AP, mAP), F1 score, IoU, etc. At the same time, this method also visualizes and saves the F1 score curve, precision-recall curve, precision curve, recall curve, confusion matrix, normalized confusion matrix, validation batch label and validation batch prediction diagram, which can more intuitively understand the performance of the model and make further optimization. When testing and evaluating the model, it is necessary to use a variety of evaluation indicators to comprehensively evaluate the performance of the model, and adjust and optimize the model.
[0160] This example uses the “snail” blind image as a sample to perform exploratory optimization on the model. Figure 6 The figure shows a schematic diagram of the tactile image of a “snail”. The model optimized and trained based on the “snail” sample has good performance and performs well on the test sample. The performance index after optimization is shown in the figure below: Figure 7 and 8 As shown;
[0161] like Figure 7 As shown in the figure, from the six pictures on the left, it can be seen that the loss function can quickly converge to a smaller value, whether in the training set or the test set, which indicates that the trained model has a strong ability to locate the sample "snail", determine it as a target and correctly classify it; at the same time, during the training process, the model's Precision (finding the right positive class / all found positive classes) and Recall (how many positive samples are recalled) values are roughly converged to 0.98, indicating that the model has high precision and accuracy; mAP is the performance metric of the contradiction between Precision and Recall as the area enclosed by the two axes, mAP50 represents the average mAP with a threshold greater than 0.5, and mAP50-95 represents the average mAP at different IoU thresholds (from 0.5 to 0.95, step size 0.05). Obviously, both indicators can converge to relatively ideal values, indicating that the trained model can also perform well when Precision and Recall are considered comprehensively;
[0162] like Figure 8 The following is a "confusion matrix", which summarizes the records in the data set in a matrix form according to the two criteria of true category and category judgment predicted by the classification model; Figure 8 It shows that among the 80 test images containing parts or whole snails, the number of misclassified images of the true category "head" is 3, "body" is 2, and "shell" is 0; it can be predicted that the model will also have good prediction performance in practical applications;
[0163] Convert the trained YOLOv8 model to TensorFlow Lite format; TensorFlow Lite is a lightweight TensorFlow solution for mobile and embedded devices; it has the advantages of low latency, low power consumption, and cross-platform, allowing machine learning models to be run on resource-constrained devices, including tasks such as target detection and image classification. Through technologies such as model quantization and model compression, the model size is reduced, and high inference accuracy is maintained to achieve efficient model inference on mobile devices; in this embodiment, the conversion tools provided by Tensorflow Lite, such as TFLiteConverter, are used for model conversion; during the conversion process, the type and size of the input and output tensors are specified to better adapt to the actual application scenario;
[0164] Dynamically quantize the YOLOv8 model in TensorFlow Lite format. Quantization technology is used to convert floating-point numbers into integers, thereby reducing the model size and running time. Tensorflow Lite supports both static and dynamic quantization methods, which may have a certain impact on the accuracy of the model. By weighing the accuracy and efficiency of the quantization process, this method ultimately chooses dynamic quantization.
[0165] Deploy the dynamically quantized YOLOv8 model to a mobile device and use the interpreter provided by Tensorflow Lite to run the model; the mobile device is specifically a smartphone based on the Android operating system; Fig. 9 As shown, this embodiment allocates computing resources to the model according to the size of the core according to the multi-core characteristics of the device and the computing resources required by the model, optimizes the parallel computing efficiency of the model, and improves the detection accuracy;
[0166] Then, the tactile image to be recognized by the blind person is detected in real time, and the recognition result of each position in the tactile image to be recognized is obtained;
[0167] Finally, the Hands module of MediaPipe is used to detect and track 21 key points of the hand in real time. At the same time, combined with the YOLOv8 target detection model, it is possible to identify and locate specific objects in the video stream. In this method, YOLOv8 is used to identify different parts in the tactile image.
[0168] like Fig.10 As shown, this embodiment takes the "snail" tactile image as an example. When the system detects that the blind person's finger enters the picture, it will capture the position of the index fingertip and check whether the position coincides with any specific target position detected by the YOLOv8 model; if it coincides, the system will broadcast the category information of the target through voice to provide instant feedback to the user; this interaction method enables the user to intuitively obtain information about a specific object, creating a unique human-computer interaction experience;
[0169] In this embodiment, the real-time voice assistance function requires the use of speech synthesis technology to convert text into speech and output it to the user; this method uses Google Text-to-Speech technology, which has high-quality speech synthesis effect and fast response speed;
[0170] Implement the real-time speech assistance function of YOLOv8 inference results on Android. First, perform object detection through the integrated YOLOv8 model, parse the detection results into text descriptions, and then use Android's TextToSpeech class to initialize the TTS engine and convert text into speech output through its speak method; at the same time, use appropriate queue modes and throttling mechanisms to ensure a smooth user experience and release resources when not needed; this can provide users with real-time speech feedback, enhance the accessibility and practicality of the application, and is particularly suitable for visually impaired users;
[0171] The implementation of this method involves multi-threaded processing, real-time video processing and optimization of computer vision algorithms, and parallel processing of target detection results, which improves the efficiency of the system. The entire process is carried out in a continuous video stream, and each frame undergoes YOLOv8 target detection and hand key point recognition, and then the processed results are superimposed and displayed, realizing a feature-rich and responsive real-time visual interaction system.
[0172] Through the trained YOLOv8 model, this method can automatically identify the tactile image area where the fingers of congenitally blind people rest, and make real-time voice broadcast reminders based on the recognition results, thereby improving the blind people's understanding of tactile images and learning efficiency, allowing blind users to further understand and learn physical concepts of different forms and improve their cognitive level to a certain extent.
[0173] Example 3
[0174] like Fig.11 As shown, this embodiment provides a tactile image recognition system based on YOLOv8 and voice broadcast, and applies a tactile image recognition method based on YOLOv8 and voice broadcast described in Embodiment 1 or 2, including:
[0175] Data preprocessing unit 301: used to take a number of photos of different tactile images, construct a tactile image data set and perform preprocessing;
[0176] Model training unit 302: used to build a YOLOv8 model based on a machine learning algorithm;
[0177] and for training the YOLOv8 model using the preprocessed tactile image dataset to obtain a trained YOLOv8 model;
[0178] The tactile image recognition unit 303 is used to detect the tactile image to be recognized by the blind person in real time by using the trained YOLOv8 model, and obtain the recognition result of each specific position in the tactile image to be recognized;
[0179] The voice broadcast unit 304 is used to detect and track the position of the fingertips of the blind person's fingers in real time. When the position of the fingertips of the blind person's fingers coincides with any specific position in the tactile image to be identified, the recognition result of the specific position is output through voice broadcast to complete the recognition of the tactile image.
[0180] In the specific implementation process, firstly, the data preprocessing unit 301 collects the tactile image data set and performs preprocessing;
[0181] Then the model training unit 302 constructs a YOLOv8 model based on the machine learning algorithm; trains the YOLOv8 model using the preprocessed tactile image data set to obtain a trained YOLOv8 model;
[0182] Then, the tactile image recognition unit 303 uses the trained YOLOv8 model to detect the tactile image to be recognized by the blind person in real time, and obtains the recognition result of each specific position in the tactile image to be recognized;
[0183] Finally, the voice broadcast unit 304 detects and tracks the position of the fingertips of the blind person's fingers in real time. When the position of the fingertips of the blind person's fingers coincides with any specific position in the tactile image to be identified, the recognition result of the specific position is outputted by voice broadcast, thereby completing the recognition of the tactile image.
[0184] Through the trained YOLOv8 model, this system can automatically identify the tactile image area where the fingers of congenitally blind people rest, and provide real-time voice broadcast reminders based on the recognition results, thereby improving the blind people's understanding of tactile images and learning efficiency, allowing blind users to further understand and learn different forms of physical concepts and improve their cognitive level to a certain extent.
[0185] The same or similar reference numerals correspond to the same or similar components;
[0186] The terms used in the drawings to describe positional relationships are only used for illustrative purposes and should not be construed as limiting this patent;
[0187] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.
Claims
1. A tactile image recognition method based on YOLOv8 and voice broadcast, characterized in that: The following methods are included: S1: Obtain tactile images fused with visual information, take several photos of different tactile images, construct a tactile image dataset and perform preprocessing; S2: Build the YOLOv8 model based on machine learning algorithm; Using the preprocessed tactile image dataset to train the YOLOv8 model to obtain a trained YOLOv8 model; S3: Using the trained YOLOv8 model to detect the tactile image to be recognized by the blind person in real time, and obtaining the recognition result of each specific position in the tactile image to be recognized; S4: Detect and track the position of the fingertips of the blind person's fingers in real time. When the position of the fingertips of the blind person's fingers coincides with any specific position in the tactile image to be identified, the recognition result of the specific position is outputted by voice broadcast to complete the recognition of the tactile image.
2. A tactile image recognition method based on YOLOv8 and voice broadcast according to claim 1, characterized in that: In the step S1, obtaining a tactile image fused with visual information includes: Visual graphics construction: Analyze the features of physical entities and simplify them to obtain simplified features; decompose the physical entities according to the simplified features; convert all decomposed structures into basic geometric figures that can be understood by the blind; select specific angles that can express all simplified features, reorganize all basic geometric figures at the specific angles to obtain visual graphics; Tactile image construction: distributing touch points on all basic geometric figure contours of the visual figure to obtain the tactile image integrated with the visual information.
3. The tactile image recognition method based on YOLOv8 and voice broadcast according to claim 1, characterized in that: In the step S1, constructing a tactile image dataset includes: Under different conditions, several photos of different tactile images are taken to construct a tactile image dataset; The conditions include: any one of resolution, lighting conditions, shooting angle and shooting distance.
4. The tactile image recognition method based on YOLOv8 and voice broadcast according to claim 1, characterized in that: In step S1, the preprocessing includes: For each photo of the tactile image, use the Make Sense annotation tool to perform online data annotation in the YOLO format to obtain an annotated tactile image dataset; Adjust all photos in the annotated tactile image dataset to a uniform size, remove photos with incorrect annotations and resolutions below a preset threshold, and obtain a tactile image dataset after data removal; Performing data enhancement on the tactile image dataset after data elimination to complete the preprocessing; The data enhancement includes: any one of random flipping, random cropping, color jittering and Mosaic data enhancement.
5. The tactile image recognition method based on YOLOv8 and voice broadcast according to claim 1, characterized in that: In step S2, the YOLOv8 model is trained using a transfer learning strategy: the YOLOv8 model is initialized using weights pre-trained on the COCO dataset, and the parameters of the YOLOv8 model are fine-tuned based on a grid search during the training process.
6. The tactile image recognition method based on YOLOv8 and voice broadcast according to claim 1, characterized in that: In step S2, the total loss function is set Used to supervise the YOLOv8 model for training; The total loss function include: Wherein, λ1 and λ2 are the first and second weight coefficients respectively, λ1+λ2=1; is the classification loss function; is the bounding box regression loss function The classification loss function Specifically, it is the BCE binary cross entropy loss, and the formula is as follows: Among them, y i is the label of sample i, the positive class is 1, the negative class is 0, and N is the number of samples; p i The probability of predicting sample i as a positive class; The bounding box regression loss function Including WIoU loss + DFL loss, the formula is as follows: IN g =in+in gt -IN i H g =h+h gt -H i Among them, λ3 and λ4 are the third and fourth weight coefficients respectively, λ3+λ4=1; (x, y) and (x gt ,y gt ) are the horizontal and vertical coordinates of the center points of the predicted box and the true box respectively; w and h are the width and height of the predicted box respectively; w gt 、h gt are the width and height of the real frame respectively; w i , H i are the lengths of the overlapping parts of the width and height of the predicted box and the real box respectively; are the predicted value and the near-predicted value output by the YOLOv8 model respectively; ′ ,y i ′ and i ′ +1 They are the actual value of the label, the integral value of the label, and the integral value of the adjacent labels; When the total loss function When the minimum value is obtained, the trained YOLOv8 model is obtained.
7. The tactile image recognition method based on YOLOv8 and voice broadcast according to claim 1, characterized in that: In the step S2, after obtaining the trained YOLOv8 model, the following steps are further included: Model conversion: Convert the trained YOLOv8 model to TensorFlow Lite format; Quantization: Dynamically quantize the YOLOv8 model in TensorFlow Lite format; Mobile deployment: The dynamically quantized YOLOv8 model is deployed to a mobile device for operation; the mobile device is specifically a mobile terminal device based on the Android operating system.
8. The tactile image recognition method based on YOLOv8 and voice broadcast according to claim 1, characterized in that: In step S4, the Hands module of MediaPipe is used to detect and track the key points of the blind person's hands in real time, and to identify the positions of the blind person's fingertips in real time; When the position of the fingertip of the blind person coincides with any specific position in the tactile image to be identified, the recognition result of the specific position is outputted by voice broadcast.
9. The tactile image recognition method based on YOLOv8 and voice broadcast according to claim 1, characterized in that: In step S4, the recognition result is parsed into a text description, and the text description is converted into speech using Google Text-to-Speech technology and outputted as a broadcast.
10. A tactile image recognition system based on YOLOv8 and voice broadcast, using the tactile image recognition method based on YOLOv8 and voice broadcast described in claims 1 to 9, characterized in that: include: Data preprocessing unit: used to take a number of photos of different tactile images, construct a tactile image dataset and perform preprocessing; Model training unit: used to build the YOLOv8 model based on the machine learning algorithm; and for training the YOLOv8 model using the preprocessed tactile image dataset to obtain a trained YOLOv8 model; A tactile image recognition unit is used to detect the tactile image to be recognized by the blind person in real time using the trained YOLOv8 model, and obtain the recognition result of each specific position in the tactile image to be recognized; Voice broadcast unit: used to detect and track the position of the blind person's fingertips in real time. When the position of the blind person's fingertips coincides with any specific position in the tactile image to be identified, the recognition result of the specific position is output through voice broadcast to complete the recognition of the tactile image.