Life necessity identification method of blind-assisting equipment

Abstract

The invention relates to the technical field of blind assisting equipment, in particular to a life necessity identification method of the blind assisting equipment. According to the method, an RGB image and a depth image are acquired through an image acquisition module, and are input into an improved multi-modal deep learning recognition model for processing. The model comprises a double-flow feature extraction network, a cross-modal feature fusion module and a space attention module which are respectively responsible for extracting RGB texture features and depth geometric features, realizing multi-modal feature fusion and calculating the importance weight of a space region. The processed features are sent to a target detection head to output the category and the position of the article, and finally the voice module converts the category and the position into a voice prompt containing the name, the position and the use suggestion of the article. According to the invention, the living necessity identification accuracy and environmental adaptability are effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of blind aid equipment, more particularly, the present application relates to a life necessity identification method of blind aid equipment. BACKGROUND

[0002] People with visual impairments face many challenges in their daily lives, among which accurately identifying and locating life necessities is a key requirement. With the development of artificial intelligence technology, computer vision-based assistive devices provide new possibilities for improving the quality of life for visually impaired people. Such technology can help users perceive their surroundings and identify everyday items through image recognition, thereby enhancing their ability to live independently and their safety. Life necessity identification, as one of the core functions of blind aid equipment, directly affects the user experience and the value of the device.

[0003] The existing life necessity identification technology mainly has the following shortcomings: first, most systems rely only on single RGB images for identification, lack of depth information makes it difficult to accurately determine the spatial position and three-dimensional structure of the object, especially in complex scenes, misjudgment is easy to occur; second, traditional identification models perform poorly in the case of multiple object overlap, light change or partial occlusion, and lack of robustness; third, existing methods often ignore the complementarity between different modal data and fail to effectively fuse color texture and geometric features; fourth, current systems generally lack understanding of the spatial relationship of objects and cannot provide accurate relative position description; finally, most identification models lack continuous learning ability and are difficult to adapt to different users' environment differences and usage habits, resulting in a decline in recognition accuracy in actual deployment. These limitations seriously restrict the applicability and reliability of blind aid equipment in real-world scenarios.

[0004] Therefore, in view of the above problems, a life necessity identification method for blind aid equipment is proposed, the main problems to be solved are: how to improve the recognition accuracy and robustness of blind aid equipment in complex real-world environments, how to effectively utilize multi-modal sensor data to improve the spatial positioning accuracy of objects, and how to make the identification system adapt to user's individual needs and environmental changes. SUMMARY

[0005] In order to overcome the above-mentioned defects of the prior art, the embodiments of the present application provide a life necessity identification method for blind aid equipment to solve the problems raised in the background art.

[0006] To achieve the above-mentioned purpose, the present application provides the following technical scheme: a life necessity identification method for blind aid equipment, the method comprising the following steps: S1, acquiring an RGB image and a depth image of a scene to be recognized through an image acquisition module provided on a blind-aiding device, wherein the image acquisition module comprises at least one RGB camera and one depth sensor, the resolution of the RGB image is 1920x1080 pixels, and the acquisition range of the depth image is between 0.5 m and 5 m; S2, inputting the RGB image and the depth image into an improved multi-modal deep learning recognition model, wherein the improved multi-modal deep learning recognition model comprises a double-flow feature extraction network, a cross-modal feature fusion module, and a spatial attention module, the double-flow feature extraction network extracts texture features of the RGB image and spatial geometric features of the depth image through two independent convolutional neural network branches respectively, the cross-modal feature fusion module fuses the features of the two modalities through feature splicing and 1x1 convolution operation, and the spatial attention module calculates the importance weight of different spatial regions based on the fused features and generates a weighted feature map; S3, inputting the weighted feature map into a target detection head to output the category label and position information of daily necessities, wherein the target detection head adopts an anchor box mechanism and predicts large, medium and small daily necessities through three detection layers of different scales; S4, converting the recognition result into a voice prompt through a voice output module of the blind-aiding device, wherein the voice prompt includes the name of the object, the relative position, and the use suggestion.

[0007] Preferably, in step S2, the two independent branches of the double-flow feature extraction network comprise: the first branch uses a pre-trained ResNet-50 network on an ImageNet dataset to extract multi-layer features from the RGB image, and the second branch uses a specially designed point cloud convolution network to extract three-dimensional geometric features from the depth image, and the output feature maps of the two branches are adjusted to the same size of 256x256 through bilinear interpolation.

[0008] Preferably, the point cloud convolution network used in the second branch comprises 4 convolution layers, each followed by a batch normalization layer and a ReLU activation function, and the convolution kernel size is 3x3, and the extracted three-dimensional geometric features are compressed into a fixed-length feature vector through a max-pooling layer.

[0009] Preferably, in step S2, the cross-modal feature fusion module realizes feature fusion in the following way: first, the RGB features and the depth features are spliced in the channel dimension, then the number of feature channels after splicing is reduced from 2048 dimensions to 512 dimensions through 1x1 convolution, and finally feature enhancement is performed through group normalization layer and Swish activation function.

[0010] Preferably, the spatial attention module generates the importance weight by first inputting the fusion feature into two consecutive convolution layers to generate an attention score map, then normalizing the score map to between 0 and 1 using a softmax function, and finally multiplying the normalized score map with the original fusion feature element by element to highlight the region features that are more important for essential goods recognition.

[0011] Preferably, the spatial attention module further comprises a temporal consistency unit that adjusts the attention weight of the current frame according to the attention distribution of the previous 3 frames through a gating mechanism when processing consecutive video frames, to maintain the stability of the recognition result.

[0012] Preferably, in step S3, the target detection head uses the YOLOv5 architecture, which contains 3 detection layers corresponding to feature map sizes of 80x80, 40x40 and 20x20 respectively, and each detection layer predicts 3 anchor boxes, for a total of 9 anchor boxes of different scales for detecting essential goods of different sizes.

[0013] Preferably, the improved multi-modal deep learning recognition model uses a focal loss function to solve the class imbalance problem during training, and a CIoU loss function to optimize the bounding box regression, with a weight ratio of 1:0.5 for the two loss functions, and a training period of 100 epochs.

[0014] Preferably, in step S4, the speech output module generates a structured speech prompt based on the recognition result, where the relative position information is described by a polar coordinate system including distance and azimuth angle, and the suggestion is generated by retrieving from a predefined knowledge base based on the category of the recognized item.

[0015] Preferably, the method further comprises an online learning step of collecting correctly recognized samples through a user feedback interface on the blind aid device, and periodically updating the improved multi-modal deep learning recognition model using these samples through an incremental learning algorithm, with a model update period of every 14 days and a sample quantity of no less than 1000 for each update.

[0016] Technical effects and advantages of the present application: Compared with the prior art, the present application constructs a deep learning architecture containing a double-flow feature extraction network and a cross-modal feature fusion module, extracts texture features and spatial geometric features from RGB images and depth images respectively, and effectively fuses them at the feature level. This design fully utilizes the complementary advantages of different modal data, with RGB features providing rich color and texture information and depth features contributing accurate spatial structure information, and through 1x1 convolution and group normalization operations, the coordination and enhancement between features are realized. This way improves the recognition ability of the system under complex lighting conditions, object occlusion and other conditions, substantially improves the detection accuracy of daily necessities, and enhances the model's understanding of the three-dimensional characteristics of the items.

[0017] Compared with the prior art, the present application introduces a spatial attention mechanism with temporal consistency, which considers the attention distribution of the current frame and the historical frame when processing continuous video streams, and dynamically adjusts the importance weight of different spatial regions using a gating mechanism. This mechanism generates an attention score map through a convolution layer, combines softmax normalization and a temporal consistency unit, so that the system can continuously focus on key areas in the scene and reduce recognition fluctuations caused by temporary occlusion or changes in viewing angle. This design effectively improves the stability of the recognition results, ensuring that users can still obtain continuous and reliable daily necessities recognition services while moving, greatly improving the user experience.

[0018] Compared with the prior art, the present application adopts a technical solution combining a multi-scale target detection architecture and an online learning mechanism, uses three detection layers of different scales to deal with large, medium and small size daily necessities detection requirements, and establishes a user feedback interface and an incremental learning process. This scheme covers the detection of various size items through an anchor box mechanism, uses a focal loss function to alleviate the class imbalance problem, and regularly updates the model parameters using user-confirmed correct samples. This method not only improves the detection sensitivity of the system for daily necessities of different sizes, but also enables the recognition model to gradually adapt to the user's individualized environment and item usage habits, achieving continuous performance optimization. BRIEF DESCRIPTION OF DRAWINGS

[0019] Fig. 1 The overall flowchart of the blind aid daily necessities recognition of the present application.

[0020] Fig. 2 The multi-modal feature extraction and fusion schematic diagram of the present application.

[0021] Fig. 3 The spatio-temporal attention mechanism working principle diagram of the present application.

[0022] Fig. 4 The multi-scale detection and online learning optimization mechanism diagram of the present application. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0024] Embodiment one As shown in the accompanying drawings Figs. 1 to 4 A life necessity identification method of a blind aid device is provided, which realizes efficient and accurate environmental perception and object identification through the cooperation of four steps. The image acquisition step is responsible for obtaining multi-modal sensor data of the environment, which is the basis of the entire system; the multi-modal recognition step uses a designed deep learning model to process heterogeneous data; the target detection step accurately locates and classifies life necessities based on a multi-scale mechanism; and the voice output step converts the recognition results into perceptible prompt information for the user. The cooperative working mechanism of these steps ensures the recognition accuracy and robustness of the system in complex environments, especially in challenging scenarios such as occlusion and light changes.

[0025] Furthermore, the image acquisition step is realized through a multi-sensor module integrated on the blind aid device, which includes a high-resolution RGB camera and a precise depth sensor. The RGB camera uses a Sony IMX477 sensor, which has a large 1 / 2.3-inch photosensitive element to ensure clear image data in low-light environments. The resolution is set to 1920x1080 pixels, and the frame rate is fixed at 30fps. This configuration ensures image quality while controlling data processing load.

[0026] The depth sensor is based on time-of-flight technology, which calculates distance by measuring the round-trip time of laser pulses. The measurement range covers 0.5 meters to 5 meters with an accuracy of ±1 centimeter. The field of view is configured as 65°x65°, which ensures that a large area in front of the user can be covered.

[0027] The two sensors ensure the time consistency of data acquisition through hardware synchronization signals. A special clock synchronization circuit is used to control the time deviation within 1 millisecond. The collected data is transmitted in real time to the embedded processing unit through the USB3.0 interface. Lossless compression algorithms are used during transmission to reduce bandwidth occupancy. This design ensures the synchronization and integrity of the input data, providing a reliable foundation for subsequent processing, especially maintaining the spatio-temporal consistency of the data during user movement.

[0028] Furthermore, the dual-stream feature extraction network used in the multi-modal recognition step is specially optimized. The network includes two independent but cooperative branches: The first branch extracts rich texture features from RGB images based on ResNet-50 architecture. The second branch extracts precise spatial geometry features from depth images using a specially designed point cloud convolutional network.

[0029] The ResNet-50 branch is implemented under the PyTorch framework, with an input size of 512x512 pixels. This size strikes a good balance between computational efficiency and feature detail. The network retains all convolutional layers from conv1 to conv5_x, which can capture multi-level information from low-level edge features to high-level semantic features. The point cloud convolutional network branch contains four designed convolutional layers, each with a kernel size of 3x3. This size of kernel can capture local features without introducing excessive computational load.

[0030] The design of channel numbers of 64, 128, 256, and 512 reflects the concept of progressive enhancement of feature expression. The batch normalization layer after each convolutional layer ensures training stability, and the ReLU activation function provides the necessary nonlinear transformation capability.

[0031] The outputs of the two branches are unified to a resolution of 256x256 through bilinear interpolation. This uniform size processing creates conditions for subsequent feature fusion. During feature extraction, the network learns the importance weights of different modal features adaptively. This design allows automatic adjustment of the dependence on depth features in specific environments (such as low-light conditions).

[0032] Furthermore, in the specific implementation of the point cloud convolutional network, the parameters of the four convolutional layers are optimized to form a hierarchical feature learning architecture. The first convolutional layer uses 64 3x3 convolutional kernels with a stride of 2. This relatively large stride can reduce computational complexity while maintaining the receptive field. The second layer uses 128 3x3 convolutional kernels with a stride of 2, further abstracting feature expression. The third layer uses 256 3x3 convolutional kernels with a stride of 1, starting to capture more detailed geometric structures. The fourth layer uses 512 3x3 convolutional kernels with a stride of 1, forming high-level geometric feature representation.

[0033] All convolutional layers use the He initialization method to set the initial weights. This method is particularly suitable for use with the ReLU activation function and can effectively avoid the problem of gradient disappearance. The batch size is fixed at 16, which strikes a good balance between training stability and memory usage. During the forward propagation of the network, each layer performs feature transformation on the input point cloud data, gradually converting the original depth information into discriminative high-dimensional features.

[0034] The hierarchical structure is designed to enable the network to learn from simple local geometric patterns to complex global geometric structures by gradually increasing the feature dimension and the level of abstraction, so as to achieve accurate understanding of the three-dimensional shape of daily necessities.

[0035] Furthermore, the cross-modal feature fusion module realizes the effective integration of multi-modal features through the designed splicing and dimension reduction operation. The module first splices the 1024-dimensional features of the RGB branch with the 512-dimensional features of the depth branch in the channel dimension to form a 1536-dimensional fusion feature vector. This splicing operation retains all the information of the two modalities, but also brings the problem of dimension disaster. Therefore, the module uses a 1x1 convolution kernel to reduce the feature dimension to 512, which not only reduces the computational complexity, but more importantly, promotes the interaction and integration between the two modalities.

[0036] The convolution weights are initialized by the Xavier method, which can maintain the stability of the variance of the features during the forward propagation process. After dimension reduction, a group normalization layer is applied to normalize the features into 32 groups. Compared with the traditional batch normalization, the group normalization is more friendly to small batch data and is more suitable for embedded device application scenarios.

[0037] Finally, the Swish activation function is used to introduce a nonlinear transformation, and the Swish function is defined as: where is the sigmoid function. Compared with the ReLU function, the Swish function has a smoother gradient characteristic, which helps to improve the convergence of the training process. The entire fusion process can be regarded as a feature refinement process, which produces a more discriminative fusion feature representation through a series of operations such as dimension reduction, normalization and nonlinear transformation.

[0038] Furthermore, the spatial attention module generates a weight distribution reflecting the importance of different spatial regions based on convolution and normalization operations.

[0039] The module inputs the 512-dimensional fusion features into two consecutive 3x3 convolution layers, the first layer outputs 256-dimensional features, and the role of this layer is to preliminarily integrate the information between channels; the second layer outputs a 1-dimensional attention score map, which is responsible for generating the attention distribution in the spatial dimension.

[0040] The score map is normalized in the spatial dimension by the softmax function to generate the attention weight: where is the score of position , and is the height and width of the feature map. This normalization operation ensures that the sum of all weights is 1, allowing the attention mechanism to explicitly represent the relative importance of different regions.

[0041] The normalized weights are element-wise multiplied with the original fused features. This multiplication operation is equivalent to enhancing the features in important regions while suppressing those in unimportant regions. In specific implementations, the module learns to assign high weights to regions containing essential items and low weights to background regions. For example, when processing a kitchen scene, the module might assign a higher attention weight to a water cup on the countertop and a lower weight to an empty floor area. This adaptive attention mechanism improves the model's ability to identify target objects in complex backgrounds.

[0042] Furthermore, the spatial attention module integrates a temporal consistency unit based on the gated recurrent mechanism, which is specifically designed to handle temporal information in consecutive video frames. The unit adopts a GRU structure with a hidden layer dimension of 256. This dimension size is sufficient to capture temporal information without introducing excessive computational burden.

[0043] The update gate of GRU is responsible for controlling the retention level of historical information, calculated as where is the hidden state of the previous time step, is the current input, is the weight matrix of the update gate. The reset gate determines which historical information needs to be forgotten, calculated as .

[0044] This gating mechanism allows the network to adaptively select important temporal information. For example, when the user moves their head quickly, the unit automatically reduces the dependence on historical information, avoiding inaccurate attention distribution caused by motion blur.

[0045] During training, a temporal consistency loss function is used to constrain the degree of change in attention distribution between adjacent frames, ensuring a stable attention pattern in static scenes. The unit uses gradient clipping with a threshold of 1.0 during training to limit the gradient range, effectively preventing the problem of gradient explosion during training.

[0046] Further, the multi-scale detection head used in the target detection step is improved based on the YOLOv5 architecture and optimized for the characteristics of essential item recognition. The detection head includes three detection layers of different scales, corresponding to feature map sizes of 80x80, 40x40, and 20x20, respectively. The 80x80 high-resolution feature map is suitable for detecting small-sized items such as keys and medicine bottles; the 40x40 medium-resolution feature map is suitable for detecting medium-sized items such as mobile phones and water bottles; and the 20x20 low-resolution feature map is suitable for detecting large-sized items such as chairs and backpacks.

[0047] Each detection layer is pre-set with 3 anchor boxes, and the best size is determined on the training data set through the k-means clustering algorithm. This data-driven method ensures a good match between the anchor box shape and the real item distribution. The detection head output includes three parts: class probability, bounding box coordinates, and confidence score. The bounding box coordinates are optimized using the CIoU loss function, which considers the overlap area, center point distance, and consistency of the aspect ratio.

[0048] In the inference stage, the non-maximum suppression algorithm is used to remove redundant detection boxes, and the IoU threshold is set to 0.5, which achieves a good balance between detection accuracy and recall rate. The entire detection process uses a multi-task learning framework to optimize both classification and regression targets, which improves the overall performance of the detection.

[0049] Further, the combined loss function used in the model training process is designed to address both the class imbalance and the bounding box regression accuracy. The focal loss function is specifically used to handle the class imbalance problem common in essential item recognition, and its calculation formula is: where is the probability predicted by the model, is set to 0.25, is set to 2. This design makes the network pay more attention to difficult-to-classify samples while reducing the loss contribution of easy-to-classify samples.

[0050] The CIoU loss function is used to optimize the bounding box regression, and its calculation formula is: where $IoU$ is the intersection over union, is the Euclidean distance between the predicted box center point and the real box center point is the diagonal length of the minimum enclosing rectangle, is the aspect ratio consistency parameter, is the weight coefficient.

[0051] ​This loss function, which takes into account multiple geometric factors, results in more accurate bounding box predictions. The weight ratio of the two loss functions is 1:0.5, which is determined by grid search to achieve the best balance between classification accuracy and localization precision. The training uses the Adam optimizer with an initial learning rate of 0.001 and a cosine annealing strategy to adjust the learning rate, which speeds up the convergence while avoiding local optima.

[0052] Furthermore, the voice output step is realized through advanced text-to-speech conversion technology, which involves multiple processing stages. The recognition result is first converted into a structured text description, and this conversion process is based on predefined template rules to ensure that the generated language is natural and complete.

[0053] The relative position information is calculated through a polar coordinate system, and the distance Based on direct acquisition of depth sensor data, the azimuth It is calculated through the mapping relationship between the image coordinate system and the real-world coordinate system. Specifically, first, the pixel coordinates in the image are converted into three-dimensional coordinates in the camera coordinate system, and then they are converted into the world coordinate system according to the current direction information of the device. The suggestions are retrieved from a knowledge base containing 500 common daily necessities, and the knowledge base is organized in a graph database structure to quickly query the usage precautions of related items.

[0054] The voice generation uses the Google TTS engine with a sampling rate of 16 kHz, which strikes a good balance between voice quality and data size. The bit rate is set to 128 kbps to ensure clear and natural speech. The delay of the entire voice generation process is controlled within 100 milliseconds, which ensures natural and smooth user interaction. The system also supports multiple voice style selection, and users can adjust parameters such as speech rate and tone according to personal preferences.

[0055] Furthermore, the online learning step realizes the continuous optimization of the model through incremental learning algorithms, which enables the system to adapt to the user's personal usage habits and environmental characteristics. Users provide feedback through physical buttons on the device, and correctly recognized samples along with their labeled information are stored in the local SQLite database.

[0056] The database adopts a lightweight design, supports fast read and write operations, and provides data integrity protection. The system automatically starts the model update process every 14 days, and this update cycle is determined through empirical research, which neither affects user experience too frequently nor integrates new learning samples in a timely manner. During the update process, 1000 samples are randomly selected from the database to form a training batch, and the elastic weight consolidation algorithm is used to update the network parameters. This algorithm calculates the importance weight of the parameters, and adjusts the loss function as follows: wherein is the loss of the new task, is the current parameter, is the important parameter of the old task, is the hyperparameter set to 0.5. This design ensures that while learning new knowledge, old knowledge will not be forgotten, effectively alleviating the problem of catastrophic forgetting. The learning rate is set to 0.0001 during the update process, which ensures the stability of parameter updating. The training period is 10 epochs, balancing the update effect and computational cost.

[0057] Embodiment Two A specific kitchen environment application scenario will be described in detail below. In this scenario, the user wears smart glasses integrated with the system and enters the kitchen, aiming to accurately identify life necessities such as water cups, bowls, and chopsticks on the operation table. The image acquisition module synchronously captures RGB images and depth images of the scene at a rate of 30 fps. The RGB image provides rich color and texture information, while the depth image provides accurate distance information. The two types of data are accurately aligned through timestamps to ensure consistency in subsequent processing.

[0058] In the data processing stage, the dual-stream feature extraction network processes data of both modalities simultaneously. The RGB branch first normalizes the input image, scaling the pixel values to the range [0, 1], and then extracts features through each convolutional layer of the ResNet-50 network. In the conv1 layer, the network learns basic edge and texture features; in the conv3_x layer, it begins to form more complex pattern features; and by the conv5_x layer, it can already recognize high-level semantic features.

[0059] Meanwhile, the depth branch converts the depth image into point cloud data and extracts geometric features through four convolutional layers of the point cloud convolutional network. The first convolutional layer learns basic geometric features such as local surface curvature, and subsequent layers gradually combine these basic features to form complete object shape representations.

[0060] In the feature fusion stage, the cross-modal feature fusion module concatenates and reduces the dimension of the feature vectors from both branches. Assuming that the RGB branch outputs a feature vector with dimensions [256, 256, 1024] and the depth branch outputs a feature vector with dimensions [256, 256, 512], the concatenated fusion feature has dimensions [256, 256, 1536]. After reducing the dimension to [256, 256, 512] through a 1x1 convolution, the group normalization layer normalizes each group of 16 channels, and the Swish activation function further introduces non-linear transformation.

[0061] At this time, the spatial attention module starts working, first calculating the attention score for each spatial location. Assuming the score at position (i,j) in the cup region is =2.5, and the score at position (m,n) in the background region is =-1.2, after softmax normalization, the attention weight of the cup region is =0.08, and the attention weight of the background region is =0.002. This weight distribution ensures that the network pays more attention to the cup region.

[0062] In the target detection stage, the multi-scale detection head makes predictions on different levels of feature maps. On the 80x80 fine-grained feature map, the detection head focuses on detecting small-sized items such as chopsticks; on the 40x40 mid-level feature map, it detects medium-sized cups; and on the 20x20 coarse-grained feature map, it detects large-sized bowls. Taking cup detection as an example, assume the predicted bounding box coordinates are The CIoU loss is calculated by comparing with the true label, driving the network to optimize parameters. During model training, the focal loss function adjusts the weights of difficult and easy samples, giving lower loss weights to easy-to-classify background regions and higher loss weights to similar items that are difficult to distinguish.

[0063] In the speech output stage, the system first generates structured text based on the detection results: "Detected cup, distance 1.2 meters, direction straight ahead, suggestion: please confirm the water temperature before drinking". Then it converts it into a speech signal through the TTS engine, with a sampling process at a rate of 16kHz, generating corresponding sound waveforms. The entire processing flow is completed on an embedded device, with an end-to-end delay from image acquisition to speech output strictly controlled within 100 milliseconds, ensuring real-time interaction experience for users.

[0064] In the online learning stage, when the user confirms the correct recognition through the button, the system stores the current frame's RGB image, depth image, and label information into the local database. When the model update period arrives, the system randomly selects 1000 samples from the database and updates the network parameters using the elastic weight consolidation algorithm.

[0065] During the update process, the update amplitude of important parameters is constrained by the factor, ensuring that the network learns new samples while retaining old knowledge. For example, for important parameters related to cup detection, has a larger value, and the parameter update amplitude is smaller; for parameters related to newly learned items, has a smaller value, and the parameter update amplitude is larger.

[0066] The complete scene shows the combination of various technical elements of the application, and through the precise cooperation of multi-modal data acquisition, deep learning feature extraction, attention mechanism optimization, multi-scale target detection, intelligent voice feedback and continuous online learning, the technical effect of accurately, stably and adaptively identifying daily necessities in a real environment is realized. The system can not only adapt to different environmental conditions, but also continuously optimize performance through continuous learning, providing reliable life assistance for visually impaired users.

[0067] Finally, it should be pointed out that: first, in the description of the present application, it should be pointed out that unless otherwise specified and limited, the terms "mounting", "connection", "connection" should be understood broadly, which can be mechanical connection or electrical connection, or the communication between two elements, or direct connection, "up", "down", "left", "right" and the like are only used to represent the relative positional relationship, when the absolute position of the described object changes, the relative positional relationship may change; Secondly: the drawings of the disclosed embodiments of the application only involve the structures involved in the disclosed embodiments, other structures can refer to the usual design, and in the case of no conflict, the same embodiment and different embodiments of the application can be combined with each other; Finally: the above only describes the preferred embodiments of the application and is not used to limit the application, any modification, equivalent replacement, improvement, etc. within the spirit and principle of the application should be included in the protection scope of the application.

Claims

1. A life necessity identifying method of a blind aid device, characterized by, The method comprises the following steps: S1, acquiring an RGB image and a depth image of a scene to be identified through an image acquisition module provided on an assistive blind device, wherein the image acquisition module comprises at least one RGB camera and one depth sensor, the resolution of the RGB image is 1920*1080 pixels, and the acquisition range of the depth image is between 0.5 m and 5 m; S2, inputting the RGB image and the depth image into an improved multi-modal deep learning identification model, wherein the improved multi-modal deep learning identification model comprises a double-flow feature extraction network, a cross-modal feature fusion module and a spatial attention module, the double-flow feature extraction network extracts texture features of the RGB image and spatial geometric features of the depth image through two independent convolutional neural network branches respectively, the cross-modal feature fusion module fuses the features of the two modalities through feature splicing and 1*1 convolution operation, and the spatial attention module calculates the importance weight of different spatial regions based on the fused features and generates a weighted feature map; S3, inputting the weighted feature map into a target detection head to output the category label and position information of daily necessities, wherein the target detection head adopts an anchor box mechanism and predicts large, medium and small daily necessities through three detection layers of different scales; S4, converting the identification result into a voice prompt through a voice output module of the assistive blind device, wherein the voice prompt comprises an article name, a relative position and a use suggestion.

2. The life necessity recognition method of a blind-assisting device according to claim 1, characterized by, In step S2, the two independent branches of the double-flow feature extraction network comprise: a first branch uses a ResNet-50 network pre-trained on an ImageNet dataset to extract multi-layer features from the RGB image, and a second branch uses a specially designed point cloud convolution network to extract three-dimensional geometric features from the depth image, and the output feature maps of the two branches are adjusted to the same size of 256*256 through bilinear interpolation.

3. The method of claim 2, wherein The point cloud convolution network used in the second branch comprises 4 convolution layers, each followed by a batch normalization layer and a ReLU activation function, and the convolution kernel size is 3*3, and the extracted three-dimensional geometric features are compressed into a fixed-length feature vector through a max-pooling layer.

4. The method of claim 1, wherein In step S2, the cross-modal feature fusion module realizes feature fusion in the following way: first, the RGB features and the depth features are spliced in the channel dimension, then a 1*1 convolution is used to reduce the number of feature channels after splicing from 2048 dimensions to 512 dimensions, and finally feature enhancement is performed through a group normalization layer and a Swish activation function.

5. The method of claim 1, wherein, The spatial attention module generates the importance weight in the following way: first, the fused features are input into two consecutive convolution layers to generate an attention score map, then the score map is normalized to 0 to 1 through a softmax function, and finally the normalized score map is multiplied element by element with the original fused features to highlight the region features that are more important for daily necessity identification.

6. The life necessity recognition method of a blind-assisting device according to claim 5, wherein The spatial attention module also includes a temporal consistency unit that adjusts the attention weights of the current frame based on the attention distribution of the previous three frames through a gating mechanism to maintain the stability of the recognition results when processing consecutive video frames.

7. The method of claim 1, wherein In step S3, the target detection head uses the YOLOv5 architecture, which includes three detection layers corresponding to feature map sizes of 80x80, 40x40, and 20x20, respectively. Each detection layer predicts three anchor boxes, and a total of nine anchor boxes of different scales are used to detect daily necessities of different sizes.

8. The method of claim 1, wherein, The improved multi-modal deep learning recognition model uses a focal loss function to solve the class imbalance problem during training, and a CIoU loss function to optimize the bounding box regression. The weight ratio of the two loss functions is 1:0.5, and the training period is set to 100 epochs.

9. The method of claim 1, wherein In step S4, the speech output module generates structured speech prompts based on the recognition results, where the relative position information is described using a polar coordinate system, including distance and azimuth angle, and the suggestions are generated by retrieving from a predefined knowledge base based on the category of the recognized items.

10. The method of claim 1, wherein The method also includes an online learning step, which collects correctly recognized samples through a user feedback interface on the assistive device, and periodically updates the improved multi-modal deep learning recognition model using these samples through an incremental learning algorithm. The model update period is set to every 14 days, and the number of samples updated each time is not less than 1000.

Images