Blind person glasses target identification method based on consciousness judgment

Abstract

The invention discloses a blind person glasses target recognition method based on consciousness judgment, and the method comprises the following steps: 1, collecting an image sequence of F frames per second through a binocular or monocular depth camera, and carrying out the primary processing of the image sequence for subsequent target segmentation and feature extraction; 2, performing target detection on each frame of image based on a universal target detection model, and extracting a key target set; step 3, performing cross-frame target tracking based on the key target set extracted in the step 2; 4, identifying interest targets, and sorting the interest targets; and 5, inputting the image information of the target and the corresponding complete track feature Queue (ID) into the visual language model. According to the blind person glasses target recognition method based on consciousness judgment, the interested target concerned by the user can be automatically recognized through real-time image analysis and target tracking, and personalized feedback is completed based on voice / text instructions.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and intelligent identification of assistive devices, and more particularly to a blind glasses target identification method based on consciousness judgment. BACKGROUND

[0002] Existing blind assistive systems are mainly focused on path obstacle avoidance and voice navigation. In addition, traditional blind interaction methods are mainly based on question-and-answer or keyword triggering. This approach cannot perceive the user's current real attention focus or interest target, and the system lacks modeling and prediction ability for the user's consciousness state, resulting in that the prompt content is usually out of context, the interaction lacks initiative and adaptability, and the intelligent guidance cannot be truly realized.

[0003] More importantly, the current system generates a large amount of voice content for each frame or each detected target when facing continuous video frames, at a fixed rhythm. A major problem brought by this mechanism is "information redundancy": the user cannot receive subsequent content before listening to the previous description, causing information accumulation and delay, which seriously affects the interaction efficiency and user experience.

[0004] Therefore, a technical solution with consciousness judgment ability is needed, which can dynamically identify the user's attention target based on user behavior characteristics (such as gaze, proximity, etc.), combine semantic compression and intelligent sorting mechanism, and only broadcast the most valuable and relevant content, thereby improving the response intelligence and practicality of the assistive system. SUMMARY

[0005] In view of the deficiencies in the prior art, the purpose of the present application is to provide a consciousness judgment interaction method combining visual perception and behavior modeling, which can automatically identify the user's interest target through real-time image analysis and target tracking, and complete personalized feedback based on voice / text instructions.

[0006] To achieve the above purpose, the present application provides the following technical solution: a blind glasses target identification method based on consciousness judgment, characterized by comprising the following steps: Step one: collect F frames of image sequences per second through binocular or monocular depth camera, and perform preliminary processing for subsequent target segmentation and feature extraction; Step two: perform target detection on each frame of image based on a general target detection model, and extract a key target set; Step three: perform cross-frame target tracking based on the key target set extracted in step two; Step four: identify the interest target and sort the interest target; Step five, after the sorting is completed, the image information of the target and its corresponding complete trajectory feature Queue (ID) are input into the visual language model, the target is subjected to multi-modal semantic analysis, and the user is fed back through voice or vibration prompt; If the model identifies that there is a dangerous factor in the target, an active warning mechanism is triggered, voice broadcast and vibration feedback are used to warn the user, and the extracted features, completed vector calculation, and the recognition results and interaction records fed back by the visual language model are stored in the memory unit for subsequent environment identification, semantic recall and context association.

[0007] As a further improvement of the application, the specific way of extracting the key target set in step two is as follows: Suppose the number of targets is k, denoted as ; Each target extracts the following attributes to form a state vector: ① Image semantic features , obtained by image encoding; ② Center point coordinates : record the center point coordinates of the target; ③ Boundary box size : represents the pixel width and height of the target; ④ Target depth value : for the pixel position of the target, extract its corresponding depth value set, and calculate the average or median of the set as the representative depth value of the target; ⑤ Timestamp : record the frame acquisition time.

[0008] As a further improvement of the application, the specific way of performing cross-frame target tracking in step three is as follows: for consecutive frames and , , compare the comprehensive similarity of each target in the first frame with all targets in the previous frame , which includes the following steps: Step three, identify the number of targets in the image, and obtain the total number of targets identified in the first frame picture ; Step three, set a minimum similarity threshold, below which it is considered not similar, otherwise it is considered similar; Step three, identify the number of targets in the image, and obtain the total number of targets identified in the first frame picture ; Steps three and four: Calculate the overall similarity of all targets using the following formula. To match similar targets: In the formula, : Target bounding box overlap Visual feature cosine similarity Position proximity is modeled using a Gaussian function constructed from the distance between center points; position sensitivity parameters... This indicates the maximum tolerable offset range, with a default setting of σ=20 pixels; Spatial overlap weight Appearance consistency weight Positional continuity weight; Step 35: If no similar target is found, then... A unique ID is created for the current target of the frame image, a trajectory queue is created for the current target, and the target is placed at the head of the queue; Step 36: If a similar target is matched, obtain the ID of the most similar target in frame t. The current target in frame t+1 inherits this ID and is placed at the head of the queue.

[0009] As a further improvement of the present invention, each ID in steps 35 and 36 maintains the following attribute time series. Depth sequence: Time series: The maximum length of the trajectory is set to n; if it exceeds this length, it will automatically slide and update.

[0010] As a further improvement to the present invention, the specific steps for sorting interest targets in step four are as follows: Step 41, determine the current detection number Frame image, corresponding to each target The length of the corresponding target The longer the length, the more attention it receives, thus completing the fixation judgment; Step 42, for the currently detected... Frame images, trajectory queues for each target To determine whether the target was actively approached by the user, the derivative is used, and the average rate of change of depth between adjacent frames is calculated. in, It is a negative number, therefore The smaller the value, the more the user wants to get closer to the target, and the more attention the target receives, thus completing the proximity judgment. Step four three, based on the gaze judgment result and the proximity judgment result in step four one and step four two, the target priority is sorted.

[0011] As a further improvement of the application, the rule for sorting in step four three is as follows: Occurrence frequency priority: in the sliding time window The longer the target trajectory in the middle, the higher the user's attention, and the higher the sorting; Proximity priority: if the occurrence frequency is the same, calculate the average change rate of target depth , The smaller the value, the closer the target to the user, and the higher the sorting.

[0012] Advantages of the application: (1) Introducing a consciousness judgment model based on gaze and proximity behavior to replace traditional question and answer human-computer interaction; (2) Realize target level visual state tracking and time evolution modeling, significantly improve the recognition robustness; (3) Combined with Prompt semantic linkage mechanism, enhance the individualization and context adaptability of interaction; (4) Support target memory and multi-modal information fusion feedback, improve the intelligent level and user experience of the auxiliary system. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 The flowchart of the consciousness judgment-based blind glasses target recognition method of the application. DETAILED DESCRIPTION

[0014] The application will be further described in conjunction with the embodiments given in the accompanying drawings.

[0015] Referring to Figure 1 The application of the present embodiment proposes a consciousness judgment interaction method combining visual perception and behavior modeling, which can automatically identify the interest target of user attention through real-time image analysis and target tracking, and complete personalized feedback based on voice / text instructions. The method mainly includes five steps of image acquisition, key target recognition and feature frame extraction, cross-frame target tracking, interest target sorting and recognition feedback.

[0016] The above five steps are described in detail in the following embodiment: 1. Image acquisition Collect F frames (settable, default 5 frames) of image sequence per second through binocular or monocular depth camera, and perform preliminary processing (such as noise reduction, alignment) for subsequent target segmentation and feature extraction.

[0017] 2. Key target recognition and feature frame extraction Based on a general object detection model (such as YOLO), each frame of image is detected for key target set (such as pedestrians, vehicles, obstacles, traffic signs, etc., which can be configured by the background to detect the target), assuming the number of targets is k, denoted as .

[0018] Each target Extract the following attributes to form the state vector: ① Image semantic features , obtained by image encoding; ② Center point coordinates : Record the center point coordinates of the target ; ③ Boundary box size : Represents the pixel width and height of the target; ④ Target depth value : For the pixel position of the target, extract its corresponding depth value set, and calculate the average or median of the set as the representative depth value of the target; ⑤ Timestamp : Record the frame acquisition time.

[0019] These information will be the basis for subsequent target matching and state reasoning.

[0020] 3. Cross-frame target tracking For consecutive frames and , , compare the similarity of each target in the frame with all targets in the previous frame . Pseudocode as follows: for(j=1 to m){ / / Total number of targets identified in the frame picture = ; / / is the minimum similarity threshold set, below which it is considered not similar, default is 0.7.

[0021] for(i=1 to k){ / / Total number of targets identified in the frame picture if( ){ , } } if( == If no similar target was matched ID= / / Give the first Create a unique ID for the current target of the frame image. Queue(ID) = new Queue(); / / Create a trajectory queue for the current target and place it at the head of the queue.

[0022] }else{ ID= / / Get the first The ID of the most similar target in the frame.

[0023] ID; / / No. The current target of the frame image inherits this ID. / / / Place the current target at the head of the queue } } ① : Target bounding box overlap; ② Visual feature cosine similarity; ③ Position proximity is modeled using a Gaussian function constructed from the distance between center points; position sensitivity parameters... This indicates the maximum tolerable offset range, with a default setting of σ=20 pixels.

[0024] ④ Weighting Term: Spatial Overlap Weight Appearance consistency weight (feature similarity) Location continuity weight (distance from center point) The weights represent the "importance" of these three factors in the overall similarity score. The weights can be reasonably combined through empirical settings (such as 0.5, 0.3, 0.2), or the optimal parameter combination can be obtained by training with labeled samples to enhance the accuracy and adaptability of target matching.

[0025] In addition, it needs to be cleared. The trajectory queue of the target whose ID does not appear in the frame.

[0026] ⑤ Each ID maintains the following attribute time series Depth sequence: .

[0027] Time series: .

[0028] The maximum length of the trajectory is set to n (which can be set in the background, and the default setting is 10 frames), and the automatic sliding update is performed when it is exceeded.

[0029] 4. Interest target ranking ① Gaze judgment For the current detected frame image, the longer the trajectory of each target corresponding to the target is, the more attention it receives. ② Proximity judgment For the current detected frame image, the longer the trajectory of each target corresponding to the target is, the more attention it receives.

[0030] ③ Derivative judgment For the current detected frame image, the longer the trajectory of each target corresponding to the target is, the more attention it receives. The derivative judgment judges whether the target is actively approached by the user, and calculates the average change rate (i.e. first derivative) of the depth between adjacent frames: Since the derivative is a negative number, the smaller the derivative is, the more the user wants to approach the target, and the more attention the target receives.

[0031] In order to improve the interaction efficiency and response accuracy of the system, all the feedback targets are ranked according to multiple indicators to ensure that the user obtains the most relevant and urgent interaction feedback.

[0032] The target priority ranking is performed according to the following rules: Frequency of occurrence priority: in the sliding time window The longer the target trajectory is, the higher the user's attention is, and the higher the ranking is. Proximity priority: if the frequency of occurrence is the same, the average change rate of the target depth , The smaller the average change rate is, the closer the target is to the user, and the higher the ranking is.

[0033] 5. Recognition feedback After ranking, the image information of the target and its corresponding complete trajectory feature Queue (ID) are input into the visual language model, and the target is analyzed in multiple modalities. The user is prompted and fed back through voice or vibration. In addition, the following situations need to be handled specially: ① Hazard identification: if the model identifies that there is a dangerous factor (such as a sharp object, a rapidly approaching obstacle, a flame, etc.) in the target, the system will trigger an active alarm mechanism, and use voice broadcast, vibration feedback, etc. to warn the user.

[0034] ​​​②Environmental memory: At the same time, the system also stores the extracted features, completed vector calculations, and the recognition results and interaction records fed back by the visual language large model into the memory unit. Through the reuse mechanism, the information is used for subsequent environmental recognition, semantic recall, and context association. Users can actively call these memory contents through voice interaction to avoid repeated calculation and repeated feedback, and realize the complete closed-loop interaction of "perception-cognition-memory-feedback".

[0035] In summary, the blind glasses target recognition method based on consciousness judgment in the embodiment, through "interest target recognition", aims to automatically identify the image target that the user may currently focus on or perceive from the image sequence collected by the user wearing the device, and provide effective input basis for subsequent consciousness judgment, information broadcast and other processes. The recognition process integrates target detection, cross-frame tracking, interest target sorting and other visual processing mechanisms to ensure that the identified target has high relevance and practicality.

[0036] In a specific implementation, the system jointly analyzes a plurality of continuous image frames. First, a general target detection model (such as YOLO) is used to identify key targets (such as pedestrians, vehicles, obstacles, traffic signs, etc.) in the image, and generate feature information for each detected target, including class labels, bounding box coordinates, and confidence scores. Subsequently, for each detected target, the system further extracts its image semantic features (semantic embedding vectors generated by convolutional neural networks or visual language model encoders), geometric center position, bounding box size (width and height), depth information, and time stamp of the frame collected, etc. Key attributes to build structured target feature vectors for subsequent trajectory tracking, interest sorting, etc.

[0037] In order to enhance the continuity and stability of interest target recognition, the system is based on cross-frame target tracking technology to match the targets identified in the current frame with the targets in the previous frame, record the same target appearing continuously into a unified trajectory queue, and construct the time sequence trajectory of the target. If a target appears for the first time, a new trajectory queue is created for it; if it already exists in the previous frame, it is appended to the corresponding queue; if a target disappears in the current frame, its corresponding trajectory queue will be cleared. Through this mechanism, the system can effectively record and model the appearance frequency and time sequence behavior of each candidate target.

[0038] On this basis, the system further infers the interest degree of the user according to the trajectory performance of the target in the continuous image frames. Specifically, the longer the length of the trajectory queue corresponding to the target, the longer the target appears continuously in the user's field of view, and the higher the possibility of being focused. If the trajectory lengths are the same, the one with faster average displacement speed is selected, which is considered to have more distinctiveness or behavior characteristics, and thus a higher interest degree is inferred. According to the above rules, the system sorts the candidate targets in the current frame according to the interest degree, and the earlier the sorting, the more likely the target is the "most interested target" of the user.

[0039] The system inputs the high-priority target and its complete trajectory in the interest sorting result into the visual language model for further semantic analysis. When a potential dangerous target (such as an oncoming obstacle or a fast-moving vehicle) is identified, the system can actively trigger a warning mechanism. At the same time, relevant semantic vector calculation results, interactive feedback states, and interest sorting records and other information will be stored in the system memory module, thereby avoiding repeated processing or broadcasting of known targets in subsequent image frames and improving the system running efficiency and the smoothness of user experience.

[0040] The above only describes the preferred embodiments of the present application, and the protection scope of the present application is not limited to the above-mentioned embodiments. Any technical solutions falling within the concept of the present application shall be within the protection scope of the present application. It should be noted that for ordinary technical personnel in the technical field, some improvements and refinements without departing from the principles of the present application shall also be considered within the protection scope of the present application.

Claims

1. A blind glasses target recognition method based on consciousness judgment, characterized in that: Comprising the following steps: Step one, collect F frames of image sequences per second through binocular or monocular depth camera, and perform preliminary processing for subsequent target segmentation and feature extraction; Step two, perform target detection on each frame of image based on a general target detection model, and extract a key target set; Step three, perform cross-frame target tracking based on the key target set extracted in step two; Step four, identify an interest target, and sort the interest target; Step five, after sorting, input the image information of the target and its corresponding complete trajectory feature Queue(ID) into a visual language model, perform multi-modal semantic analysis on the target, and feed back to the user through voice or vibration prompt; Wherein, if the model identifies that there is a dangerous factor in the target, an active alarm mechanism will be triggered, and a warning will be issued to the user through voice broadcast and vibration feedback. At the same time, the extracted features, completed vector calculation, and the recognition results and interaction records fed back by the visual language model will be stored in the memory unit, which will be used for subsequent environment recognition, semantic recall and context association through reuse mechanism. 2.The method of claim 1, wherein the method further comprises: determining a distance between the blind person and the target object based on the at least one of the first and second images. The specific way of extracting the key target set in step two is as follows: Assume the number of targets is k, denoted as ; Each target The following attributes are extracted to form the state vector: ① image semantic features obtained by image encoding; ②center point coordinates : record the center point coordinates of the target ; 3) bounding box size : represents the width and height of the pixels of the object; (iv) target depth value : for a target, extract its corresponding set of depth values, and compute the mean or median of the set as the representative depth value of the target; • Time stamp : Records the time of acquisition of the frame. 3.The method of claim 1 or 2, wherein: The specific method for cross-frame target tracking in step three is as follows: For consecutive frames... and , Comparison of the first Frame each target Compared to the previous frame The overall similarity of all targets in a frame includes the following steps: Step three, identify the number of targets within the image, obtain the first Total number of targets identified in the frame picture ; Step three two, set a minimum similarity threshold, and consider that it is not similar if it is lower than the threshold, and consider that it is similar if it is higher than the threshold; Step three, identify the number of targets within the image, obtain the first Total number of targets identified in the frame picture ; Step three four, calculate the overall similarity of all targets by the following formula to match similar targets: ； In the formula, : target bounding box overlap degree, : visual feature cosine similarity, : position proximity is modeled by a Gaussian function constructed by the distance between the center points, and the position sensitivity parameter represents the maximum tolerance offset range, and the default setting is σ = 20 pixels; : spatial overlap weight : appearance consistency weight : position continuity weight; Step three five, if no similar target is matched, give the first An independent ID is created for the current target of the frame picture, a track queue is created for the current target, and the queue is placed at the head of the queue. Step three six, if a similar target is matched, obtain the ID of the most similar target of the tth frame, and the current target of the t+1th frame inherits the ID, and the current target is placed at the head of the queue.

4. The method of claim 3, wherein the method further comprises: Each ID in steps three five and three six maintains the following attribute time sequence: Depth sequence: ; Time series: ; The maximum length of the trajectory is set to n, and it is automatically updated when it exceeds.

5. The method according to claim 1 or 2, wherein the method is characterized by: The specific steps of sorting the interest target in step four are as follows: Step four, judging whether the current detected first target is the target of interest according to the length of the first target in the frame image and the length of the corresponding each target in the frame image frame image, the length of the corresponding each target in the frame image frame image, the length of the corresponding each target in the frame image frame image, the length of the corresponding each target in the frame image Step four two, for the current detection of the first frame image, the trajectory queue of each target Derivative judgment whether the target is actively approached by the user, calculate the average change rate of depth between adjacent frames: ； wherein, is negative, thus The smaller, the more the user wants to approach the target, the more attention the target is completed to approach the judgment; Step four three, sort the target priority based on the gaze judgment result and the proximity judgment result in steps four one and four two.

6. The method of claim 5, wherein the method further comprises: The rules for sorting in step four three are as follows: Frequency priority: within the sliding time window The longer the target trajectory, the higher the user's attention is considered, and the higher the ranking is. Proximity priority: if the frequency is the same, then calculate the average change rate of the target depth , The smaller the value, the closer the target is to the user, and the higher the ranking.

Images