AR intelligent glasses system based on brain-like visual processing

Abstract

The invention relates to the technical field of AR intelligent glasses, in particular to an AR intelligent glasses system based on brain-like visual processing. The system comprises a visual data acquisition module, a visual feature extraction module, a semantic construction module, a behavior intention prediction module and a visual interaction module. Visual images and environment information are acquired through a multi-mode sensor, and spatial-temporal feature coding is performed through a pulse neural network to generate a brain-like visual feature sequence; constructing a semantic expression vector based on a graph structure method; a comparative learning model is adopted to complete behavior intention prediction; and the visual interaction module generates AR (Augmented Reality) content according to a prediction result, so that real-time interaction presentation is realized. According to the method, multi-modal perception precision, behavior understanding intelligence and interaction personalization enhancement are realized, and the perception, understanding and response capabilities of an AR system in a complex application scene are comprehensively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of AR smart glasses, and particularly relates to an AR smart glasses system based on brain-like visual processing. BACKGROUND

[0002] In related technologies, with the rapid development of augmented reality (AR) technology, AR smart glasses, as a new wearable device integrating visual perception and human-computer interaction, have gradually been applied to many fields such as industrial manufacturing, medical assistance and education training. The existing AR smart glasses usually rely on traditional computer vision methods to perceive and understand the external environment, and realize the superposition and display of augmented information through target detection and recognition of captured images.

[0003] However, in complex scenes such as dramatic changes in light, significant background interference or diversified user behaviors, traditional visual processing algorithms are prone to problems such as unstable recognition and lagging response, affecting the interactivity and practicality of the system. Secondly, the existing AR augmented presentation methods are mostly static presentation mechanisms based on fixed rules or preset scenes, which cannot dynamically adjust the augmented interactive content according to the real-time behavior intention of the user, resulting in lagging interactive expression, insufficient immersion and weak credibility, and cannot guarantee real-time performance and adaptability in complex, open and dynamically changing real scenes. In addition, the existing AR systems generally rely on low-dimensional feature vectors or shallow semantic embedding, which is difficult to fully capture the spatial structure and semantic dependency relationship in visual data, resulting in ambiguous behavior semantic expression.

[0004] To sum up, the technical problems in related technologies need to be improved. SUMMARY

[0005] The main purpose of the embodiments of the present application is to propose an AR smart glasses system based on brain-like visual processing, aiming to realize multi-modal perception precision, intelligent behavior understanding and personalized augmented interaction, and comprehensively improve the perception, understanding and response capabilities of the AR system in complex application scenarios.

[0006] To achieve the above purpose, one aspect of the embodiments of the present application proposes an AR smart glasses system based on brain-like visual processing, comprising: a visual data acquisition module, configured to acquire visual data of the AR smart glasses in real time through a multi-modal sensor; a visual feature extraction module, configured to perform spatiotemporal feature coding and sparse pulse mapping on the visual data by using a brain-like visual algorithm based on a pulse neural network, and generate a brain-like visual feature sequence; a semantic construction module, configured to construct a visual semantic expression vector based on a graph structure-based semantic association modeling method based on the brain-like visual feature sequence; The behavior intention prediction module is configured to perform behavior intention prediction based on the semantic expression vector by using a behavior intention decoding model based on contrast learning, and generate a behavior intention prediction result. The visual interaction module is configured to dynamically generate AR augmented presentation content based on the intention prediction result, and perform visual interaction output on a display end of the AR smart glasses.

[0007] In some embodiments, the visual data includes visual images, inertial parameters, voice signals, and environmental state information.

[0008] In some embodiments, the construction process of the visual feature extraction module includes: The visual data is time-series pulse coded by using a pulse firing threshold dynamic adjustment mechanism, and a time-series pulse event stream is generated; Based on the time-series pulse event stream, multi-scale sparse pulse convolution calculation is performed by using a brain-like visual algorithm based on a pulse neural network, and deep brain-like visual features are extracted; The deep brain-like visual features are sparsely mapped and compressed, and a brain-like visual feature sequence is generated.

[0009] In some embodiments, the visual data is time-series pulse coded by using a pulse firing threshold dynamic adjustment mechanism, and a time-series pulse event stream is generated, including: The pulse firing threshold is initialized; The pulse firing threshold is adaptively adjusted based on local pixel gradient changes and dynamic contrast changes of the visual data; The continuous visual data is converted into discrete pulse events according to the adjusted pulse firing threshold; The time-series pulse event stream is generated based on the discrete pulse events, and the pulse firing threshold is continuously updated during the event stream generation process.

[0010] In some embodiments, the formula of the brain-like visual algorithm is as follows: ; Wherein, represents an output brain-like visual feature response function, that is, a sparse pulse feature intensity at a spatial position and a time ; represents an excitation function of a pulse neuron; represents a pulse convolution kernel weight of the scale; represents an input time-series pulse event stream signal of the layer; represents a continuous visual image at a pixel coordinate and a time sequence pixel intensity data of the visual image at a time point t; pixel intensity temporal gradient changes of the visual image between adjacent frames; local spatial structure responses of the visual image processed by a Laplacian operator in a spatial dimension; a temporal response adjustment coefficient; a spatial sparsity adjustment coefficient; N represents a number of layers of a sparse pulse convolution layer.

[0011] In some embodiments, the construction process of the semantic construction module includes: based on the brain-like visual feature sequence, a graph structure semantic representation corresponding to nodes and edges is established; the nodes represent target feature points, objects or regions in a visual scene; the edges represent spatial adjacency relationships or semantic dependency relationships between different nodes; According to the graph structure semantic representation, a graph structure-based semantic association modeling method is used to perform graph convolution propagation and weighted aggregation on the semantic correlation between nodes to form a multi-level semantic embedding representation; Global context normalization and attention weight adjustment are performed on the semantic embedding representation to generate a visual semantic expression vector; the visual semantic expression vector is used to represent the semantic connection and structural hierarchy between different objects in the visual scene.

[0012] In some embodiments, the construction process of the behavior intention prediction module includes: Based on the visual semantic expression vector, a semantic comparison sample pair of a positive sample semantic vector and a negative sample semantic vector is constructed; the positive sample semantic vector is used to represent the semantic expression distribution corresponding to the target behavior intention, and the negative sample semantic vector is used to represent the semantic interference distribution corresponding to the non-target behavior intention; A behavior intention semantic separation feature space is obtained by using a behavior intention decoding model based on contrastive learning to perform semantic differentiation training on the positive sample semantic vector and the negative sample semantic vector; Based on the behavior intention semantic separation feature space, the visual semantic expression vector is decoded and mapped in the inference stage to obtain a target behavior intention prediction result.

[0013] In some embodiments, the behavior intention decoding model has the following formula: ; wherein, represents a behavior intention prediction score at a semantic sample pair index ; represents a nonlinear semantic discriminant function; M represents a dimension number of a contrastive semantic subspace; represents a weight coefficient of an i-th semantic subspace; ; represents the embedding representation of the positive sample semantic vector in the semantic subspace; represents the embedding representation of the negative sample semantic vector in the semantic subspace; cos(·) represents the cosine similarity measure between semantic vectors; represents a tiny constant to prevent the denominator from being zero; represents the context semantic consistency coefficient between the sample pair; represents the hyperbolic tangent activation function; represents the context consistency adjustment coefficient; represents the scaling factor of the sensitivity of the semantic subspace to the context consistency.

[0014] In some embodiments, the visualization interaction module performs high-density rendering output in the visual focus area and low-density rendering output in the non-focus area by tracking the user's visual line trajectory in real time.

[0015] To achieve the above object, another aspect of the embodiment of the present application provides an implementation method of an AR intelligent glasses system based on brain-like visual processing, comprising: acquiring visual data of the AR intelligent glasses in real time through a multi-modal sensor; performing spatiotemporal feature coding and sparse pulse mapping on the visual data by using a brain-like visual algorithm based on a pulse neural network to generate a brain-like visual feature sequence; constructing a visual semantic expression vector based on the brain-like visual feature sequence by using a graph structure-based semantic association modeling method; performing behavior intention prediction by using a behavior intention decoding model based on contrast learning based on the semantic expression vector to generate a behavior intention prediction result; generating AR augmented presentation content dynamically based on the intention prediction result and performing visual interaction output on a display end of the AR intelligent glasses.

[0016] To achieve the above object, another aspect of the embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the above system when executing the computer program.

[0017] To achieve the above object, another aspect of the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program implements the above system when executed by a processor.

[0018] To achieve the above object, another aspect of the embodiment of the present application provides a computer program product, comprising a computer program, and the computer program implements the above system when executed by a processor.

[0019] The application provides an AR intelligent glasses system based on brain-like visual processing, which can comprehensively perceive the environment and interaction state of a user by fusing visual data of multi-modal sensors through a visual data acquisition module, realizes real-time input of high-dimensional information, and lays a data foundation for subsequent feature extraction and semantic understanding. The brain-like visual algorithm based on pulse neural network is used to process the visual data, which can be closer to the pulse information processing mode of human brain neurons, effectively realize sparse coding, dynamic response and low-power processing, and improve the feature extraction accuracy and real-time performance in a high dynamic environment. The semantic construction module uses a graph structure modeling method to model the time sequence relationship and spatial correlation in the brain-like visual feature sequence, extracts a semantic expression vector with more context meaning, and enables the system to have the ability to transition from "perception" to "understanding", and improves the accuracy of semantic reasoning. The behavior intention prediction module introduces a behavior intention decoding mechanism based on contrast learning, optimizes the embedding distance of positive and negative samples, improves the model's ability to distinguish similar behaviors, and enhances the generalization recognition performance of behavior intention in different users and different scenes. The visual interaction module dynamically generates augmented reality content combined with the prediction result, and directly displays it on the AR glasses terminal, realizes the closed-loop connection between the user's behavior intention and the interactive output, enables the system to have an immersive and adaptive augmented display interaction experience, and significantly improves the naturalness and efficiency of human-computer interaction. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 Fig. 1 is a schematic diagram of a module of an AR intelligent glasses system based on brain-like visual processing provided by the application; Figure 2 Fig. 2 is a flowchart of a visual feature extraction module constructed in Fig. 1; Figure 1 Figure 3 Fig. 3 is a flowchart of a behavior intention prediction module constructed in Fig. 1. Figure 1 DETAILED DESCRIPTION

[0021] In order to make the purpose, technical scheme and advantages of the application clearer, the following further describes the application in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the application and not to limit the application. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementation described in the following example embodiments does not represent all the implementations consistent with the embodiments of the application, but is only an example of devices and methods consistent with some aspects of the embodiments of the application as described in the appended claims.

[0022] ​​It can be understood that the terms "first", "second", etc. used in the present application can be used herein to describe various concepts, but unless specifically stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "upon determining" or "in response to determining".

[0023] The terms "at least one", "multiple", "each", "any" and the like used in the present application include one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as understood by a person skilled in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0025] An AR intelligent glasses system based on brain-like visual processing is provided in the embodiments of the present application. The scheme fuses visual data of multi-modal sensors through a visual data acquisition module, can comprehensively perceive the environment and interaction state of the user, realizes real-time input of high-dimensional information, and lays a data foundation for subsequent feature extraction and semantic understanding. The brain-like visual algorithm based on pulse neural network is used to process the visual data, which can be closer to the pulse information processing mode of human brain neurons, effectively realize sparse coding, dynamic response and low-power processing, and improve the feature extraction accuracy and real-time performance in high dynamic environment. The semantic construction module uses a graph structure modeling method to model the time sequence relationship and spatial correlation in the brain-like visual feature sequence, extracts semantic expression vectors with more context meaning, and enables the system to have the ability to transition from "perception" to "understanding", and improves the accuracy of semantic reasoning. The behavior intention prediction module introduces a behavior intention decoding mechanism based on contrast learning, optimizes the embedding distance of positive and negative samples, improves the ability of the model to distinguish similar behaviors, and enhances the generalization recognition performance of behavior intention in different users and different scenes. The visual interaction module dynamically generates augmented reality content combined with the prediction result, and directly displays it on the AR glasses terminal, realizes the closed-loop connection between the user behavior intention and the interaction output, enables the system to have an immersive and adaptive augmented display interaction experience, and significantly improves the naturalness and efficiency of human-computer interaction.

[0026] The embodiment of the application provides an AR smart glasses system based on brain-like visual processing, and relates to the technical field of AR smart glasses.

[0027] The application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, small computers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The application can also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0028] Please refer to Figures 1-3 The application relates to an AR smart glasses system based on brain-like visual processing.

[0029] Figure 1 is an optional flowchart of the AR smart glasses system based on brain-like visual processing provided by the embodiment of the application, Figure 1 The system in the embodiment of the application can include but is not limited to including: The visual data acquisition module is used for acquiring visual data of the AR smart glasses in real time through a multi-modal sensor; wherein the visual data includes visual images, inertial parameters, voice signals and environmental state information.

[0030] In this embodiment, the visual data acquisition module is deployed in the front-end perception hardware layer of the AR smart glasses, and real-time global information of the human-machine environment is acquired through multi-modal sensors. The multi-modal sensors include a binocular RGB camera, an infrared depth camera, an IMU inertial unit, a microphone array, an ambient light and color temperature sensor, and a near-eye visual line tracking sensor. The details are as follows: The binocular RGB camera is located on the left and right sides in front of the glasses, and realizes stereo vision acquisition of the real scene through structural arrangement. It collects continuous image sequences at 60-120 fps to ensure visual continuity in fast motion scenes; at the same time, it automatically adjusts the exposure time and gain according to the scene dynamics. The built-in ISP (Image Signal Processor) is used to perform noise reduction, color calibration, and HDR fusion to maintain clarity in indoor high-contrast environments. At the acquisition end, a rough disparity map can be generated through hardware acceleration for subsequent spatiotemporal encoding reference of the brain-like vision model.

[0031] The infrared depth camera is used to supplement the structural information of the scene, so that the brain-like vision model can combine geometric features for spatiotemporal reasoning. Its acquisition process includes: emitting an array of coded light spots forward, obtaining three-dimensional depth point cloud according to the reflection light deformation, and adapting to indoor, night, and other low-light scenes. Spatial calibration of depth map and RGB map is performed at the acquisition end, so that the back-end model can directly use the aligned multi-channel image input. When the user moves his head quickly, use IMU pose information to compensate the depth frame for motion, reduce jitter and holes.

[0032] The IMU inertial unit contains a three-axis accelerometer, a three-axis gyroscope, and a magnetometer, which is used to capture head pose, view transformation, and motion trajectory. Its acquisition details include: enabling visual features to be closely associated with motion state, realizing time-domain synchronous coding in brain-like vision algorithm. Roll, pitch, and yaw angles are generated through hardware-level fusion algorithm, which are used to predict the change of field of view and synchronize with the back-end vision encoding. The magnetic force meter is used to periodically correct the inertial drift, improving the stability of the attitude under long-time operation.

[0033] The microphone array is used to collect voice signals, environmental noise, and spatial sound field direction information. Through sound source positioning, the direction of the user's voice is determined, and the noise on the side and behind is suppressed. Noise reduction and echo cancellation algorithms are run at the front end to provide clearer voice modality input for graph semantics construction. Capture footstep sound, device noise, and sudden sound sources, etc. as scene label nodes in the semantic relationship graph.

[0034] The ambient light and color temperature sensor is used to assist in adjusting the rendering brightness and the light condition judgment of the vision model. It automatically adjusts the camera exposure and the display end brightness in outdoor / indoor / backlight environments. It ensures the color consistency of the visual data before entering the brain-like vision model.

[0035] Near-eye visual line tracking sensor is used to capture user eye rotation and observation intention. Eye reflection feature points are collected at high frame rate, and pupil center and pupil edge contour are identified. Eye posture is mapped to visual field coordinates to determine user focus area. It is used for decision of high-density rendering area of visual interaction module, attention area verification of intention prediction module, etc.

[0036] The visual data acquisition module is embedded with a multi-modal data synchronous sampling control unit. The unit controls the sampling process of different sensors in time synchronization through a unified clock domain control strategy. Specifically, the system uses a high-precision crystal oscillator clock as the main time base, and uses a synchronous trigger signal to periodically trigger each sensor to sample, so as to ensure that the alignment accuracy of different modal data on the time axis is not more than 1 millisecond.

[0037] In addition, in order to support subsequent brain-like visual processing, the visual data acquisition module also has a multi-modal buffer and frame mechanism. Based on a shared memory area, the mechanism constructs a multi-modal buffer structure, frames the sampling data of each mode based on time as the key value, and forms a multi-modal data packet containing visual image frames, inertial frames, speech frames and environment frames. Each data packet has a unified timestamp and frame index number, and can be quickly transmitted to the visual feature extraction module through the DMA (Direct Memory Access) mechanism to realize zero-copy data transfer.

[0038] The visual feature extraction module is used to encode and sparse pulse map the visual data in space-time feature using a brain-like visual algorithm based on a pulse neural network, and generate a brain-like visual feature sequence; As shown in the reference Figure 2 The construction process of the visual feature extraction module includes: The visual data is encoded in time sequence using a pulse firing threshold dynamic adjustment mechanism to generate a time sequence pulse event stream; Based on the time sequence pulse event stream, a brain-like visual algorithm based on a pulse neural network is used for multi-scale sparse pulse convolution calculation to extract deep brain-like visual features; In this embodiment, the time sequence pulse event stream is input into a multi-layer brain-like visual processing structure. The structure is divided into multiple adjacent time windows along the time dimension. In each time window, a local pulse map is constructed according to the spatial position and time sequence of the events.

[0039] In the first layer processing stage, multi-scale convolution kernels of different receptive field sizes are used to perform sparse convolution operation on the pulse map. Small receptive fields are used to capture fine edges and texture changes, and large receptive fields are used to perceive object contours and large-scale motion trends. When the pulses in a certain spatial region are frequent and have consistent directional changes, the region will output a stronger response at the corresponding scale, thereby forming encoding of low-level features such as edges, corners and motion directions.

[0040] In the deeper processing stage, the brain-like vision algorithm introduces the accumulation and inhibition mechanism in time, integrates the pulse patterns that exist continuously or repeatedly, and links the short-term local motion with the long-term trajectory change. For example, for the user gesture, moving object or interactive operation of interface elements, the system extracts the stable motion trajectory features and shape change features by superimposing and screening the pulse responses in multiple time windows.

[0041] At the same time, by setting sparse constraints and inhibition strategies between layers, only the most representative responses in the same local area are retained, and unimportant or repeated responses are suppressed, so that the brain-like vision features of the deep output remain highly sparse in space and time, highlighting the key targets and significant actions. Finally, after multi-scale convolution and multi-layer accumulation processing, the deep brain-like vision features representing scene structure, motion pattern and significant area are obtained.

[0042] The deep brain-like vision features are sparsely mapped and compressed to generate a brain-like vision feature sequence.

[0043] In this embodiment, the system performs spatio-temporal reorganization and compression on the deep features obtained by multi-layer brain-like vision processing. First, the deep features in each time window are arranged in time according to the time sequence, and a spatial aggregation operation is performed on the feature maps in each time window to combine the feature responses of adjacent regions and form several representative spatial feature points, each of which records the most prominent edge, motion or saliency information in the region.

[0044] Subsequently, these spatial feature points are analyzed on the time axis, and the features that exist stably or frequently are considered as key features, and the accidental and isolated features are considered as noise or secondary information and are weakened or deleted. In this way, the redundant information in the entire time sequence is greatly compressed, and only the most valuable features for subsequent semantic construction and behavior intention prediction are retained.

[0045] After completing the spatio-temporal screening and compression, the remaining features are organized into a brain-like vision feature sequence according to the time sequence. Each item in the feature sequence corresponds to a high-level visual representation in a time segment, containing the position, contour change, motion trend of the significant target in the scene and the significant area information related to the user's line of sight.

[0046] In practical applications, the brain-like visual feature sequence can be aligned with the rendering frame rate of the AR smart glasses, for example, a corresponding feature vector is output for a certain number of frames of images as a time slice, so as to realize the effect of "one high-abstraction brain-like visual description corresponding to each scene segment" in the subsequent semantic construction module, so that the whole feature sequence has sufficient time resolution, maintains high sparsity and compactness, and is suitable for real-time processing and reasoning on an end-side device.

[0047] The visual data is time-sequentially pulse coded by adopting a pulse firing threshold dynamic adjustment mechanism to generate a time-sequential pulse event stream, including: initializing the pulse firing threshold; In this embodiment, when the system is started or the scene is switched, firstly, a plurality of frames of continuously collected visual images are preprocessed, including grayscale, normalization and noise filtering, and the overall brightness distribution and contrast level of the current scene are counted. According to the statistical result, an initial pulse firing threshold is set for each pixel point or pixel block, for example, a higher threshold is set in a high-contrast scene to avoid a large number of meaningless pulses caused by noise or slight changes; the threshold is appropriately lowered in a low-contrast or weak-light scene to ensure that subtle motion or brightness changes can still be triggered as pulse events. At the same time, it can be judged whether there is a violent motion according to the inertia parameter, and a slightly higher threshold is initialized in the state of violent motion to suppress the pseudo-pulses caused by motion blur.

[0048] The pulse firing threshold is adaptively adjusted based on the local pixel gradient change and dynamic contrast change of the visual data; In this embodiment, in the normal operation of the system, the local pixel gradient change and the regional contrast change are calculated by differentiating each frame of visual image and the previous frame of image. When the gradient value of a certain local region is continuously large, it indicates that there is a clear edge or moving target in the region, and the threshold of the corresponding region is appropriately lowered to make the detail change more easily trigger pulses; on the contrary, when the gradient change of a certain region is small and the contrast is stable for a long time, the threshold of the region is gradually increased to reduce the pulse firing on the static background.

[0049] At the same time, the overall threshold is globally fine-tuned according to the environmental state information (such as light mutation): when the light suddenly increases is detected, the global threshold is increased in a short time to suppress the pseudo-events caused by a large number of transient changes; when the environment becomes dark, the threshold is lowered to maintain the sensitivity to slight brightness changes. For the region where there is a voice instruction or a high-priority task prompt, the threshold of the relevant field of view region can be appropriately lowered to make the target related to the current task more prominent.

[0050] The continuous visual data is converted into discrete pulse events according to the adjusted pulse firing threshold; In this embodiment, after completing the threshold adaptive update, the current frame is compared with the previous frame image pixel by pixel, and if the brightness change amplitude or gradient change of a certain pixel exceeds the threshold value set for the pixel, a pulse event is generated at the pixel position; if the threshold value is not exceeded, no event is generated. For inertia parameters, the head rotation speed, acceleration, etc. can be mapped as threshold correction terms, so that only the target area with stable and continuous changes is generated when the head is quickly rotated, thereby filtering out large-area changes caused by overall view changes.

[0051] The generated pulse events are encoded with information such as "pixel position, timestamp, change direction (lightening or darkening), channel category", forming a series of discrete events, and the complete frame image is no longer retained, thereby greatly reducing the amount of redundant data.

[0052] Based on the discrete pulse events, a time-sequential pulse event stream is generated, and the pulse firing threshold is continuously updated during the event stream generation process.

[0053] In this embodiment, the discrete pulse events generated at multiple time steps are arranged in chronological order to form a time-sequential pulse event stream, which is used for subsequent brain-like visual computing. In this process, the system continuously monitors the pulse firing frequency of each pixel or pixel block within a certain time window: if a certain area has too dense pulses in a short time, it is considered that the threshold value of the area is too low, and the threshold value of the area is gradually increased; if there is almost no pulse in a certain area for a long time, the threshold value of the area is gradually reduced to avoid missing subtle changes.

[0054] Through this "event-driven + threshold feedback" cycle mechanism, the time-sequential pulse event stream remains sparse and representative in the time dimension, and can reflect key dynamic changes and significant structural information in the scene with a lower data volume, providing high-quality input for subsequent deep brain-like visual feature extraction.

[0055] Further, the formula of the brain-like visual algorithm is as follows: ; Wherein, represents the output brain-like visual feature response function, i.e. the sparse pulse feature intensity at spatial position and time ; represents the firing function of the pulse neuron, which is used to simulate the discharge mechanism of the biological neuron; represents the scale pulse convolution kernel weight, which is used to extract visual pulse features at different scales; represents the input layer time-sequential pulse event stream signal; represents the continuous visual image in pixel coordinates and time series pixel intensity data under the time series; representing the pixel intensity temporal gradient change of the visual image between adjacent frames; representing the local spatial structure response of the visual image after being processed by a Laplacian operator in the spatial dimension, for highlighting edge structure, object contour and salient region; representing a time response adjustment coefficient, for adjusting the influence intensity of the temporal gradient term in the pulse feature calculation; representing a spatial sparsity adjustment coefficient, for balancing spatial structure enhancement and background noise suppression; N represents the number of layers of the sparse pulse convolution layer.

[0056] a semantic construction module, for constructing a visual semantic expression vector based on a graph structure semantic association modeling method based on the brain-like visual feature sequence; The construction process of the semantic construction module includes: based on the brain-like visual feature sequence, a graph structure semantic representation corresponding to nodes and edges is established; wherein the nodes represent target feature points, objects or regions in the visual scene; and the edges represent spatial adjacency relationships or semantic dependency relationships between different nodes; In this embodiment, in a specific embodiment, the system first receives the brain-like visual feature sequence output by the previous module, each time corresponding to a set of high-dimensional feature vectors, for describing the salient targets, motion trajectories and region features in the current scene.

[0057] In each time slice, the system performs spatial clustering and target detection on the brain-like visual feature sequence, aggregates features with similar positions, motion directions or visual attributes into a number of candidate objects, such as pedestrians, vehicles, signs, user hands, interface virtual buttons, etc.; at the same time, for regions that do not form complete targets but are near the line of sight or have high saliency, these regions are introduced into the graph structure as region nodes according to the preset grid or superpixel division rules.

[0058] Finally, each node is bound with the following information: position coordinates in the scene, target class label (such as "person", "gesture", "text region", "UI element"), time index, and corresponding brain-like visual feature vector, for subsequent semantic modeling.

[0059] In the spatial dimension, the system calculates the proximity between nodes according to the relative position relationship of each node in the image plane. For example, for nodes within a certain pixel range, they are identified as spatially adjacent nodes, and a spatial connection relationship is established between them; for nodes located near the focus of the user's line of sight, the system gives priority to the connection with surrounding nodes when constructing the graph, so that the line of sight region has a more dense connection structure, thereby improving the semantic weight of the interaction-related region.

[0060] At the semantic level, the system constructs semantic dependencies according to the target category, motion pattern and historical co-occurrence relationship of the nodes: If some nodes frequently appear simultaneously for a long time (e.g. face and nameplate, hand gesture and virtual button), a strong semantic connection is established between them to represent the semantic co-occurrence relationship of "frequently appearing together"; If a node is of key significance to the task (e.g. navigation arrow, prompt box, warning sign), a guiding semantic relationship is established between the user-related nodes (e.g. gaze focus, hand position); If there is an obvious inclusion or subordination relationship between nodes (e.g. text area inside the billboard area), a hierarchical dependency relationship is established.

[0061] Through the above spatial adjacency relationship and semantic dependency relationship, the system organizes all nodes and their connection relationships into a graph structure semantic representation for subsequent semantic association modeling of the graph structure.

[0062] According to the graph structure semantic representation, a graph structure-based semantic association modeling method is used to perform graph convolution propagation and weighted aggregation of the semantic correlation between nodes to form a multi-level semantic embedding representation; In this embodiment, for each node in the graph, the system traverses its adjacent node set in the current layer and aggregates the features of the adjacent nodes combined with the relationship information represented by the edges. For example, when a node represents a "button object" and its adjacent nodes include "gaze focus area" and "hand trajectory area", in the aggregation process, the system will emphasize the features of these interaction-related nodes, making the updated representation of the button node more prominent in the semantic properties of "being focused" and "to be operated".

[0063] In this process, different sources of edges can be assigned different semantic weights: edges generated by spatial adjacency emphasize position and proximity, edges generated by semantic dependency emphasize functional relationship and interaction logic, and edges generated by temporal co-occurrence emphasize simultaneous or sequential appearance patterns. Through the comprehensive aggregation of these information, the updated node representation of each layer not only contains its own local attributes, but also incorporates the contextual semantics from the neighborhood.

[0064] After the local aggregation of a layer is completed, the updated node representation is taken as the input of the next layer, and the similar neighborhood aggregation and relationship weighting process is performed again. As the number of layers increases, the node can perceive semantic information at a longer distance and more complex path, realizing semantic diffusion from local relationship to global structure.

[0065] For example, the node representing the "target location" can gradually perceive the semantics of the "navigation arrow" "cue sign" "obstacle" nodes related to its path through multi-layer propagation, and then embody higher-level semantic features such as "reachability" "path complexity" "whether blocked" in its embedded representation.

[0066] After multi-level propagation, each node in the graph obtains an embedded representation containing local features, neighborhood structure, and global semantics, forming a multi-level semantic embedding set, providing a rich foundation for subsequent global semantic expression.

[0067] Global context normalization and attention weight adjustment are performed on the semantic embedding representation to generate a visual semantic expression vector; the visual semantic expression vector is used to represent the semantic relationship and structural hierarchy between different objects in the visual scene.

[0068] In this embodiment, after obtaining the multi-level node semantic embedding, the system needs to compress the semantic information of the entire graph into a global vector as the output of the visual semantic expression vector.

[0069] First, the system performs statistical analysis on the semantic embedding of all nodes in the entire graph structure, calculates the distribution of various semantic features in the global range, and identifies which features are commonly present in the current scene and which are key features.

[0070] For features representing the background environment or having strong repetition (such as large-area walls, floors, and areas with no obvious semantics), the system reduces their influence through normalization and weight reduction to avoid these features occupying too much space in the final expression vector.

[0071] For features related to the current task, high saliency areas, and user gaze, the system appropriately preserves their relative amplitude during normalization, making them more easily focused on in subsequent attention adjustment.

[0072] Based on global context normalization, the system introduces an attention mechanism to assign different attention weights to different nodes and different types of semantic features: For nodes located near the focal point of the gaze, which are highly related to user gestures or interaction targets, higher attention weights are assigned to make these nodes occupy a larger proportion in the final visual semantic expression; For nodes representing potential risks or important cues, such as warning icons, navigation path inflection points, and prohibited area identifiers, even if they are not in the center of the gaze, they will be given higher weights to ensure that safety and critical information are not ignored; For background nodes or nodes with less impact on the current task, the attention weight is reduced to make them only serve as auxiliary context and be weakly retained.

[0073] By attention weight adjustment, the system forms a weighted perspective on the basis of the whole graph, so that the final expression vector is more consistent with the semantic content that the user is really "concerned" about in the current interaction context.

[0074] After completing normalization and attention weight adjustment, the system weights and fuses the semantic embeddings of all nodes according to their attention weights, compresses the graph structure from a complex representation of "multiple nodes and multiple connections" into a fixed-length vector, and outputs the vector as a visual semantic expression vector.

[0075] The visual semantic expression vector comprehensively reflects: the class distribution and spatial layout of the main objects in the scene; the semantic relationship between the user's line of sight, gestures and key objects; the structural hierarchy between the navigation target, prompt information and the surrounding environment; the relative importance between the current task-related semantics and the background semantics.

[0076] In the subsequent behavior intention prediction process, the system takes the visual semantic expression vector as input, so that it can quickly judge the user's current possible operation intention without directly processing the complex graph structure, such as "preparing to click a certain button", "following the navigation instructions to proceed", "paying attention to the risk prompt area", etc.

[0077] The behavior intention prediction module is configured to predict the behavior intention based on the semantic expression vector by using a behavior intention decoding model based on contrast learning, and generate a behavior intention prediction result. As shown in the reference Figure 3 The construction process of the behavior intention prediction module includes: Based on the visual semantic expression vector, a semantic contrast sample pair of a positive sample semantic vector and a negative sample semantic vector is constructed; wherein the positive sample semantic vector is used to represent the semantic expression distribution corresponding to the target behavior intention, and the negative sample semantic vector is used to represent the semantic interference distribution corresponding to the non-target behavior intention. In this embodiment, first, the training data is collected according to the interaction records of the AR smart glasses in actual use. Each training sample contains: a visual semantic expression vector, a current interface state, a user gaze trajectory, a gesture or head movement, a voice instruction, and a real behavior label triggered at the end (such as "select button", "page up", "zoom in", "ignore", "browse only", etc.).

[0078] For a certain target behavior intention, such as "confirmation selection", the visual semantic expression vectors within a certain time step before and after the behavior occur are selected from the historical data. These vectors have the following common characteristics: The user's gaze is focused on an interactive object, such as a virtual button, a menu item or a navigation marker; The weight of the node related to the object in the semantic expression is high, and the prompt or navigation elements related to the current task dominate in the global semantics; Subsequently, the operation result directly associated with the object occurs, such as interface jump, window opening, etc.

[0079] The semantic expression vectors are classified into a positive sample semantic vector set corresponding to the intent, for representing typical distribution of the intent in a semantic space.

[0080] For the same target behavior intent, visual semantic expression vectors are extracted from records of other behavior categories or no-intent operations as negative samples, for example: The user's gaze wanders in the interface background or non-interactive area, only for browsing but without triggering an operation; The user's attention is on other functional elements, such as viewing text instructions or observing the surrounding environment, rather than the current target button; There are still interactive objects in the scene, but the user does not generate a click, confirmation or switching behavior.

[0081] In particular, some "difficult negative samples" are also selected, that is, the scene layout and gaze position are similar to the target behavior, but the final triggered behavior is different, for example: the user gazes at a button, but finally selects another option through voice, or only stays to observe without confirming the operation. Adding such negative samples to the contrast sample pair can enhance the model's ability to distinguish subtle differences.

[0082] In the specific construction process, for each positive sample semantic vector, other vectors of similar time periods or similar scenes are selected from the positive sample set of the corresponding intent as "similar contrasts", and several negative sample semantic vectors are randomly or similarly selected from other behavior categories to form a "positive sample + multiple negative samples" semantic contrast sample pair.

[0083] By repeating the above operation in the training data set, a large number of contrast sample pairs covering different scenes, different interaction habits and different environmental conditions can be obtained, providing a rich semantic contrast basis for subsequent contrast learning.

[0084] The behavior intent decoding model based on contrast learning is used to perform semantic distinction training on the positive sample semantic vectors and the negative sample semantic vectors, to obtain a behavior intent semantic separation feature space. In this embodiment, the behavior intent decoding model takes the visual semantic expression vector as input, and gradually separates different behavior intents in the feature space through contrast learning to form a behavior intent semantic separation feature space.

[0085] First, the input visual semantic expression vector is mapped to a low-dimensional but discriminative semantic feature space. During the mapping process, the vector is normalized to make the vectors collected by different scenes and different devices comparable in scale, reducing the influence of light changes, noise and scene complexity.

[0086] During training, pre-built semantic contrastive sample pairs are input into the model in batches, with each batch containing multiple target behavioral intentions and their corresponding positive and negative samples: For positive samples with the same intent, the model adjusts its internal parameters to make these samples closer to each other in the feature space, forming a compact semantic cluster. For negative samples with different intentions, the model is trained to separate them from positive samples with the target intention, thus avoiding overlap or confusion in the feature space. For hard-to-bear samples, the model is encouraged to learn more subtle semantic differences, such as distinguishing the visual semantic details of "just browsing" versus "about to confirm selection".

[0087] Through continuous iteration, the model gradually learns to group semantic combinations such as "gaze lingering + prominent virtual button + subsequent interface changes" into a certain intent area, while excluding unintended scenarios such as "gaze drifting + background as the main focus + no operation" into other areas.

[0088] As training progresses, several relatively stable regions gradually form in the feature space, each corresponding to one or a group of similar behavioral intentions. For example, one type of region represents "confirm / click" intentions; another type represents "swipe / page turn" intentions; and yet another type represents the "browse only / no operation" state, used to reduce the false trigger rate.

[0089] Simultaneously, the model automatically forms substructures that reflect user habits. For example, for users who frequently complete operations by lingering their gaze, their "confirmation" intent area will be more concentrated in the subspace with higher gaze feature weights; for users who prefer voice commands, the corresponding area will rely more on the voice-related feature dimensions in the semantic vector. Ultimately, this feature space possesses excellent characteristics of "compactness of similar types and dispersion of dissimilar types," which is the behavioral intent semantic separation feature space.

[0090] Based on the semantic separation feature space of behavioral intent, the visual semantic expression vector is decoded and mapped during the reasoning stage to obtain the target behavioral intent prediction result.

[0091] In this embodiment, after obtaining the visual semantic representation vector at the current moment, it is input into the pre-trained behavior intention decoding model. The model first maps the vector to the aforementioned behavior intention semantic separation feature space to obtain the position representation of that moment in the feature space. This position is not only related to factors such as the current gaze focus, scene objects, and task prompts, but is also affected by the semantic change trends at several historical moments.

[0092] During the training phase, the system forms corresponding clustering regions or reference distributions for each behavioral intent. During the inference phase, the model calculates the similarity or distance between the current vector and these intent regions, and determines the most suitable candidate behavioral intent by comparing the results.

[0093] For example, when the current vector falls near the "Confirm Selection" region and is significantly different from other intent regions, the model will output "Confirm Selection" as a candidate intent; if the vector is located in the "Browse Only / No Action" region and is far from the interaction-related region, the model will output "No Significant Intent" or "Continue Observing" status to suppress erroneous triggering.

[0094] To avoid intent flickering caused by single-frame fluctuations, the system also introduces continuity constraints in the time dimension. For example, an intent is considered a stable result only when the visual semantic expression vectors of multiple consecutive time steps fall near the same intent region; if there is only a brief deviation in one or two frames, it is filtered through a time smoothing strategy.

[0095] Meanwhile, the system imposes constraints based on the current interface context: if there are no clickable objects in the current scene, even if the feature space is somewhat close to the "click" area, it will not directly output the "click" intent, but will be more inclined to "browse"; if the user is already in navigation mode, the weight of intents related to the navigation path will be amplified, such as "progress along the path" or "view the next target".

[0096] After considering the integrated feature space matching results, temporal continuity judgment, and scene context constraints, the system finally outputs the target behavior intent prediction result, including the intent category (such as "confirm selection", "swipe to browse", "open menu", "keep observing") and its corresponding confidence level. This prediction result will be passed to the subsequent visualization interaction process to trigger interface response, update AR augmented content, or adjust the interaction method.

[0097] Furthermore, the formula for the behavioral intent decoding model is as follows: ; in, Indicates the index of semantic sample pairs The behavioral intent prediction score is used to characterize the confidence level of the AR smart glasses' target behavioral intent in the user's current scenario; denoted as a nonlinear semantic discriminant function, whose input is the weighted similarity result after the fusion of semantic subspaces, and whose output is a standardized predicted probability, used to generate a normalized intent classification result at the model output; M represents the number of dimensions of the contrastive semantic subspace; Indicates the first Weight coefficients for each semantic subspace; The semantic vector of the positive sample is represented in the th case. Embedded representation in semantic subspace; The semantic vector of the negative sample is represented in the th... Embedding representation in semantic subspace; cos(⋅) represents the cosine similarity measure between semantic vectors; This represents a small constant to prevent the denominator from being zero; This represents the contextual semantic consistency coefficient between sample pairs; Represents the hypercurvature tangent activation function; This represents the context consistency adjustment coefficient; The scaling factor represents the sensitivity of the semantic subspace to context consistency.

[0098] in, The calculation formula is as follows: ; in, and The visual semantic representation vector generated based on brain-like visual feature sequences represents the first... Semantic embeddings on graph structure nodes; k represents the number of graph structure nodes participating in the consensus computation; A nonlinear compression function representing the semantic association of nodes; This represents a small constant used to ensure computational stability.

[0099] The visualization interaction module is used to dynamically generate AR-enhanced presentation content based on intent prediction results and perform visualization interaction output on the display end of AR smart glasses. Specifically, the visualization interaction module performs high-density rendering output in the focal area of ​​the user's gaze and low-density rendering output in the non-focal area by real-time tracking of the user's gaze trajectory.

[0100] In this embodiment, the visualization interaction module includes a rendering management unit, a gaze tracking unit, an interaction control unit, and a resource scheduling unit.

[0101] When the behavior intent prediction module outputs behavior intents such as "select," "confirm," "browse," or "navigate," the rendering management unit first retrieves virtual UI elements or 3D augmented objects corresponding to the intent from a pre-set AR content library based on the semantic information of the current scene. These virtual objects include menu lists, directional arrows, label boxes, and virtual buttons. Based on the front-end spatial coordinate calibration results, these virtual objects are then anchored to designated positions in the real scene. To avoid obscuring key real-world targets, the rendering management unit performs target detection on the current scene before overlay rendering, setting occlusion priorities for areas such as faces, pedestrian walkways, and warning signs. Unnecessary augmented content is reduced or hidden in high-priority areas, thereby ensuring visual safety and information clarity.

[0102] The line-of-sight tracking unit collects the positions of the pupils of the user's eyes, the offset of the gaze point and the blinking rhythm in real time through the built-in eye movement sensor, and maps them to the display screen plane or three-dimensional field of view coordinate system to locate the current line-of-sight focus area of the user. The module divides the display area into multiple line-of-sight interest sub-areas, and determines the sub-area in which the user's gaze time exceeds a set threshold as the focus area. The rendering management unit accordingly enables a high-density rendering strategy in the focus area, including increasing the pixel resolution, increasing the texture details, increasing the refresh frame rate and enabling more light and shadow effects; at the same time, a low-density rendering strategy is adopted for the non-focus area, only the necessary outline and basic color are retained, so as to reduce the GPU load and power consumption, and to realize a hierarchical display mode of 'fine presentation where the line of sight reaches and simple presentation of the surrounding environment'.

[0103] In the aspect of interaction control, the interaction control unit fuses and analyzes the line-of-sight trajectory, the slight head movement and the voice instruction: for example, when the user stably stays the line of sight on a certain virtual button for a first threshold time, it is identified as a 'pre-selected' state, and feedback is given in the form of highlighted outline, slight enlargement or animated flickering; if the user has a slight nodding action or issues a 'confirmation' 'turn on' voice instruction in the pre-selected state, the interaction control unit triggers the corresponding function execution, such as expanding the menu, switching the page or starting the navigation. For sliding operations, the module can identify the continuous line-of-sight movement trajectory of the user, and map it to a list scrolling or content paging instruction, avoiding complex gestures.

[0104] The resource scheduling unit dynamically adjusts the rendering precision and update frequency according to the current battery power, chip temperature and task load: when the power is sufficient and the scene complexity is low, the module enables a high-quality rendering mode to enhance the sense of reality and immersion; when the power is low or the system load is high, the rendering frequency of non-critical areas is reduced, the special effect layer and the secondary UI animation are appropriately weakened, and the rendering quality of the focus area and the response delay of the feedback of the behavior intention are preferentially guaranteed, so that the user interaction is still smooth and usable.

[0105] In addition, in the embodiment, the visual interaction module also supports personalized configuration, and the system records information such as the line-of-sight stay time of the user habit, the frequently used interaction area, the preferred color style and the like in the long-term use process, and dynamically adjusts the focus determination threshold, the UI layout density and the prompt style. For example, for the user with slow line-of-sight switching, the system can appropriately prolong the focus judgment time and slow down the interface animation speed; and for the skilled user, the focus confirmation time is shortened and the pop-up prompt is reduced, so as to improve the overall operation efficiency. Through the above mechanism, the visual interaction module realizes fine control and adaptive optimization from the behavior intention result to the AR augmented content presentation and the human-computer interaction process.

[0106] The embodiment of the application further provides an implementation method of an AR intelligent glasses system based on brain-like visual processing, comprising: S1: Real-time acquisition of visual data of AR smart glasses through a multi-modal sensor; S2: Spatiotemporal feature coding and sparse pulse mapping of the visual data are performed using a brain-like visual algorithm based on a pulse neural network to generate a brain-like visual feature sequence; S3: Based on the brain-like visual feature sequence, a visual semantic expression vector is constructed using a graph structure-based semantic association modeling method; S4: Based on the semantic expression vector, a behavior intention prediction result is generated by performing behavior intention prediction using a behavior intention decoding model based on contrast learning; S5: AR augmented presentation content is dynamically generated based on the intention prediction result, and visual interaction output is performed on the display end of the AR smart glasses.

[0107] It can be understood that the contents in the above system embodiments are all applicable to the present method embodiment, the present method embodiment specifically implements the same functions as the above system embodiments, and achieves the same beneficial effects as the above system embodiments.

[0108] The present application embodiment further provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the above system when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0109] It can be understood that the contents in the above system embodiments are all applicable to the present device embodiment, the present device embodiment specifically implements the same functions as the above system embodiments, and achieves the same beneficial effects as the above system embodiments.

[0110] The present application embodiment further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the above system.

[0111] It can be understood that the contents in the above system embodiments are all applicable to the present storage medium embodiment, the present storage medium embodiment specifically implements the same functions as the above system embodiments, and achieves the same beneficial effects as the above system embodiments.

[0112] The present application embodiment further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the above system.

[0113] It can be understood that the contents in the above system embodiments are all applicable to the present program product embodiment, the present program product embodiment specifically implements the same functions as the above system embodiments, and achieves the same beneficial effects as the above system embodiments.

[0114] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include a high-speed random access memory and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory that is remotely disposed relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0115] The AR intelligent glasses system based on brain-like visual processing provided in the embodiments of the present application can comprehensively perceive the environment and interaction state of the user by fusing the visual data of the multi-modal sensors through the visual data acquisition module, realize real-time input of high-dimensional information, and lay a data foundation for subsequent feature extraction and semantic understanding. The brain-like visual algorithm based on the pulse neural network is used to process the visual data, which can be closer to the pulse information processing mode of human brain neurons, effectively realize sparse coding, dynamic response and low-power processing, and improve the feature extraction accuracy and real-time performance in a high dynamic environment. The semantic construction module uses a graph structure modeling method to model the time sequence relationship and spatial correlation in the brain-like visual feature sequence, extracts a semantic expression vector with more context meaning, and enables the system to have the ability to transition from "perception" to "understanding", and improves the accuracy of semantic reasoning. The behavior intention prediction module introduces a behavior intention decoding mechanism based on contrast learning, optimizes the embedding distance of positive and negative samples, improves the ability of the model to distinguish similar behaviors, and enhances the generalization recognition performance of behavior intention in different users and different scenes. The visual interaction module dynamically generates augmented reality content combined with the prediction result and directly displays it on the AR glasses terminal, realizes the closed-loop connection between the user behavior intention and the interactive output, enables the system to have an immersive and adaptive augmented display interaction experience, and significantly improves the naturalness and efficiency of human-computer interaction.

[0116] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0117] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than the figures, or combine certain steps or different steps.

[0118] The apparatus embodiments described above are merely illustrative, and units described as separate units can or can not be physically separate, i.e., can be located in one place, or can be distributed over multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purposes of the embodiments.

[0119] Those skilled in the art can understand that all or some of the steps in the method disclosed above, the functional modules / units in the system and the device can be implemented by software, firmware, hardware, or a suitable combination thereof.

[0120] The terms "first", "second", "third", "fourth" and the like in the description of the application and in the claims of the foregoing drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so construed can be interchanged, such that, for example, a description

[0121] The preferred embodiments of the application are described above with reference to the accompanying drawings, and are not intended to limit the scope of the application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and spirit of the application shall fall within the scope of the application.

Claims

1. An AR smart glasses system based on neuromorphic visual processing, characterized in that, include: The visual data acquisition module is used to acquire visual data from AR smart glasses in real time through a multimodal sensor; The visual feature extraction module is used to perform spatiotemporal feature encoding and sparse pulse mapping on the visual data using a brain-like visual algorithm based on a spiking neural network to generate a brain-like visual feature sequence. The semantic construction module is used to construct a visual semantic expression vector based on the brain-like visual feature sequence and a graph-based semantic association modeling method. The behavioral intent prediction module is used to predict behavioral intent based on the semantic expression vector using a contrastive learning-based behavioral intent decoding model, and generate behavioral intent prediction results. The visualization interaction module is used to dynamically generate AR-enhanced presentation content based on the intent prediction results, and to perform visualization interaction output on the display end of the AR smart glasses.

2. The system according to claim 1, characterized in that, The visual data includes visual images, inertial parameters, voice signals, and environmental status information.

3. The system according to claim 1, characterized in that, The construction process of the visual feature extraction module includes: A dynamic pulse firing threshold adjustment mechanism is used to perform temporal pulse encoding on the visual data to generate a temporal pulse event stream. Based on the aforementioned temporal pulse event stream, a brain-like vision algorithm based on a spiking neural network is used to perform multi-scale sparse pulse convolution calculations to extract deep brain-like vision features. The deep brain-like visual features are sparsely mapped and compressed to generate a sequence with brain-like visual features.

4. The system according to claim 3, characterized in that, The step of using a dynamic adjustment mechanism for pulse firing thresholds to perform temporal pulse encoding on the visual data to generate a temporal pulse event stream includes: Initialize the pulse firing threshold; Based on the local pixel gradient changes and dynamic contrast changes of the visual data, the pulse firing threshold is adaptively adjusted. Based on the adjusted pulse firing threshold, continuous visual data is converted into discrete pulse events; A time-series pulse event stream is generated based on the discrete pulse events, and the pulse firing threshold is continuously updated during the event stream generation process.

5. The system according to claim 3, characterized in that, The formula for the neuromorphic vision algorithm is as follows: ； in, This represents the brain-like visual feature response function of the output, i.e., in spatial location. With time The sparse pulse characteristic intensity; This represents the activation function of a spiking neuron. Indicates the first Scaled pulse convolution kernel weights; Indicates the input number of the first... Layer-based timing pulse event stream signal; Represents the pixel coordinates of a continuous visual image and time series The pixel intensity data below; This represents the temporal gradient change in pixel intensity of a visual image between adjacent frames. This represents the local spatial structure response of a visual image after processing with the Laplacian operator in the spatial dimension. Indicates the time response adjustment coefficient; represents the spatial sparsity adjustment coefficient; N represents the number of sparse pulse convolutional layers.

6. The system according to claim 1, characterized in that, The construction process of the semantic building module includes: Based on the aforementioned brain-like visual feature sequence, a graph structure semantic representation corresponding to nodes and edges is established; the nodes represent target feature points, objects, or regions in the visual scene; the edges represent spatial adjacency relationships or semantic dependencies between different nodes. Based on the graph structure semantic representation, a graph structure-based semantic association modeling method is adopted to perform graph convolution propagation and weighted aggregation on the semantic correlation between nodes, forming a multi-level semantic embedding representation; Global context normalization and attention weight adjustment are performed on the semantic embedding representation to generate a visual semantic expression vector; the visual semantic expression vector is used to characterize the semantic relationships and structural hierarchy between different objects in the visual scene.

7. The system according to claim 1, characterized in that, The construction process of the behavioral intent prediction module includes: Based on the visual semantic expression vector, a semantic comparison sample pair of positive sample semantic vector and negative sample semantic vector is constructed; the positive sample semantic vector is used to characterize the semantic expression distribution corresponding to the target behavior intention, and the negative sample semantic vector is used to characterize the semantic interference distribution corresponding to the non-target behavior intention. A behavior intent decoding model based on contrastive learning is used to train the semantic distinction between the positive sample semantic vector and the negative sample semantic vector to obtain the behavior intent semantic separation feature space. Based on the behavioral intent semantic separation feature space, the visual semantic expression vector is decoded and mapped during the reasoning stage to obtain the target behavioral intent prediction result.

8. The system according to claim 7, characterized in that, The formula for the behavioral intent decoding model is as follows: ； in, Indicates the index of semantic sample pairs The behavioral intention prediction score is as follows; represents the nonlinear semantic discriminant function; M represents the dimension of the contrastive semantic subspace; Indicates the first Weight coefficients for each semantic subspace; The semantic vector of the positive sample is represented in the th case. Embedded representation in semantic subspace; The negative sample semantic vector represents the first... Embedding representation in semantic subspace; cos(⋅) represents the cosine similarity measure between semantic vectors; This represents a small constant to prevent the denominator from being zero; This represents the contextual semantic consistency coefficient between sample pairs; Represents the hypercurvature tangent activation function; This represents the context consistency adjustment coefficient; The scaling factor represents the sensitivity of the semantic subspace to context consistency.

9. The system according to claim 1, characterized in that, The visualization interaction module performs high-density rendering output in the focal area and low-density rendering output in the non-focal area by tracking the user's gaze trajectory in real time.

Images