Multi-modal interaction and image processing method of AI glasses, electronic equipment and storage medium

Abstract

The invention provides a multi-modal interaction and image processing method for AI glasses, electronic equipment and a storage medium, and the method comprises the steps: carrying out the preprocessing of an original image collected by the AI glasses in real time based on a space-time attention mechanism, and obtaining a preprocessed image; performing voice recognition and semantic analysis on a voice instruction input by a user through the AI glasses to obtain a text instruction; performing keyword matching on the text instruction and a preset image processing instruction template library, and determining a corresponding image processing strategy; if the image processing strategy is dynamic tracking, predicting the position of a target in the preprocessed image in the next frame by using a Kalman filtering algorithm, and searching and positioning the target around the predicted position by using a twin network; and the appearance features of the target are learned and updated in real time based on the twin network, stable tracking of the target is realized, and a tracking target image sequence is output. According to the method, the defects that the subsequent appearance position of the target cannot be predicted and the target cannot be stably tracked at present are overcome.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a multimodal interaction and image processing method of AI glasses, an electronic device and a storage medium. Background Art

[0002] AI glasses are integrated into daily life and many professional fields. In daily life, users hope to use it to easily identify objects of interest, track moving signs, and optimize captured images when socializing.

[0003] However, traditional image tracking technology has its flaws. It often focuses on a single mode or relies only on static features of the image, without considering the dynamic changes of the actual scene and the advantages of multi-modal fusion. Current image tracking is prone to losing the target when the target suddenly accelerates, turns, or is blocked. It is impossible to predict the target's subsequent position, so it is impossible to track the target stably. Summary of the invention

[0004] The main purpose of the present invention is to provide a multimodal interaction and image processing method, electronic device and storage medium for AI glasses, aiming to overcome the current defects of being unable to predict the subsequent appearance position of a target and being unable to stably track the target.

[0005] To achieve the above object, the present invention provides a multimodal interaction and image processing method of AI glasses, comprising the following steps: Based on the spatiotemporal attention mechanism, the original images collected by the AI ​​glasses in real time are preprocessed to obtain preprocessed images; the voice commands input by the user through the AI ​​glasses are subjected to speech recognition and semantic analysis to obtain text commands; Perform keyword matching between the text instruction and a preset image processing instruction template library to determine a corresponding image processing strategy; If the image processing strategy is dynamic tracking, the Kalman filter algorithm is used to predict the position of the target in the preprocessed image in the next frame, and the twin network is used to search around the predicted position to accurately locate the target; Based on the twin network, the appearance features of the target are updated in real time through real-time learning, so as to achieve stable tracking of the target and output a tracking target image sequence for display to the user on the AI ​​glasses display.

[0006] Furthermore, the Kalman filter algorithm dynamically adjusts the noise covariance matrix of the prediction model in the Kalman filter algorithm according to the historical acceleration and angular velocity of the target motion to improve the prediction accuracy.

[0007] Furthermore, after achieving stable tracking of the target and outputting a tracking target image sequence, it also includes: Performing detail enhancement on the tracking target image sequence based on the generator to obtain an enhanced image; Based on the discriminator determining the authenticity of the enhanced image, the generator and the discriminator are anti-iteratively optimized to improve the clarity and texture richness of the enhanced image for display to the user on the AI ​​glasses display.

[0008] Furthermore, the original image collected by the AI ​​glasses in real time is preprocessed based on the spatiotemporal attention mechanism to obtain a preprocessed image, including: Use a three-dimensional convolutional neural network to convolve the original image in the spatial dimension, extract local detail features and global semantic features in the image, and obtain a spatial feature map; In the time dimension, for the original image, based on the long short-term memory network model, the temporal correlation between adjacent frames is learned, the dynamic change characteristics of the image sequence over time are captured, and the temporal feature sequence is output; Based on the attention mechanism, the spatial feature map is fused with the temporal feature sequence to obtain a spatiotemporal coherent feature map, and the preprocessed image is obtained through normalization.

[0009] Furthermore, based on the twin network, the appearance features of the target are updated in real time to achieve stable tracking of the target and output a tracking target image sequence, including: The image area of ​​the target tracked in the current frame is cropped and input into the twin network. The convolutional neural network based on the residual structure is used to extract its features to obtain the template feature vector containing the key features of the target. For the entire image of the current frame, a densely connected convolutional network is used to extract features to obtain a search feature map containing potential target areas; Calculate the similarity between each potential target area in the search feature map and the template feature vector, and compare it with the similarity threshold; construct a dynamic environment assessment model to dynamically adjust the similarity threshold in real time according to the rate of change of light intensity in the current image, the moving speed of background objects and the acceleration of target motion; The area with the greatest similarity and a similarity greater than the similarity threshold is obtained, and its corresponding feature information is fused with the template feature vector by assigning different weights so that the template feature vector is continuously updated to achieve stable tracking of the target and output the tracking target image sequence.

[0010] Furthermore, the method further comprises: Collecting multimodal interaction information between the user and the AI ​​glasses within a specified time period; wherein the multimodal interaction information includes voice information sent by the user, touch instructions of the user to the AI ​​glasses, and image information collected by the user controlling the AI ​​glasses; Sending the multimodal interaction information to a server; The server generates permission information based on the multimodal interaction information and sends it to the AI ​​glasses to configure the usage permissions of the AI ​​glasses.

[0011] Further, generating permission information based on the multimodal interaction information includes: Performing speech recognition on the speech information to identify keywords; encoding the keywords into numbers to obtain key numbers; The touch instruction is identified to obtain a touch pattern; along the touch direction of the touch pattern, the digital characters in the key number are added to the touch pattern one by one in sequence to obtain a digital pattern; Encoding the image information into a data stream character string, and discretely dividing the data stream character string into a plurality of adjacent areas of the digital pattern; In each of the adjacent areas, characters that satisfy a preset association relationship with the digital characters on the digital graphic are obtained as target characters, and the authority information is obtained by combining them.

[0012] The present invention also provides a multimodal interaction and image processing device for AI glasses, comprising: A processing unit is used to pre-process the original image collected by the AI ​​glasses in real time based on the spatiotemporal attention mechanism to obtain a pre-processed image; perform speech recognition and semantic analysis on the voice command input by the user through the AI ​​glasses to obtain a text command; A matching unit, used for matching the text instruction with a preset image processing instruction template library by keywords to determine a corresponding image processing strategy; A positioning unit, which is used to predict the position of the target in the preprocessed image in the next frame by using a Kalman filter algorithm if the image processing strategy is dynamic tracking, and to search around the predicted position by using a twin network to accurately locate the target; A tracking unit is used to learn and update the appearance features of the target in real time based on the twin network, to achieve stable tracking of the target and output a tracking target image sequence for display to the user on the AI ​​glasses display.

[0013] The present invention also provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the above methods when executing the computer program.

[0014] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned methods are implemented.

[0015] The multimodal interaction and image processing method, electronic device and storage medium of AI glasses provided by the present invention include: preprocessing the original image collected by AI glasses in real time based on the spatiotemporal attention mechanism to obtain a preprocessed image; performing voice recognition and semantic parsing on the voice instructions input by the user through the AI ​​glasses to obtain a text instruction; matching the text instruction with a preset image processing instruction template library for keywords to determine the corresponding image processing strategy; if the image processing strategy is dynamic tracking, the Kalman filter algorithm is used to predict the position of the target in the preprocessed image in the next frame, and the twin network is used to search around the predicted position to accurately locate the target; based on the twin network, the appearance features of the target are updated in real time to achieve stable tracking of the target and output the tracking target image sequence for display to the user on the AI ​​glasses display. In the present invention, the position of the target in the image in the next frame can be predicted, and then the target can be accurately located, and the appearance features of the target can be updated in real time to achieve stable tracking of the target. It overcomes the current defects of being unable to predict the subsequent appearance position of the target and being unable to stably track the target. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 1 is a schematic diagram of the steps of a multimodal interaction and image processing method of AI glasses in one embodiment of the present invention; Figure 2 is a structural block diagram of a multimodal interaction and image processing device of AI glasses in one embodiment of the present invention; Figure 3 It is a schematic block diagram of the structure of an electronic device according to an embodiment of the present invention.

[0017] The implementation, functional features and advantages of the present invention will be further described in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0019] Reference Figure 1 In one embodiment of the present invention, a multimodal interaction and image processing method of AI glasses is provided, comprising the following steps: Step S1, preprocessing the original image collected by the AI ​​glasses in real time based on the spatiotemporal attention mechanism to obtain a preprocessed image; performing speech recognition and semantic analysis on the voice command input by the user through the AI ​​glasses to obtain a text command; Step S2, matching the text instruction with a preset image processing instruction template library by keywords to determine a corresponding image processing strategy; Step S3, if the image processing strategy is dynamic tracking, the Kalman filter algorithm is used to predict the position of the target in the preprocessed image in the next frame, and the twin network is used to search around the predicted position to accurately locate the target; Step S4, based on the real-time learning of the twin network, the appearance features of the target are updated to achieve stable tracking of the target and output a tracking target image sequence for display to the user on the AI ​​glasses display.

[0020] In this embodiment, as described in step S1 above, the spatiotemporal attention mechanism is intended to perform targeted processing on the original images collected by the AI ​​glasses in real time from two dimensions: time and space. In the spatial dimension, it can focus on the importance of different areas in the image, such as highlighting the parts of the image that may contain key targets or subjects, weakening those relatively unimportant background areas, etc.; in the temporal dimension, considering that the acquisition of images by AI glasses is a continuous dynamic process, the mechanism can utilize the correlation between images at different times, and filter out noise and interference information in the image by analyzing the changes between the previous and next frame images, so as to obtain a clearer pre-processed image with more prominent key information, laying a good foundation for subsequent image processing operations.

[0021] After the user inputs a voice command through the AI ​​glasses, voice recognition is required first. This step will convert the voice signal into a text form that the computer can understand and process. Then semantic parsing is performed, which is to deeply analyze the converted text content, understand what the user really wants to express, and finally get accurate text instructions. For example, if the user says "track the flying bird in the picture", after this series of processing, a text instruction containing key semantic information such as "tracking" and "flying bird" will be generated for further operation in subsequent steps.

[0022] As described in step S2 above, this step is to match the text instructions obtained in step S1 with the preset image processing instruction template library for keywords. A large number of standard templates corresponding to different image processing functions and strategies are pre-stored in the image processing instruction template library, and each template is associated with a specific set of keywords. For example, there is a template corresponding to "dynamic tracking", and its keywords may include "tracking", "following", "target movement", etc.; there is a template corresponding to "image enhancement", and the relevant keywords may be "clear", "sharpening", "color optimization", etc. By extracting the keywords in the text instructions and matching and searching with the keywords in the template library, the corresponding image processing strategy can be determined, so as to clarify what specific image processing method should be adopted next to meet user needs.

[0023] As described in step S3 above, when the image processing strategy is determined to be dynamic tracking, the Kalman filter algorithm is first used to predict the position of the target in the preprocessed image in the next frame. The Kalman filter algorithm has the ability to optimally estimate the state of a dynamic system. In this scenario, it is based on the state information such as the motion trajectory and speed of the target in the previous frames of images, combined with a certain system model (for example, considering the common motion model assumptions such as uniform speed and uniform acceleration of the target), to predict the position where the target is likely to appear in the next frame of the image. It can effectively filter out noise interference in the measurement process, improve the accuracy and reliability of position prediction, and provide a rough search range for more accurate positioning of the target in the future.

[0024] The twin network is used to further search for the target around the position predicted by the Kalman filter algorithm. The twin network is a special neural network architecture, which is usually composed of two sub-networks with the same structure. It can extract features and compare and analyze different input image data (such as the image area around the predicted position in the current frame image and the reference image features of the target, etc.), and accurately determine the specific position of the target in the current frame image by calculating indicators such as the similarity between the two. Even if the target has appearance changes or is partially blocked in the image, it can still be better identified and located.

[0025] As described in step S4 above, updating the appearance features of the target based on the twin network in real time learning is a key point in the entire dynamic tracking process. As time goes by and the target is presented in different environments, angles, etc., its appearance may change, such as color changes and posture changes caused by lighting. The twin network continuously learns the newly emerging target image features and updates its cognition of the target's appearance features in real time, so that it can continuously and stably track the target, and will not easily lose the tracking clues even if the target's external appearance changes a lot. Finally, the tracking target image sequence is output, which can be displayed to the user on the AI ​​glasses display according to the appropriate frame rate and other requirements, so that the user can clearly observe the dynamic situation of the target of interest in real time.

[0026] In one embodiment, the Kalman filter algorithm dynamically adjusts the noise covariance matrix of the prediction model in the Kalman filter algorithm according to the historical acceleration and angular velocity of the target motion to improve the prediction accuracy.

[0027] In this embodiment, the Kalman filter algorithm is essentially an optimal estimation method based on a linear system state space model. It continuously corrects the estimation of the system state by predicting the system state and combining it with actual observations, thereby accurately grasping the state of the dynamic system. In the target tracking scenario of AI glasses, the system state usually contains key information such as the position and speed of the target. Its working process is generally divided into a prediction step and an update step. The prediction step is based on the state estimation of the previous moment and the state transfer equation of the system to infer the state at the current moment. The update step uses the observations obtained at the current moment (such as target-related information detected in the image) to correct the predicted state.

[0028] In the Kalman filter algorithm, the noise covariance matrix plays a vital role. It mainly involves two aspects of noise description: one is the process noise covariance matrix, which reflects the influence of various uncertain factors in the state transfer process, such as the uncertainty of the change of motion state caused by the unmodeled interference force in the actual motion process of the target and the irregularity of its own motion; the other is the measurement noise covariance matrix, which reflects the uncertainty of the observation value caused by the sensor accuracy limitation (in AI glasses, such as the resolution of the image acquisition device, the noise in the acquisition process, etc.). Reasonable setting and adjustment of the values ​​of these noise covariance matrices can enable the Kalman filter algorithm to better balance the weights of the predicted value and the observed value, thereby improving the accuracy of the target state estimation.

[0029] The historical acceleration of the target motion can reflect the speed change trend and force conditions of the target during the motion process. For example, if the target shows a gradual increase in acceleration over the past period of time, it means that the target may be accelerating, and this acceleration trend may continue until the next moment. At this time, by analyzing the magnitude and direction of change of the historical acceleration, the process noise covariance matrix can be adjusted accordingly. Because accelerated motion means that there is greater uncertainty in the motion state of the target, the prediction model based on assumptions such as conventional uniform motion may not be very accurate. By appropriately increasing the value of the corresponding dimension in the process noise covariance matrix (such as the dimension related to speed and position), the Kalman filter algorithm can consider the uncertainty caused by this dynamic change more when predicting the state of the target position at the next moment, so as not to rely too much on the relatively stable prediction model before, so as to infer the target position more in line with the actual situation and improve the prediction accuracy.

[0030] For targets with rotation characteristics or targets whose posture changes during movement, historical angular velocity is particularly important. Angular velocity reflects the speed of the target's rotation and the direction of the change. When a large historical angular velocity change is detected in the target, it means that the target has obvious dynamic changes in posture, rotation angle, etc., which will also bring greater uncertainty to the target's position prediction. Based on this, the noise covariance matrix can also be dynamically adjusted. For example, the value of the process noise covariance matrix can be appropriately increased for the dimensions related to the target angle and rotation, so that the Kalman filter algorithm can fully consider the position uncertainty caused by rotation and posture changes when predicting the target position, so that the prediction result can more accurately reflect the actual position of the target at the next moment, avoiding large prediction errors caused by ignoring these dynamic change factors.

[0031] By dynamically adjusting the noise covariance matrix of the prediction model in the Kalman filter algorithm according to the historical acceleration and angular velocity of the target's motion, the Kalman filter algorithm can be more adaptive to the state changes of the target in complex and changeable motion scenes. It no longer uses a fixed noise covariance setting, but closely combines the actual motion characteristics of the target to optimize its own prediction mechanism in real time. In this way, in the dynamic tracking application of AI glasses, the position of the target in the next frame of the image can be more accurately predicted, providing a more reliable foundation for the subsequent use of twin networks and other methods to further accurately locate the target, ultimately improving the accuracy and stability of the entire AI glasses in tracking the target, and providing users with a better visual experience of target tracking.

[0032] In one embodiment, after achieving stable tracking of the target and outputting a tracking target image sequence, the method further includes: Performing detail enhancement on the tracking target image sequence based on the generator to obtain an enhanced image; Based on the discriminator determining the authenticity of the enhanced image, the generator and the discriminator are anti-iteratively optimized to improve the clarity and texture richness of the enhanced image for display to the user on the AI ​​glasses display.

[0033] In this embodiment, the generator is usually a part of the generative adversarial network (GAN) architecture based on deep learning. Its core function is to generate new images based on the input tracking target image sequence, with the purpose of enhancing the details of the original image. For example, it can learn the texture features, edge features, color levels and other detailed information of different objects in the image, and then transform and reconstruct the features through its own network structure (often a complex architecture consisting of multiple convolutional layers, activation functions and other components), and refine the target image area that may be relatively fuzzy and not rich in texture. For example, for a distant object in the tracking target image sequence, its original outline is not clear enough and the surface texture is difficult to distinguish. The generator can use the general rules of the object's appearance features that it has learned to add clearer contour lines to it and enrich its surface texture, so that the object presents more details in the image, thereby improving the visual effect of the entire tracking target image sequence.

[0034] The quality of the tracking target image sequence displayed to the user by AI glasses directly affects the user's observation experience. Detail enhancement through the generator can enable users to see the various features of the target more clearly, especially in some complex environments or when the target itself is difficult to distinguish. Clear and detailed images help users better understand the target's state, shape and other key information, thereby meeting the user's demand for high-quality image display and making the functions of AI glasses more practical.

[0035] The above-mentioned discriminator is also an important part of the generative adversarial network (GAN) architecture. Its main task is to receive the enhanced images output by the generator and judge the authenticity of these images. Specifically, the discriminator measures the similarity between the received enhanced image and the real image through its own knowledge of the statistical characteristics and distribution laws of the real image (these are learned through a large amount of real image data during the training phase). For example, it will judge from multiple dimensions such as whether the color distribution of the image is reasonable, whether the texture conforms to the laws of nature, and whether the structure of the object conforms to common sense. The discriminator is generally a neural network composed of multiple convolutional layers, fully connected layers, etc., and its output is usually a numerical value representing the probability of the authenticity of the image. For example, an output value close to 1 means that the discriminator believes that the image is likely to be a real image, and close to 0 means that it is considered a fake image.

[0036] The discriminator exists to provide feedback to the generator, guiding it to generate enhanced images that are more in line with real visual perception. In the entire technical solution, only when the generated enhanced image can deceive the discriminator, that is, when it is difficult for the discriminator to distinguish the difference between it and the real image, it means that the image generated by the generator has reached a high quality standard in terms of details and overall visual effects, and meets the requirements of improving image clarity and texture richness. Therefore, the discriminator's determination of the authenticity of the enhanced image plays a key role in controlling the quality of the image finally displayed to the user.

[0037] The generator and the discriminator are in a mutually antagonistic and collaborative relationship in the whole scheme. In the initial stage, the enhanced image generated by the generator may have many deficiencies in details and authenticity, and the discriminator can easily identify it as a fake image. Then, based on the discriminator's judgment result, the generator will adjust its own network parameters (for example, through the back-propagation algorithm, update the weights of the convolution layer and other parameters according to the error information fed back by the discriminator), and try to generate a more realistic and detailed enhanced image. The discriminator will also continue to optimize its ability to distinguish the authenticity of the image and improve the accuracy of the judgment based on the newly received enhanced image output by the generator and the newly supplemented real image data (if any). This process will be repeated. In multiple iterations, the generator will continuously improve its ability to generate high-quality enhanced images, and the discriminator will also continuously improve its ability to identify the authenticity of the image. The two are like a game, gradually reaching a relatively balanced state.

[0038] Through such an anti-iterative optimization process, the final generator can output enhanced images with significantly improved clarity and texture richness. When these images are displayed to users on the AI ​​glasses display, they not only look more realistic, but also allow users to more clearly observe the subtle details of the target, such as the texture patterns and tiny structural features on the surface of the target object, which greatly optimizes the visual experience of users when using AI glasses to observe and track targets, making the image processing function of the entire AI glasses more advantageous and better meeting users' expectations for high-quality image display.

[0039] In one embodiment, the original image collected by the AI ​​glasses in real time is preprocessed based on the spatiotemporal attention mechanism to obtain a preprocessed image, including: Use a three-dimensional convolutional neural network to convolve the original image in the spatial dimension, extract local detail features and global semantic features in the image, and obtain a spatial feature map; In the time dimension, for the original image, based on the long short-term memory network model, the temporal correlation between adjacent frames is learned, the dynamic change characteristics of the image sequence over time are captured, and the temporal feature sequence is output; Based on the attention mechanism, the spatial feature map is fused with the temporal feature sequence to obtain a spatiotemporal coherent feature map, and the preprocessed image is obtained through normalization.

[0040] In this embodiment, the traditional two-dimensional convolutional neural network is mainly used to process a single image, focusing on the spatial information mining of the image, such as extracting features such as texture, edge, shape, etc. in the image. On this basis, the three-dimensional convolutional neural network adds the consideration of the time dimension, but in this step, the focus is on exerting its convolution ability in the spatial dimension. It slides on the original image to perform convolution operations by setting a convolution kernel of a specific size (such as a 3×3×3 convolution kernel, the first two dimensions correspond to the plane space size of the image, and the third dimension can be used to process different channels of the image, etc.). This convolution operation can extract features from local areas of the image. For example, for a local area where an object is located in the image, the convolution kernel can capture its local texture details, such as whether the surface of the object is smooth or rough. At the same time, as the convolution kernel continues to slide on the entire image plane, it can also gradually integrate the information of each local area, and then extract the global semantic features of the image, that is, understand the content expressed by the image as a whole, such as judging whether the image presents a street scene or an indoor scene. In this way, the feature information of the original image in the spatial dimension can be fully and deeply mined to obtain a spatial feature map.

[0041] The spatial feature map carries the key feature information extracted from the original image after spatial dimension convolution processing. It abstracts and summarizes the complex pixel information in the original image at the feature level. These features are very important for subsequent accurate recognition of targets in the image, distinguishing different objects, and understanding image scenes.

[0042] The original image collected by AI glasses is a continuous image sequence, and there is a temporal correlation between adjacent frames. The long short-term memory network model is very suitable for processing this kind of data with time series characteristics. Through a special gate structure (including input gate, forget gate, output gate, etc.), it can effectively solve the gradient vanishing or gradient explosion problems that are prone to occur in traditional neural networks when processing long sequence data, so that it can better learn the long-distance temporal dependencies in the image sequence. For example, for a moving target, its position and posture will change in the image frames at different times. LSTM can remember the target's motion state change trend based on the target's performance in multiple previous frames, such as whether the target is moving at a constant speed or accelerating, and then capture the dynamic change characteristics of the image sequence over time.

[0043] The output time feature sequence records the key change information of the image sequence in the time dimension, which reflects the evolution of the target and the entire image scene over time. For example, in a dynamic tracking scenario, the time feature sequence can help subsequent links better predict the state of the target at the next moment, understand the trend of the target's motion trajectory, etc., and provide an important basis in the time dimension for achieving stable and accurate tracking of the target, so that the entire image processing process is not limited to the spatial information processing of a single image, but is combined with time dynamic changes for comprehensive consideration.

[0044] The attention mechanism can simulate the difference in the degree of attention paid to different information in the human visual system. When fusing the spatial feature map with the temporal feature sequence, it can assign different weights according to the importance of each of them and perform targeted fusion. For example, for a target that is moving rapidly, the information in the temporal feature sequence may be relatively more important, because the position change of the target is more significant in a short period of time. At this time, the attention mechanism will assign a higher weight to the temporal feature sequence, making it occupy a more dominant position in the fused spatiotemporal coherent feature map; while for the relatively static background area, the information about its texture, layout and other spatial features in the spatial feature map is more worthy of attention, and will be given appropriate weights for fusion accordingly. In this way, the key features of the two dimensions of space and time can be organically combined to form a spatiotemporal coherent feature map, so that the feature map contains both the rich details and semantic information of the image in space and the dynamic change characteristics in the temporal dimension, which more comprehensively reflects the actual situation of the original image.

[0045] After obtaining the spatiotemporal coherent feature map, normalization is performed to normalize the eigenvalues ​​in the feature map so that it meets a certain numerical range requirement, which is convenient for subsequent calculation and processing. It also helps to improve the stability and convergence speed of the model.

[0046] In one embodiment, the appearance features of the target are updated based on the real-time learning of the twin network to achieve stable tracking of the target and output a tracking target image sequence, including: The image area of ​​the target tracked in the current frame is cropped and input into the twin network. The convolutional neural network based on the residual structure is used to extract its features to obtain the template feature vector containing the key features of the target. For the entire image of the current frame, a densely connected convolutional network is used to extract features to obtain a search feature map containing potential target areas; Calculate the similarity between each potential target area in the search feature map and the template feature vector, and compare it with the similarity threshold; construct a dynamic environment assessment model to dynamically adjust the similarity threshold in real time according to the rate of change of light intensity in the current image, the moving speed of background objects and the acceleration of target motion; The area with the greatest similarity and a similarity greater than the similarity threshold is obtained, and its corresponding feature information is fused with the template feature vector by assigning different weights so that the template feature vector is continuously updated to achieve stable tracking of the target and output the tracking target image sequence.

[0047] In this embodiment, in the process of tracking the target, after each current frame tracks the target, the corresponding image area is cropped out, so that the target itself can be focused on, and a large amount of irrelevant background information interference is eliminated, so that the subsequent feature extraction operation can more intensively mine the key features of the target itself. For example, in a complex scene, if no cropping is performed, the entire image may contain many background objects, complex light and shadow changes, and other factors. After the target area is cropped out, the shape, texture, color, and other information closely related to target identification and tracking can be processed specifically, thereby improving the efficiency and accuracy of feature extraction.

[0048] Convolutional neural networks with residual structures have significant advantages in feature extraction. In the process of deepening the number of network layers to obtain more abstract and richer features, traditional convolutional neural networks are prone to problems such as gradient vanishing or gradient explosion, which makes the network difficult to train and reduces performance. The residual structure introduces shortcut connections (i.e., shortcut connections that can directly pass input information to the following layers), making it easier for the network to learn the residual mapping between input and output, effectively alleviating the above problems. When such a network is used to extract features from the cropped target image area, detailed features and abstract features of the target can be extracted from different levels, such as extracting basic features such as edges and textures of the target from shallow network layers, and extracting more semantic overall morphology, category-related and other key features of the target from deep network layers. Finally, these features are integrated to form a template feature vector containing the key features of the target, providing accurate target feature representation for subsequent operations such as similarity comparison.

[0049] The core feature of the densely connected convolutional network (DenseNet) is that each layer is directly connected to all previous layers, making the information transfer in the network more sufficient and the features can be reused to the maximum extent. When processing the entire image of the current frame, since the image may contain multiple potential target areas (for example, the target may not be easy to directly determine its exact location due to occlusion, interference from similar objects, etc.), this network structure can fully mine various feature information in the image, without missing any local area features that may contain the target. It can extract features from images from different angles and different local ranges, and combine these features to form a search feature map containing potential target areas. The search feature map is actually a representation of the entire image at the feature level, providing rich candidate area information for subsequent searches for areas that match the target template feature vector.

[0050] The search feature map carries rich feature information of the current frame image, covering the feature clues of possible targets at various locations and local areas in the image. Through subsequent operations such as similarity calculation with the template feature vector, the area most similar to the target can be located in this search feature map, and then the actual position of the target in the current frame image can be determined. Even if the appearance of the target, the environment it is in, etc., changes, effective search and positioning can be performed based on the feature information in the search feature map.

[0051] The purpose of calculating the similarity between each potential target area in the search feature map and the template feature vector is to find the area that is most likely to be the target in the current frame image. Commonly used similarity calculation methods, such as cosine similarity and Euclidean distance, can measure the similarity between two feature vectors. By comparing the feature vector corresponding to each potential target area with the template feature vector, the corresponding similarity value is obtained, and then compared with the similarity threshold. If the similarity is greater than the threshold, it is considered that the area may be the target area; otherwise, it is highly likely not the target area. Such a comparison operation can screen out target candidate areas that meet the requirements, narrow the scope of target positioning, and improve the accuracy of target tracking.

[0052] In actual tracking scenarios, environmental factors are complex and changeable. For example, when the rate of change of light intensity is large, the appearance color, light and dark contrast and other characteristics of the target will change, which will cause the areas that were originally judged similar according to the fixed threshold to become dissimilar; when the background object moves faster, it is easy to produce interference areas similar to the target, which will also affect the similarity judgment; the acceleration change of the target movement will also make the appearance characteristics of the target itself show different degrees of change at different times. The purpose of building a dynamic environmental assessment model is to consider the impact of these environmental factors on the appearance characteristics and similarity judgment of the target in real time. According to indicators such as the rate of change of light intensity, the speed of background object movement and the acceleration of target movement, the similarity threshold is dynamically adjusted. For example, when the light intensity changes drastically, the similarity threshold is appropriately relaxed to avoid the target characteristics being unable to be recognized due to slight changes in light changes; when the background object moves fast and there are more interferences, the similarity threshold may be appropriately increased to enhance the strictness of screening and ensure that the real target area is found, so that the similarity judgment is more in line with the actual changing environment and improve the stability and accuracy of target tracking.

[0053] Obtaining the area with the greatest similarity greater than the similarity threshold means finding the area in the current frame image that best matches the target features, that is, the most likely location of the target. The feature information corresponding to the area is fused with the template feature vector, taking into account that the appearance features of the target will continue to change during movement, such as changes in angle, appearance changes caused by partial occlusion, etc. Fusion is performed by assigning different weights, for example, giving a higher weight to the more reliable part of the newly acquired regional features that is related to the stable features of the target, and giving a lower weight to the part that may be affected by noise and temporary interference, so as to update the template feature vector so that it can keep up with the changes in the target appearance features in a timely manner and always maintain an accurate description of the target features, thereby achieving stable tracking of the target.

[0054] The final output is a tracking target image sequence, so that the user can intuitively see the state of the target at different times through the AI ​​glasses. The continuous image sequence can clearly show the target's motion trajectory, appearance changes, etc., meeting the user's needs for dynamic observation of the target. It also reflects the integrity and practicality of the entire AI glasses in terms of target tracking function, providing users with a good user experience.

[0055] In one embodiment, the method further comprises: Collecting multimodal interaction information between the user and the AI ​​glasses within a specified time period; wherein the multimodal interaction information includes voice information sent by the user, touch instructions of the user to the AI ​​glasses, and image information collected by the user controlling the AI ​​glasses; Sending the multimodal interaction information to a server; The server generates permission information based on the multimodal interaction information and sends it to the AI ​​glasses to configure the usage permissions of the AI ​​glasses.

[0056] In this embodiment, the multimodal interaction information covers data on the user's interaction with the AI ​​glasses in multiple dimensions, including voice information emitted by the user, touch instructions given by the user to the AI ​​glasses, and image information collected by the user controlling the AI ​​glasses.

[0057] Voice information reflects the intentions and needs that users convey to AI glasses through voice, such as issuing voice commands such as "set permissions" and "enlarge objects in the picture". It reflects the specific functional operations that users expect AI glasses to perform. It is an intuitive and convenient way of interaction. Collecting these voice information helps to fully understand the various needs expressed by users during use.

[0058] Touch commands are commands generated by users’ touch operations on AI glasses, and are also important interactive data. For example, by touching a specific area, you can switch function interfaces, adjust display brightness, and other operations. These touch commands can reflect the user’s habits in manual control and the frequency of use of different functions, and are of great value for analyzing user behavior and personalized needs.

[0059] The image information collected by the user-controlled AI glasses records the visual content that the user is concerned about, which may be specific scenes, specific objects, etc. This can reflect the user's interests. For example, if the user frequently collects images of natural scenery, it means that he or she may have a higher demand for functions related to such images (such as image enhancement, sharing, etc.).

[0060] By collecting these multimodal interaction information within a specified time period, we can fully understand the interaction between users and AI glasses from multiple angles, and provide a rich data foundation for further analyzing user behavior, configuring corresponding permissions, and other operations based on this information.

[0061] The purpose of sending the collected multimodal interaction information to the server is to process the data with the help of the server's powerful computing, storage and analysis capabilities. Due to the limitations of hardware resources (such as processor performance, storage space, etc.), AI glasses themselves are difficult to independently complete operations such as in-depth analysis and permission configuration of large amounts of complex interaction information. Through network communication technology, the data is stably transmitted to the server to ensure that the server can receive complete and accurate multimodal interaction information, so that it can be used as input data for the subsequent generation of permission information, thereby better coordinating the management and allocation of the use rights of AI glasses.

[0062] After receiving the multimodal interaction information, the server will use the corresponding data analysis algorithm and permission configuration strategy to generate permission information. For example, based on the functions corresponding to the voice commands frequently used by users, the server can determine the core functional modules commonly used by users, and grant higher usage permissions to these core functions to ensure that users can use them smoothly and continuously; and for those functions that are rarely touched by users, are relatively complex, or have certain security risks (such as advanced settings involving privacy data access, etc.), their permissions can be appropriately restricted based on comprehensive judgments such as their frequency of use and user proficiency.

[0063] Sending the generated permission information back to the AI ​​glasses to configure their usage permissions has several important meanings. On the one hand, it can improve the user experience, by ensuring the high permissions and smooth use of the user's commonly used functions, so that users can operate the AI ​​glasses more easily and reduce the operation obstruction caused by insufficient permissions; on the other hand, from the perspective of security and privacy protection, it can avoid data leakage and infringement of the rights of others caused by misoperation or inappropriate use of users, standardize the scope and method of use of AI glasses, and ensure the legality, safety and rationality of the entire use process. At the same time, it also helps to better manage and maintain the stable operation of the AI ​​glasses system. In some embodiments, the above permission information can also be a permission verification code, and the user of the AI ​​glasses can use all the functions of the AI ​​glasses using the permission verification code.

[0064] In one embodiment, generating permission information based on the multimodal interaction information includes: Performing speech recognition on the speech information to identify keywords; encoding the keywords into numbers to obtain key numbers; The touch instruction is identified to obtain a touch pattern; along the touch direction of the touch pattern, the digital characters in the key number are added to the touch pattern one by one in sequence to obtain a digital pattern; Encoding the image information into a data stream character string, and discretely dividing the data stream character string into a plurality of adjacent areas of the digital pattern; In each of the adjacent areas, characters that satisfy a preset association relationship with the digital characters on the digital graphic are obtained as target characters, and the authority information is obtained by combining them.

[0065] In this embodiment, the collected voice information is first subjected to voice recognition. This process uses voice recognition technology to convert the content expressed by the user through voice into a text form that can be understood by the computer. Then keywords are extracted from these texts. Keywords are usually core words that can reflect the user's intentions and the functional requirements involved. For example, the user says "I want to set permissions". After voice recognition, the extracted keywords are "settings" and "permissions". These keywords carry key information about the user's operational expectations and are an important basis for subsequent further processing.

[0066] Furthermore, the purpose of encoding the identified keywords into numbers is to convert the information at the semantic level into a digital form that is convenient for subsequent calculation and processing. Pre-set encoding rules can be used, such as establishing a mapping table between keywords and numbers, and corresponding different keywords to specific numbers. Through such encoding, key numbers are obtained, so that the voice information exists in a concise and standardized digital sequence form, which is convenient for subsequent fusion with interactive information of other modalities and other operations, and is also conducive to unified regular processing in the process of generating permission information.

[0067] For the user's touch instructions to the AI ​​glasses, the corresponding touch sensing and recognition technology is needed to clarify the specific touch operation situation and then obtain the touch graph. The touch graph can be understood as an abstract representation of the user's touch path, touch range and other touch behaviors on a plane. For example, the user slides his finger in the touch area of ​​the AI ​​glasses to form a circular touch track, then after recognition, the corresponding circular touch graph will be obtained; or the user clicks on several specific points in sequence, and these points are connected to form a touch graph, which intuitively reflects the trajectory and method of the user's manual operation.

[0068] After obtaining the touch pattern, the digital characters in the key numbers previously encoded from the voice information are added to the touch pattern one by one along the touch direction. The purpose of this is to integrate the functional intent represented by the voice information with the operation behavior embodied by the touch command. By adding digital characters to the touch pattern, a new digital pattern is formed. This digital pattern contains both the spatial information of the touch operation and the digital coding information corresponding to the voice command, making it a unique representation that integrates the interactive information of the two modes.

[0069] Image information itself is a complex data composed of information such as the color and brightness of many pixels. In order to facilitate the combination and processing with other modal information, it needs to be encoded and converted into a data stream string. This can be achieved through image encoding algorithms. For example, it is common to arrange image pixel data in a certain order (such as from left to right, from top to bottom) and use a suitable encoding method (such as binary encoding) to convert it into a continuous data stream string, so that the image information is presented in a linear and easy-to-process data form.

[0070] Discretely distributing the encoded data stream string to multiple adjacent areas of the digital graphics is to integrate the image information into the digital graphics that have already integrated voice and touch command information in a decentralized manner. Each adjacent area can be regarded as a unit that carries a part of the image data stream. Through this discrete distribution method, the image information can be closely associated with the comprehensive interactive information represented by the digital graphics, further enriching the data content used to generate permission information, so that the finally generated permission information can comprehensively consider all aspects of multimodal interactive information.

[0071] In each adjacent area, characters associated with the digital characters on the digital graphic are searched based on a preset association relationship. This preset association relationship can be set based on a preset algorithm rule or mapping logic. For example, it is stipulated that in the adjacent area corresponding to the area where the number "1" in the digital graphic is located, characters that conform to a specific encoding rule (such as characters that meet the conditions after performing a certain operation on the ASCII code value of the character and the number, etc.) are searched. These characters found are the target characters. Alternatively, the positional relationship (distance, angle, etc.) between the characters in the adjacent area and the digital characters in the digital graphic can be calculated. When the positional relationship meets the preset conditions, it is used as the target character. In this way, characters that meet the requirements are screened out from a complex data structure that incorporates multimodal interactive information. These characters carry key information related to permission configuration extracted from different modal interactive information.

[0072] The target characters obtained in each adjacent area are combined to obtain the final permission information. The permission information is presented in the form of a character combination that has been processed in multiple steps and extracted by integrating multi-modal interaction information. It can accurately reflect the AI ​​glasses usage permissions configured based on the user's voice, touch, image acquisition and other interactive behaviors, and can then be sent back to the AI ​​glasses as a permission code for AI verification. The AI ​​glasses user can fully use the permissions of the AI ​​glasses by using this permission information, and other users cannot use it without permission information.

[0073] In one embodiment, generating permission information based on the multimodal interaction information includes: Performing speech recognition on the speech information to identify keywords; encoding the keywords to obtain encoded characters; Encoding the image information into a data stream character string; Based on the quantity characteristics and the character type characteristics of the coded characters, the data stream character string is rearranged to obtain a rearranged character string; The rearranged character string is discretely distributed into a preset pattern; wherein the preset pattern is composed of a plurality of circles with the same center and different radii; Identify the touch instruction to obtain a touch pattern; and superimpose the touch pattern onto the preset pattern in a preset manner; The characters at the position where the touch graphic and the preset graphic intersect are obtained as target characters, and the target characters are combined in order from outside to inside to obtain a character combination as the permission information.

[0074] In this embodiment, the voice information input by the user through the AI ​​glasses is converted into text form, and then keywords are screened out from the converted text. These keywords are usually words closely related to the operations that the user expects to perform and the objects of interest. The purpose of encoding the identified keywords is to convert the keywords at the semantic level into a coded character form that is convenient for subsequent calculation and processing. A set of flexible and extensible encoding rules can be pre-set, such as assigning different character codes according to factors such as the functional category, frequency of occurrence, and importance of the keywords.

[0075] The purpose of encoding image information into a data stream string is to convert complex image data at the visual level into a linear data form that is easy for computers to process. Using algorithms such as lossless compression coding, it is possible to arrange and combine multi-dimensional information such as color, brightness, and position of image pixels in a specific order while retaining key image information, generating a compact data stream string so that image information can work in conjunction with information such as voice and touch commands under the same data processing framework.

[0076] The data stream character string is rearranged based on the quantity characteristics and character type characteristics of the coded characters. For example, if there are more numeric characters in the coded characters, the numeric characters are placed in front when the data stream character string is rearranged; if there are more English characters, the English characters are placed in front when the data stream character string is rearranged.

[0077] The preset graph consisting of multiple circles with the same center and different radii is selected because it has natural hierarchical and regional division characteristics. The rearranged character string is discretely distributed to each area of ​​the preset graph according to certain rules. For example, according to the order of the characters in the character string, the characters at the beginning of the test are placed in the inner circle area, and the characters at the end are placed in the outer circle area, so as to achieve a structured layout of the information.

[0078] Then, with the help of high-precision touch sensing technology, the user's touch operation trajectory, touch point distribution, touch duration and other information on the AI ​​glasses are identified, and a touch graph is abstracted. This touch graph intuitively reflects the user's manual control intention, such as the user drawing a right-pointing arrow or a closed circle.

[0079] According to the preset method, such as taking the center of the preset graphic as the reference point, the touch graphic is aligned and superimposed according to the geometric center position to ensure that the touch graphic and the preset graphic are accurately integrated in space. After the touch graphic and the preset graphic are superimposed, the intersection area of ​​the two contains characters that combine the key information of the three modes of voice, image, and touch. The characters at these intersections are extracted as target characters because they are information carriers after multi-modal interaction and fusion.

[0080] Combine each target character in order from the outside to the inside, and the final character combination is the permission information, which accurately reflects the usage permissions (permission code) that should be granted to AI glasses based on the user's multimodal interaction behavior, and can be used for subsequent permission configuration management.

[0081] In one embodiment, generating permission information based on the multimodal interaction information includes: Analyze the intonation information of voice information and construct intonation curve; Analyze the touch duration of touch instructions and construct a touch curve; Analyze the color characteristics and color characteristic curves of each specified area of ​​the graphic information; The tone curve, the touch curve, and the color characteristic curve are superimposed in the same coordinate system to generate a combined curve; the combined curve is encoded to generate an authority code as the authority information.

[0082] In this embodiment, first, the intonation information of the voice information is analyzed to construct an intonation curve. When the user speaks to the AI ​​glasses, the voice analysis method is used to dig out the emotions and emphasis hidden in the intonation. By extracting fundamental frequency and other technologies, the intonation curve is drawn with time as the horizontal axis and intonation quantitative indicators as the vertical axis.

[0083] Next, analyze the touch duration of the touch command and construct a touch curve. The AI ​​glasses' high-sensitivity touch sensing hardware and time recording module can accurately measure the duration of each touch from start to end. Short clicks are often used to quickly select basic functions, such as switching display modes; long presses are often used to go into deep settings and activate complex functions, such as calling up image detail adjustments. Draw a curve with the touch time sequence as the horizontal axis and the duration as the vertical axis to show the user's manual control rhythm and gain insight into their operating habits and current needs.

[0084] Then, the color characteristics of each designated area of ​​the graphic information are analyzed to construct a color characteristic curve. When processing images collected by users, the areas are divided according to preset rules, such as the center of the picture, the target hot spot area, etc. For each area, the image color analysis algorithm is used to extract features such as average color value, contrast, saturation, etc., and a curve is constructed with the area identifier as the horizontal axis and the characteristic value as the vertical axis. The color dimension reflects the image dynamics or regional differences, adding visual perception characteristics to permission generation.

[0085] After that, the intonation curve, touch curve, and color feature curve are superimposed on the same coordinate system to generate a combined curve. With time or operation sequence as the horizontal axis, the three curves are merged to intuitively present the coordinated changes of multimodal information. For example, when the peak of intonation, long touch, and high contrast of color in the key area of ​​the image are synchronized, it may indicate that the user has made a fine operation on a specific area of ​​the screen. The combined curve fully captures the cross-modal linkage information.

[0086] Finally, the combined curve is encoded and a permission code is generated as permission information. The coding system is designed based on the geometric characteristics of the combined curve, such as the peak value, valley value, slope, area under the curve, and its distribution in different time periods and regions. The complex curve information is converted into a simple permission code to obtain the AI ​​glasses usage permission that the user deserves for multimodal interaction, which is used for subsequent permission configuration to ensure that it meets the needs and is safe and controllable.

[0087] Reference Figure 2 In another embodiment of the present invention, a multimodal interaction and image processing device for AI glasses is provided, comprising: A processing unit is used to pre-process the original image collected by the AI ​​glasses in real time based on the spatiotemporal attention mechanism to obtain a pre-processed image; perform speech recognition and semantic analysis on the voice command input by the user through the AI ​​glasses to obtain a text command; A matching unit, used for matching the text instruction with a preset image processing instruction template library by keywords to determine a corresponding image processing strategy; A positioning unit, which is used to predict the position of the target in the preprocessed image in the next frame by using a Kalman filter algorithm if the image processing strategy is dynamic tracking, and to search around the predicted position by using a twin network to accurately locate the target; A tracking unit is used to learn and update the appearance features of the target in real time based on the twin network, to achieve stable tracking of the target and output a tracking target image sequence for display to the user on the AI ​​glasses display.

[0088] In this embodiment, for the specific implementation of each unit in the above device embodiment, please refer to the above method embodiment, which will not be repeated here.

[0089] Reference Figure 3 In an embodiment of the present invention, an electronic device is also provided, the internal structure of the electronic device can be as follows Figure 3As shown. The electronic device includes a processor, a memory, a display screen, an input device, a network interface and a database connected via a system bus. Among them, the computer-designed processor is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the electronic device is used to store the corresponding data in this embodiment. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the above method is implemented.

[0090] Those skilled in the art will understand that Figure 3 The structure shown in the figure is merely a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the electronic device to which the solution of the present invention is applied.

[0091] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented. It can be understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0092] In summary, the multimodal interaction and image processing method, electronic device and storage medium of AI glasses provided in the embodiments of the present invention include: preprocessing the original image collected by AI glasses in real time based on the spatiotemporal attention mechanism to obtain a preprocessed image; performing speech recognition and semantic parsing on the voice instructions input by the user through the AI ​​glasses to obtain a text instruction; matching the text instruction with the preset image processing instruction template library for keywords to determine the corresponding image processing strategy; if the image processing strategy is dynamic tracking, the Kalman filter algorithm is used to predict the position of the target in the preprocessed image in the next frame, and the twin network is used to search around the predicted position to accurately locate the target; based on the twin network, the appearance features of the target are updated in real time to achieve stable tracking of the target and output the tracking target image sequence for display to the user on the AI ​​glasses display. In the present invention, the position of the target in the image in the next frame can be predicted, and then the target can be accurately located, and the appearance features of the target can be updated in real time to achieve stable tracking of the target. It overcomes the current defects of being unable to predict the subsequent appearance position of the target and being unable to stably track the target.

[0093] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided by the present invention and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-speed data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM.

[0094] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0095] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A multimodal interaction and image processing method for AI glasses, characterized in that: The following steps are involved: Based on the spatiotemporal attention mechanism, the original images collected by the AI ​​glasses in real time are preprocessed to obtain preprocessed images; the voice commands input by the user through the AI ​​glasses are subjected to speech recognition and semantic analysis to obtain text commands; Perform keyword matching between the text instruction and a preset image processing instruction template library to determine a corresponding image processing strategy; If the image processing strategy is dynamic tracking, the Kalman filter algorithm is used to predict the position of the target in the preprocessed image in the next frame, and the twin network is used to search around the predicted position to accurately locate the target; Based on the twin network, the appearance features of the target are updated in real time through real-time learning, so as to achieve stable tracking of the target and output a tracking target image sequence for display to the user on the AI ​​glasses display.

2. The multimodal interaction and image processing method of AI glasses according to claim 1, characterized in that: in, The Kalman filter algorithm dynamically adjusts the noise covariance matrix of the prediction model in the Kalman filter algorithm according to the historical acceleration and angular velocity of the target motion to improve the prediction accuracy.

3. The multimodal interaction and image processing method of AI glasses according to claim 1, characterized in that: After achieving stable tracking of the target and outputting the tracking target image sequence, it also includes: Performing detail enhancement on the tracking target image sequence based on the generator to obtain an enhanced image; Based on the discriminator determining the authenticity of the enhanced image, the generator and the discriminator are anti-iteratively optimized to improve the clarity and texture richness of the enhanced image for display to the user on the AI ​​glasses display.

4. The multimodal interaction and image processing method of AI glasses according to claim 1, characterized in that: The original images collected by AI glasses in real time are preprocessed based on the spatiotemporal attention mechanism to obtain preprocessed images, including: Use a three-dimensional convolutional neural network to convolve the original image in the spatial dimension, extract local detail features and global semantic features in the image, and obtain a spatial feature map; In the time dimension, for the original image, based on the long short-term memory network model, the temporal correlation between adjacent frames is learned, the dynamic change characteristics of the image sequence over time are captured, and the temporal feature sequence is output; Based on the attention mechanism, the spatial feature map is fused with the temporal feature sequence to obtain a spatiotemporal coherent feature map, and the preprocessed image is obtained through normalization.

5. The multimodal interaction and image processing method of AI glasses according to claim 1, characterized in that: Based on the twin network, the appearance features of the target are updated in real time through learning, the stable tracking of the target is achieved, and a tracking target image sequence is output, including: The image area of ​​the target tracked in the current frame is cropped and input into the twin network. The convolutional neural network based on the residual structure is used to extract its features to obtain the template feature vector containing the key features of the target. For the entire image of the current frame, a densely connected convolutional network is used to extract features to obtain a search feature map containing potential target areas; Calculate the similarity between each potential target area in the search feature map and the template feature vector, and compare it with the similarity threshold; construct a dynamic environment assessment model to dynamically adjust the similarity threshold in real time according to the rate of change of light intensity in the current image, the moving speed of background objects and the acceleration of target motion; The area with the greatest similarity and a similarity greater than the similarity threshold is obtained, and its corresponding feature information is fused with the template feature vector by assigning different weights so that the template feature vector is continuously updated to achieve stable tracking of the target and output the tracking target image sequence.

6. The multimodal interaction and image processing method of AI glasses according to claim 1, characterized in that: The method further comprises: Collecting multimodal interaction information between the user and the AI ​​glasses within a specified time period; wherein the multimodal interaction information includes voice information sent by the user, touch instructions of the user to the AI ​​glasses, and image information collected by the user controlling the AI ​​glasses; Sending the multimodal interaction information to a server; The server generates permission information based on the multimodal interaction information and sends it to the AI ​​glasses to configure the usage permissions of the AI ​​glasses.

7. The multimodal interaction and image processing method of AI glasses according to claim 6, characterized in that: Generating permission information based on the multimodal interaction information includes: Performing speech recognition on the speech information to identify keywords; encoding the keywords into numbers to obtain key numbers; The touch instruction is identified to obtain a touch pattern; along the touch direction of the touch pattern, the digital characters in the key number are added to the touch pattern one by one in sequence to obtain a digital pattern; Encoding the image information into a data stream character string, and discretely dividing the data stream character string into a plurality of adjacent areas of the digital pattern; In each of the adjacent areas, characters that satisfy a preset association relationship with the digital characters on the digital graphic are obtained as target characters, and the authority information is obtained by combining them.

8. A multimodal interaction and image processing device for AI glasses, characterized in that: include: A processing unit, used for preprocessing the original image collected by the AI ​​glasses in real time based on the spatiotemporal attention mechanism to obtain a preprocessed image; Perform speech recognition and semantic analysis on the voice commands input by the user through AI glasses to obtain text commands; A matching unit, used for matching the text instruction with a preset image processing instruction template library by keywords to determine a corresponding image processing strategy; A positioning unit, which is used to predict the position of the target in the preprocessed image in the next frame by using a Kalman filter algorithm if the image processing strategy is dynamic tracking, and to search around the predicted position by using a twin network to accurately locate the target; A tracking unit is used to learn and update the appearance features of the target in real time based on the twin network, to achieve stable tracking of the target and output a tracking target image sequence for display to the user on the AI ​​glasses display.

9. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images