Data processing method and device of intelligent glasses and electronic equipment

Abstract

The invention relates to the technical field of computer vision and artificial intelligence, and discloses a data processing method and device of intelligent glasses and electronic equipment. The method comprises the following steps: acquiring image data and acceleration data of a preset duration; extracting an image feature vector of the image data through a first network model; extracting an acceleration feature vector of the acceleration data through a second network model; and performing fusion analysis on the image feature vector and the acceleration feature vector through a third network model, and outputting a scene recognition result. According to the method, data acquisition, feature extraction and scene recognition are realized by deploying the lightweight private network model at the equipment end, the problem of high power consumption caused by continuous data acquisition is avoided, dependence on cloud processing is eliminated, and operation power consumption is reduced.

Description

Technical Field

[0001] This application relates to the fields of computer vision and artificial intelligence technology, specifically to a data processing method, device, and electronic device for smart glasses. Background Technology

[0002] Smart glasses are being integrated into daily life and many professional fields. However, the continuous collection of large amounts of data in these technologies leads to a significant increase in computing costs and power consumption, severely limiting the battery life and practicality of smart glasses. Furthermore, in order to identify specific scenarios with limited data, these technologies rely on multimodal or large visual language models for data processing. These models require enormous computing power, making edge deployment on smart glasses devices difficult. Summary of the Invention

[0003] This application provides a data processing method, apparatus, and electronic device for smart glasses. It features low power consumption, independent processing capability on the device side, and reduced unnecessary data transmission.

[0004] In a first aspect, this application provides a data processing method for smart glasses, comprising: collecting image data and acceleration data for a preset duration; extracting image feature vectors from the image data through a first network model; extracting acceleration feature vectors from the acceleration data through a second network model; and performing fusion analysis on the image feature vectors and acceleration feature vectors through a third network model to output scene recognition results.

[0005] In an optional implementation, this application further proposes that the second network model includes a statistical feature vector extraction branch and a temporal feature vector extraction branch. Extracting acceleration feature vectors from acceleration data using the second network model includes: extracting multi-dimensional statistical features from the acceleration data through the statistical feature vector extraction branch to obtain an acceleration statistical feature vector; performing temporal processing on the acceleration data based on the temporal feature vector extraction branch to obtain an acceleration temporal feature vector; and concatenating the acceleration statistical feature vector with the acceleration temporal feature vector to obtain the acceleration feature vector.

[0006] In an optional implementation, this application also proposes to extract multi-dimensional statistical features from acceleration data through a statistical feature vector extraction branch to obtain an acceleration statistical feature vector, including: grouping the acceleration data into a first preset number of groups, and extracting features from each group of acceleration data using a corresponding statistical feature vector extraction sub-branch to obtain a first-dimensional statistical feature vector corresponding to each group, wherein each statistical feature vector extraction sub-branch constitutes a statistical feature vector extraction branch; converting each first-dimensional statistical feature vector from three-dimensional data including the accelerometer x-axis, y-axis, and z-axis into a second-dimensional statistical feature vector of a preset dimension; and fusing the second-dimensional statistical feature vectors corresponding to each group to obtain the acceleration statistical feature vector.

[0007] In an optional implementation, this application also proposes to perform time-series processing on acceleration data based on a time-series feature vector extraction branch to obtain an acceleration time-series feature vector, including: performing a first channel swap on the acceleration data to obtain a time-series feature vector of a first preset channel; fusing the time-series feature vector of the first preset channel to obtain a time-series feature vector of a second preset channel; and performing a second channel swap on the time-series feature vector of the second preset channel to obtain an acceleration time-series feature vector.

[0008] In an optional implementation, this application also proposes to perform fusion analysis on image feature vectors and acceleration feature vectors through a third network model to output scene recognition results, including: fusing image feature vectors and acceleration feature vectors to obtain fused feature vectors; and classifying scenes based on the fused feature vectors to obtain scene recognition results.

[0009] In an optional implementation, this application also proposes to fuse image feature vectors and acceleration feature vectors to obtain a fused feature vector, including: unifying the dimensions of image feature vectors and acceleration feature vectors and then concatenating the feature vectors to obtain a concatenated feature vector; and performing fusion processing on the concatenated feature vector to obtain a fused feature vector.

[0010] In an optional implementation, this application also proposes that, after outputting the scene recognition result, the method further includes: if the scene recognition result is a target scene, then recording and saving the collected image data and acceleration data; if the scene recognition result is a non-target scene, then stopping the saving of the collected image data and acceleration data; if the scene recognition result is an unknown scene, then maintaining the data processing method of the previous scene.

[0011] In an optional implementation, this application also proposes that, after recording and saving the acquired image data and acceleration data, the method further includes: uploading the saved image data and acceleration data to a cloud server within a preset time period.

[0012] Secondly, this application provides a data processing device for smart glasses, the device comprising: a first acquisition module for acquiring image data and acceleration data for a preset duration; a second acquisition module for extracting image feature vectors from the image data through a first network model; a third acquisition module for extracting acceleration feature vectors from the acceleration data through a second network model; and a recognition module for fusing and analyzing the image feature vectors and acceleration feature vectors through a third network model, and outputting scene recognition results.

[0013] Thirdly, this application provides an electronic device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the data processing method for smart glasses described in the first aspect or any corresponding embodiment.

[0014] Fourthly, this application provides a computer-readable storage medium storing computer instructions for causing a computer to execute the data processing method for smart glasses described in the first aspect or any corresponding embodiment.

[0015] Fifthly, this application provides a computer program product, including computer instructions for causing a computer to execute the data processing method for smart glasses described in the first aspect or any corresponding embodiment.

[0016] The data processing method for smart glasses provided in this application firstly avoids the high computational cost and power consumption problems caused by continuous data collection by limiting the collection time; secondly, it uses dedicated network models to efficiently process image data and acceleration data, reducing the demand for high computing power; and finally, it completes multimodal decision-making directly on the edge through a fusion module, eliminating the dependence on network connection. This solves the technical problems of high computational cost, high power consumption, and network dependence in traditional methods, eliminates the dependence on cloud processing, reduces operating power consumption, and improves the independent processing capability of the edge. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating a data processing method according to an embodiment of this application; Figure 2This is a block diagram of a data processing method for smart glasses according to an embodiment of this application; Figure 3 This is a flowchart of the second network model processing according to an embodiment of this application; Figure 4 This is a flowchart of the third network model processing according to an embodiment of this application; Figure 5 This is a structural block diagram of a data processing apparatus according to an embodiment of this application; Figure 6 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] It is understood that before using the technical solutions disclosed in the various embodiments of this application, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this application in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0021] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0022] This application proposes a data processing method for smart glasses, such as... Figure 1 As shown, it includes: Step S101: Collect image data and acceleration data for a preset duration.

[0023] Specifically, acquiring image and acceleration data for a preset duration includes the process of obtaining visual and motion information recorded by the smart glasses' sensors within a specific time period. Specifically, the start and end times of data acquisition are controlled by setting a fixed time window, such as a time period in seconds, or the acquisition duration is dynamically adjusted according to environmental changes, reducing redundant data and computational burden.

[0024] Step S102: Extract image feature vectors from image data using the first network model.

[0025] Specifically, the extraction of image feature vectors from image data through the first network model can be achieved in various ways. For example, in addition to the convolutional neural network backbone network, lightweight deep learning models or traditional algorithms based on hand-designed features can also be used, in order to adapt to the computing power limitations of edge devices and ensure the efficiency of feature extraction.

[0026] Step S103: Extract acceleration feature vectors from the acceleration data using the second network model.

[0027] Specifically, different techniques can be used to extract acceleration feature vectors from acceleration data through the second network model. For example, statistical methods or frequency domain analysis can be used to extract key frequency components.

[0028] Step S104: The image feature vector and acceleration feature vector are fused and analyzed using the third network model to output the scene recognition result.

[0029] Specifically, various fusion strategies can be employed to fuse and analyze image feature vectors and acceleration feature vectors using a third network model. For example, a weighted average can be used to simply combine the two feature vectors, or an attention mechanism can be used to dynamically adjust the importance of different features, thereby achieving the collaborative integration of multimodal data and outputting reliable scene recognition results.

[0030] In one example, the first network model includes a convolutional neural network backbone, the second network model includes a triaxial dynamic compressor module, and the third network model includes a cross-modal fusion sensing module.

[0031] This application embodiment achieves low-power offline scene recognition through multiple steps. First, image data and acceleration data are collected for a preset duration. The preset duration effectively avoids redundant data from continuous collection, thereby reducing computational burden and power consumption. Further, image feature vectors are extracted from the image data using a first network model. This model employs a convolutional neural network backbone specifically designed for image feature extraction, ensuring recognition accuracy while reducing computational complexity and adapting to the computing power limitations of edge devices. Simultaneously, acceleration feature vectors are extracted from the acceleration data using a second network model. The integrated three-axis dynamic compressor module dynamically compresses the three-axis accelerometer data, effectively reducing data dimensionality while retaining key motion features, avoiding the resource overhead associated with high-dimensional acceleration data in traditional methods. Based on this, a third network model fuses and analyzes the image feature vectors and acceleration feature vectors. Its built-in cross-modal fusion perception module integrates visual and motion features collaboratively, directly outputting the scene recognition result. Because motion feature data is very small (Kb level), and image feature vectors are limited by the hardware of smart glasses, the hardware computing resources of smart glasses are sufficient to support the local data volume. Moreover, even if the image feature vectors are not rich enough, the collaborative analysis of motion feature data and image feature vectors ensures the accuracy of scene recognition results, thus eliminating the need to rely on external networks or large semantic models. It is understood that the technical solution of this embodiment achieves a low-power, high-efficiency offline working mode on resource-constrained edge devices such as smart glasses, significantly improving the practicality and applicability of the device.

[0032] In one example, such as Figure 2 As shown, image data and acceleration data are collected for a preset duration, specifically including: From the moment the smart glasses are powered on, the camera and accelerometer begin working. The camera uses a frame-per-second sampling method, capturing only one image per second; the accelerometer's sampling frequency is 50Hz, meaning it collects data 50 times per second. If the continuous data collection time after the smart glasses are powered on is less than 3 seconds, no model inference is performed; after the continuous data collection time reaches 3 seconds, two types of data are constructed for each current video frame: One type is image data, which consists of a two-dimensional array of three frames: one frame per second from the previous two seconds and the current frame. Taking a 1080p original image as an example, the array size is 3×224×224×3, where the first 3 represents the three frames, the third 3 represents the RGB three channels, and the two 224s in the middle are the width and height of the original image after uniform scaling.

[0033] Another type is accelerometer data, which is 50 three-dimensional vector data of the accelerometer per second in the first 2 seconds, forming a 100×3 array, where 100 represents 50×2 data points and 3 represents the three components of the accelerometer x-axis, y-axis and z-axis.

[0034] Continue to refer to Figure 2 Using a convolutional neural network backbone, taking MobileNetV3 as an example, the weights are reused through a Siamese network to perform inference on the three frames of images. After inference, the output one-dimensional vector (dimension 1×1280) of the last pooling layer of the network corresponding to each image is taken as the single-frame image feature. Since there are a total of 3 frames of images, a 3×1280 image feature matrix is ​​finally obtained.

[0035] Through a second network model, for example Figure 2 The three-axis dynamic compressor module in the model performs inference processing on the 100×3 accelerometer counting group to obtain a 3×192 accelerometer feature matrix.

[0036] Through a third network model, for example Figure 2 The (accelerometer-image) cross-modal fusion sensing module performs a fusion operation on the features of two modalities: the image feature matrix (3×1280) and the accelerometer feature matrix (3×192). The fused feature matrix is ​​a vector with a dimension of 3. The three elements of the vector correspond to three scenarios (target scenario, non-target scenario, and unknown scenario).

[0037] It is understandable that this application addresses the resource-constrained scenarios of smart glasses by optimizing the data acquisition and processing flow to achieve low-power offline recognition. Firstly, limiting the acquisition duration avoids the high computational cost and power consumption issues caused by continuous acquisition. Secondly, dedicated network models are used to efficiently process image and acceleration data, reducing the need for high computing power. Finally, the multimodal decision-making is completed directly on the edge through a fusion perception module, eliminating dependence on network connectivity and solving the technical problems of high computational cost, high power consumption, and network dependence inherent in traditional methods.

[0038] In some embodiments, this application further proposes a second network model including a statistical feature vector extraction branch and a temporal feature vector extraction branch, which extracts acceleration feature vectors from acceleration data, including: Step a1: Through the statistical feature vector extraction branch, multi-dimensional statistical features are extracted from the acceleration data to obtain the acceleration statistical feature vector.

[0039] The statistical feature vector extraction branch is a module used to extract static distribution characteristics from acceleration data. It employs various statistical analysis algorithms, such as sliding window-based statistical calculations or grouped statistical processing. This captures the spatial characteristics of acceleration data, providing more comprehensive static feature support for scene recognition. The multi-dimensional features include at least two of the following: minimum, maximum, mean, variance, standard deviation, range, median, median absolute deviation, root mean square value, signal energy value, zero-crossing rate, and slope.

[0040] Step a2: Based on the extraction branch of the time-series feature vector, perform time-series processing on the acceleration data to obtain the acceleration time-series feature vector.

[0041] The time-series feature vector extraction branch is a module used to extract dynamic change patterns from acceleration data, which can be implemented using time series modeling techniques. This branch is designed to capture the changing patterns of acceleration data over time, thereby supplementing the deficiencies of statistical features.

[0042] Step a3: Concatenate the acceleration statistical feature vector with the acceleration time series feature vector to obtain the acceleration feature vector.

[0043] Among them, the splicing operation refers to the process of merging two feature vectors from different sources into a comprehensive feature vector according to a preset rule. It can be achieved through simple vector concatenation or weighted fusion, and is used to integrate multi-dimensional information to improve feature expression capabilities.

[0044] The technical solution in this embodiment effectively solves the problem of incomplete feature extraction from acceleration data through a dual-branch network structure design, thereby achieving efficient and low-power feature processing on the edge device. The statistical feature vector extraction branch and the temporal feature vector extraction branch work in parallel, independently processing the static distribution characteristics and dynamic change patterns of the acceleration data, respectively. This structural division allows for the full exploitation of different characteristics of the acceleration data, avoiding information loss that might occur with a single feature extraction method. By extracting multi-dimensional statistical features from the acceleration data, relevant information about the device's stable state can be obtained, while temporal processing of the acceleration data can capture the dynamic changes in the device's motion behavior. Finally, the statistical feature vector and the temporal feature vector are concatenated to form a comprehensive acceleration feature vector. This processing method not only preserves the multi-dimensional information of the original data but also simplifies the subsequent multimodal analysis process, thereby reducing computational complexity, power consumption, and improving the reliability of scene recognition on resource-constrained smart glasses devices.

[0045] In some embodiments, this application further proposes specific steps for extracting multi-dimensional statistical features from acceleration data through a statistical feature vector extraction branch to obtain acceleration statistical feature vectors, including: Step b1: The acceleration data is grouped into a first preset number of groups, and the corresponding statistical feature vector extraction sub-branch is used to extract features from each group of acceleration data to obtain the first dimension statistical feature vector corresponding to each group. Each statistical feature vector extraction sub-branch constitutes a statistical feature vector extraction branch.

[0046] The first preset number of groups is designed to decompose the continuous data stream into multiple manageable small-scale units, thereby reducing the amount of data processed per instance and avoiding memory spikes and computational overload. The statistical feature vector extraction sub-branch refers to an independent processing unit designed for the specific characteristics of the grouped data; it can achieve efficient feature extraction through customized statistical functions or machine learning models.

[0047] Step b2: Convert each first-dimensional statistical feature vector from the three-dimensional data including the accelerometer x-axis, y-axis and z-axis into a second-dimensional statistical feature vector of a preset dimension.

[0048] The first dimension statistical feature vector is an intermediate result extracted from the original triaxial data, which retains key statistical information. It is used to ensure that the basic data for subsequent processing is representative and has appropriate dimensionality.

[0049] Step b3: Fuse the second-dimensional statistical feature vectors corresponding to each group to obtain the acceleration statistical feature vector.

[0050] The second-dimensional statistical feature vector is obtained by concatenating the dimensions of the first-dimensional statistical feature vector. This is used to eliminate axial differences and reduce the feature size, while maintaining the expressive power of the features to support accurate scene recognition.

[0051] The technical solution implemented here significantly optimizes the efficiency of statistical feature extraction from acceleration data through a collaborative design of group processing, dimensional transformation, and feature fusion. First, grouping the acceleration data effectively reduces the data size per processing iteration, making it particularly suitable for edge devices with limited computing power. Second, feature extraction is performed using corresponding statistical feature vector extraction sub-branches based on the grouped data, allowing each sub-branch to process data specific to the characteristics of its group, ensuring the integrity of key statistical features. Third, each first-dimensional statistical feature vector is transformed from three-dimensional data into a second-dimensional statistical feature vector with a preset dimension, achieving dimensionality compression based on feature redundancy analysis, significantly reducing the feature vector size and lowering subsequent transmission and storage overhead. Finally, the second-dimensional statistical feature vectors corresponding to each group are fused, integrating the local features from the group processing, avoiding information fragmentation, and generating compact and information-rich acceleration statistical feature vectors, providing high-quality input for multimodal fusion.

[0052] In some embodiments, this application further proposes a method for performing time-series processing on acceleration data based on a time-series feature vector extraction branch to obtain acceleration time-series feature vectors, including: Step c1: Perform a first channel swap on the acceleration data to obtain the temporal feature vector of the first preset channel.

[0053] The first channel swap refers to the operation of dynamically adjusting the data dimensions by reorganizing the original time-series structure of the accelerometer's three-axis data. This reduces the computational redundancy caused by directly processing high-dimensional raw data, while preserving key time-series patterns for efficient subsequent capture.

[0054] Step c2: Fuse the timing feature vectors of the first preset channel to obtain the timing feature vectors of the second preset channel.

[0055] The second channel exchange refers to the operation of readjusting the data format according to the shape of the fused feature vector. This can be achieved through multidimensional tensor transformation or channel mapping technology to ensure that the output acceleration time-series feature vector can be seamlessly connected with the image feature vector in structure, thereby providing compatibility support for subsequent multimodal fusion analysis.

[0056] Step c3: The timing feature vector of the second preset channel is swapped to obtain the acceleration timing feature vector.

[0057] The technical solution in this embodiment optimizes the feature extraction process for the multi-axis temporal characteristics of acceleration data through a channel swapping and fusion mechanism. First, the first channel swapping operation reorganizes the data dimensions according to the temporal structure characteristics of the accelerometer's three-axis data, avoiding redundant calculations from directly processing high-dimensional raw data and making the temporal patterns easier to capture efficiently. Next, the temporal feature vectors of the first preset channel are fused. This step, based on the data layout after channel swapping, integrates multi-channel feature information to achieve dimensional compression and extraction of key dynamic features, preserving the temporal regularity of acceleration changes while significantly reducing the computational load of subsequent processing. Finally, the second channel swapping operation adjusts the data format according to the shape of the fused feature vector, ensuring seamless structural connection between the output acceleration temporal feature vector and the image feature vector. This embodiment utilizes the inter-axis correlation of acceleration data through phased channel operations, replacing the high-overhead processing of traditional temporal models with data rearrangement and simplification, reducing the computational burden and adapting it to the resource constraints of edge devices. This solves the problem of low efficiency caused by insufficient computing resources in real-time scene recognition by edge devices.

[0058] In one example, such as Figure 3 The flowchart of the second network model shown is as follows. Specifically, it includes: Step 1: Construct a statistical feature extraction branch, splitting the input 100×3 data into three parts: 33×3, 34×3, and 33×3. Feature extraction is then performed on each of these three parts. For each statistical feature extraction branch, 12 statistical features are extracted: minimum, maximum, mean, variance, standard deviation, range, median, median absolute deviation, root mean square value, signal energy value, zero-crossing rate, and slope. The data along the x, y, and z axes are then concatenated to obtain a 36-dimensional vector. Finally, a fusion layer is used to fuse these features, resulting in a 128-dimensional feature vector. Since the data is split into three time components, this branch ultimately outputs a 3×128 feature vector.

[0059] The specific structure of the fusion layer is as follows: 1. A fully connected layer with an input dimension of 36 and an output dimension of 256; 2. A one-dimensional batch normalization layer; 3. A ReLU activation layer; 4. A random deactivation layer, wherein the deactivation rate is set to 0.3; 5. A fully connected layer with an input dimension of 256 and an output dimension of 128; 6. A Tanh activation layer.

[0060] Step 2, simultaneously construct the feature extraction branch of the temporal convolutional network, such as... Figure 3The fourth branch of the algorithm has the following processing flow: First, the input data channels are swapped to convert it into a 3×100 dimension (i.e., 3 is considered as the number of channels); then, it passes through a one-dimensional convolutional layer with 3 input channels, 32 output channels, and a kernel size of 5; then, it passes through a one-dimensional batch normalization layer and a ReLU activation layer; then, it passes through a one-dimensional convolutional layer with 32 input channels, 64 output channels, and a kernel size of 5; then, it passes through a one-dimensional batch normalization layer and a ReLU activation layer; then, it passes through an adaptive average pooling layer with an output of 3; finally, the channels are swapped again to obtain a 3×64 output vector (where 3 represents the three time parts and 64 is the dimension of the extracted feature vector).

[0061] Step 3: Concatenate the feature dimensions of the two branches to obtain a 3×192 vector, which is the output feature dimension of the three-axis dynamic compressor module.

[0062] It is understandable that this embodiment takes into account both statistical features and temporal features. Through a dual-branch design, it extracts 12 statistical features such as minimum value and variance to capture the global pattern of the data. On the other hand, it retains the dynamic trend of acceleration change through a temporal convolutional network, thereby solving the problem of scene discrimination bias caused by a single feature extraction method.

[0063] In addition, this embodiment is adapted to non-stationary motion data. For the irregularity of user actions in housework scenarios, such as chopping vegetables and folding clothes, it improves the feature representation ability under complex motion states by splitting time segments for feature extraction, thereby making the scene recognition results more accurate.

[0064] The technical solution in this embodiment, combined with the aforementioned statistical feature vector extraction branch, constitutes a complete acceleration feature vector extraction process. By extracting and concatenating statistical features and temporal features separately, the final generated acceleration feature vector not only contains rich static statistical information but also retains key patterns of dynamic temporal changes, thus providing high-quality input data for subsequent multimodal fusion analysis. This improves the real-time performance and accuracy of edge devices such as smart glasses under low-power conditions.

[0065] In some implementations, this application further proposes to fuse and analyze the image feature vector and the acceleration feature vector using a third network model to output scene recognition results, including: Step d1: The image feature vector and acceleration feature vector are fused using a third network model to obtain a fused feature vector.

[0066] Among them, the fused feature vector refers to a compact representation generated by integrating image feature vectors and acceleration feature vectors. It can be implemented by methods such as concatenation, weighted summation, or attention mechanisms. It is used to reduce the complexity of multimodal data processing while retaining key spatial and motion information, thereby providing efficient and concise input for subsequent scene classification.

[0067] Step d2: Perform scene classification based on the fused feature vector to obtain scene recognition results.

[0068] Scene classification refers to the process of using a classifier to predict based on fused feature vectors. It is used to map the fused features to specific scene categories in order to achieve accurate scene recognition.

[0069] The technical solution in this embodiment optimizes the efficiency of multimodal data processing through a structured decomposition and fusion analysis process. First, the fusion operation of image feature vectors and acceleration feature vectors is based on dimensionality-reduced features extracted from the preceding model, avoiding the computational burden of directly processing the original high-dimensional data. Second, the fusion process fully utilizes the complementarity between the spatial information of image features and the motion information of acceleration features to construct a compact fusion representation, thereby achieving efficient integration at the feature level. Subsequently, when classifying scenes based on the fused feature vectors, only the simplified fusion vectors need to be processed instead of the original multimodal inputs, significantly simplifying the computational requirements of the classifier and improving recognition speed and energy efficiency. This is suitable for the core requirements of real-time performance and low power consumption for edge devices such as smart glasses, effectively solving the problems of high computational complexity, high power consumption, and insufficient real-time performance in traditional methods.

[0070] In some implementations, this application further proposes fusing the image feature vector and the acceleration feature vector to obtain a fused feature vector, including: Step e1: After unifying the dimensions of the image feature vector and the acceleration feature vector, the feature vectors are concatenated to obtain the concatenated feature vector.

[0071] Unifying the dimension refers to adjusting feature vectors from different modalities to the same spatial representation, which can be achieved through linear transformations, embedding mappings, or normalization operations. Since image feature vectors are typically generated by convolutional neural networks and contain high-dimensional spatial information, while acceleration feature vectors may focus on low-dimensional temporal dynamic characteristics, unifying the dimension effectively avoids information loss or computational redundancy caused by modal differences. This operation ensures that multimodal data are aligned in the same feature space, thus laying the foundation for subsequent stitching and fusion.

[0072] Step e2: Perform fusion processing on the concatenated feature vectors to obtain the fused feature vector.

[0073] In this process, feature vector concatenation refers to combining image feature vectors processed with uniform dimensions with acceleration feature vectors according to preset rules to form a new feature vector. This can be achieved through methods such as direct concatenation, weighted superposition, or sparse coding. This preserves the complementarity of the two modalities while avoiding increased computational complexity caused by dimensionality conflicts. Furthermore, fusion processing involves further refining the concatenated feature vectors to enhance discriminative features and compress redundant information. This can be achieved using techniques such as attention mechanisms, principal component analysis, or deep neural networks. The generated fused feature vector is suitable for scene classification tasks.

[0074] In one example, such as Figure 4 The flowchart for the third network model processing is shown below. Specifically, it includes: For an input image feature vector of 3x1280, a projection layer (a fully connected layer with 1280 inputs and 256 outputs) is first applied to obtain a 3x256 feature vector. For an acceleration feature vector of 3x192, a projection layer (a fully connected layer with 192 inputs and 256 outputs) is applied to obtain a 3x256 feature vector.

[0075] Next, the image features 3x256 and the acceleration features 3x256 are concatenated to obtain 3x512.

[0076] Then, it passes through a fully connected layer with an input of 512 and an output of 256, a ReLU activation layer, and another fully connected layer with an input of 256 and an output of 256.

[0077] Next, a GRU time series layer is input, with 256 input features and 256 output features. The last time step of the output is taken to obtain 256 features.

[0078] Finally, input a fully connected classification layer with an input of 256 and an output of 3, resulting in a vector with an output dimension of 3 for each output data point.

[0079] It is understood that this embodiment breaks through the limitations of a single modality. The image modality has advantages in recognizing visual details in a scene, while the acceleration modality is good at capturing motion. By fusing the two modalities, it can solve the problem of static visual misjudgment, such as misjudging a cooking scene when only observing a kitchen scene; at the same time, it can solve the problem of motion blur misjudgment, such as the problem of misjudgment caused by camera shaking during walking, thereby improving the accuracy of target scene recognition.

[0080] Furthermore, this embodiment reduces the difficulty of edge deployment by using a projection layer to uniformly convert the features of the two modalities into a 256-dimensional vector, thereby reducing the computational complexity in the cross-modal fusion process. Combined with the GRU temporal layer's ability to capture temporal correlations, it can adapt to the edge computing capabilities of smart glasses.

[0081] Furthermore, for intermittent data acquisition schemes with 1 frame per second and 50 accelerations, the fusion perception module can effectively associate cross-modal features at different time steps, solve the feature breakage problem caused by data sampling intervals, and ensure the continuity and stability of scene discrimination.

[0082] In some implementations, this application further proposes that after outputting the scene recognition results, the method also includes: Step f1: If the scene recognition result is the target scene, then record and save the collected image data and acceleration data.

[0083] In this context, the target scenario refers to a scenario type with specific value or significance that is pre-defined based on actual application needs. This is achieved by classifying and matching the scenario recognition results with relevant labels. For example, in the scenario where smart glasses are used for home health management, the target scenario could be behaviors such as cooking or eating.

[0084] Step f2: If the scene recognition result is not the target scene, then stop saving the collected image data and acceleration data.

[0085] Non-target scenes refer to other scene types that are unrelated to the target scene. The basis for this judgment is that the scene recognition result does not match the preset target scene classification label.

[0086] Step f3: If the scene recognition result is an unknown scene, then maintain the data processing method of the previous scene.

[0087] The technical solution in this embodiment improves the data acquisition efficiency of smart glasses in resource-constrained environments by dynamically associating scene recognition results with data storage behavior. First, the data storage mechanism for target scenes ensures that data is captured only for high-value scenes, effectively reducing storage space usage and subsequent transmission traffic requirements. Second, the data termination and storage mechanism for non-target scenes reduces continuous power consumption and processor load by immediately stopping the storage process of invalid data, making it suitable for the low-power operation requirements of edge devices. Furthermore, the maintenance of the previous scene data processing method for unknown scenes leverages the continuity of historical decisions to address recognition uncertainties, preventing the loss of critical data due to temporary misjudgments or system anomalies caused by frequent strategy switching. Conditional storage based on recognition results not only tightly couples the data acquisition process with scene value but also enhances the system's adaptability in offline environments.

[0088] Furthermore, the technical solution in this embodiment forms an organic whole with the technical process of scene recognition result output. By classifying and judging the scene recognition results, precise control of data saving behavior is achieved, thereby maximizing the utilization efficiency of storage, computing, and transmission resources while ensuring the capture of key data. This solution is suitable for the high-efficiency, low-power data acquisition needs of edge devices such as smart glasses in offline or network-constrained environments, demonstrating its innovation in solving the data acquisition problem for resource-constrained devices.

[0089] This application further proposes a method that, after recording and saving the acquired image data and acceleration data, includes: Step g1: Upload the saved image data and acceleration data to the cloud server within a preset time period.

[0090] The preset time period refers to a time interval or specific moment set according to actual needs. This can be achieved through fixed time intervals, dynamically adjusted time windows, or triggering events based on device status. It is used to optimize data transmission strategies and avoid resource waste caused by frequent network interactions. Cloud servers include remote server clusters with large-scale storage and computing capabilities, deployed through public, private, or hybrid cloud architectures to provide a stable and reliable data storage and processing environment.

[0091] The technical solution in this embodiment filters target scene data through a conditional data storage mechanism, temporarily stores this data locally, and introduces a preset time period upload mechanism to centrally process data transmission tasks. This effectively reduces frequent network requests caused by real-time uploads, thereby reducing traffic consumption and power burden. Simultaneously, the preset time period allows devices to upload data in batches when network conditions are good or the device is charging, adapting to the offline working needs of edge devices while ensuring data transmission reliability. Especially in network-free environments, the device can temporarily store data and automatically resume transmission after the network is restored. This mechanism is unique in that it takes into account the low-power characteristics and offline operation requirements of edge devices such as smart glasses, providing a feasible path for the efficient collection of embodied intelligence training datasets.

[0092] Furthermore, the technical solution in this embodiment is combined with the conditional data storage mechanism for scene recognition results. By first filtering the target scene data and storing it locally, and then uploading it to the cloud server based on a preset time period, it not only solves the problem of redundant accumulation of unnecessary data, but also overcomes the risks of real-time upload failure and storage overflow, thereby improving the continuity and reliability of data collection.

[0093] This embodiment also provides a data processing device for smart glasses, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0094] This embodiment provides a data processing device for smart glasses, such as... Figure 5 As shown, it includes: The first acquisition module 501 is used to acquire image data and acceleration data for a preset duration.

[0095] The second acquisition module 502 is used to extract the image feature vector of the image data through the first network model.

[0096] The third acquisition module 503 is used to extract the acceleration feature vector of the acceleration data through the second network model.

[0097] The fourth acquisition module 504 is used to perform fusion analysis on the image feature vector and the acceleration feature vector through the third network model, and output the scene recognition result.

[0098] The data processing device for smart glasses provided in this application can execute the data processing method for smart glasses provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects for executing the method. Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0099] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0100] The following is a detailed reference. Figure 6 This diagram illustrates a suitable structural schematic for implementing the electronic device described in the embodiments of this application. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from memory 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of the electronic device. The processor 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0101] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0102] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a memory 608, or installed from a ROM 602. When the computer program is executed by the processor 601, it performs the functions defined in the data processing method for smart glasses according to embodiments of this application.

[0103] Figure 6 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0104] This application also provides a computer-readable storage medium. The methods described in this application can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the data processing method for smart glasses shown in the above embodiments is implemented.

[0105] A portion of this application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0106] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and all such modifications and variations fall within the scope defined by the appended claims.

Claims

1. A data processing method for smart glasses, characterized in that, The method includes: Collect image and acceleration data for a preset duration; The image feature vector of the image data is extracted using the first network model; The acceleration feature vector of the acceleration data is extracted using a second network model; The image feature vector and the acceleration feature vector are fused and analyzed using a third network model to output scene recognition results.

2. The method according to claim 1, characterized in that, The second network model includes a statistical feature vector extraction branch and a temporal feature vector extraction branch. Extracting the acceleration feature vector from the acceleration data using the second network model includes: Through the statistical feature vector extraction branch, multi-dimensional statistical features are extracted from the acceleration data to obtain the acceleration statistical feature vector; Based on the extraction branch of the time-series feature vector, the acceleration data is processed in a time-series manner to obtain the acceleration time-series feature vector; The acceleration statistical feature vector is concatenated with the acceleration time series feature vector to obtain the acceleration feature vector.

3. The method according to claim 2, characterized in that, The step of extracting multi-dimensional statistical features from the acceleration data through the statistical feature vector extraction branch to obtain an acceleration statistical feature vector includes: The acceleration data is grouped into a first preset number of groups, and the acceleration data of each group is extracted using the corresponding statistical feature vector extraction sub-branch to obtain the first dimension statistical feature vector corresponding to each group, wherein each statistical feature vector extraction sub-branch constitutes the statistical feature vector extraction branch; Each first-dimensional statistical feature vector is converted from three-dimensional data including the x-axis, y-axis, and z-axis of the accelerometer into a second-dimensional statistical feature vector of a preset dimension; The second dimension statistical feature vectors corresponding to each group are fused to obtain the acceleration statistical feature vector.

4. The method according to claim 2, characterized in that, The step of extracting the acceleration data based on the time-series feature vector, and performing time-series processing on the acceleration data to obtain the acceleration time-series feature vector, includes: The acceleration data is subjected to a first channel swap to obtain the time-series feature vector of the first preset channel; The timing feature vectors of the first preset channel are fused to obtain the timing feature vectors of the second preset channel; The timing feature vector of the second preset channel is swapped with the second channel to obtain the acceleration timing feature vector.

5. The method according to claim 1, characterized in that, The process of fusing and analyzing the image feature vector and the acceleration feature vector using a third network model to output scene recognition results includes: The image feature vector and the acceleration feature vector are fused to obtain a fused feature vector; Scene classification is performed based on the fused feature vector to obtain the scene recognition result.

6. The method according to claim 5, characterized in that, The process of fusing the image feature vector and the acceleration feature vector to obtain a fused feature vector includes: The image feature vector and the acceleration feature vector are unified in dimension and then concatenated to obtain the concatenated feature vector; The concatenated feature vectors are fused to obtain the fused feature vector.

7. The method according to claim 1, characterized in that, After outputting the scene recognition result, the method further includes: If the scene recognition result is the target scene, then the collected image data and acceleration data are recorded and saved; If the scene recognition result is a non-target scene, then stop saving the collected image data and acceleration data; If the scene recognition result is an unknown scene, the data processing method of the previous scene will be maintained.

8. The method according to claim 7, characterized in that, After recording and saving the acquired image data and acceleration data, the method further includes: Within a preset time period, the saved image data and acceleration data are uploaded to the cloud server.

9. A data processing device for smart glasses, characterized in that, The device includes: The first acquisition module is used to collect image data and acceleration data for a preset duration; The second acquisition module is used to extract the image feature vector of the image data through the first network model; The third acquisition module is used to extract the acceleration feature vector of the acceleration data through the second network model; The recognition module is used to perform fusion analysis on the image feature vector and the acceleration feature vector through a third network model, and output the scene recognition result.

10. An electronic device, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the data processing method for smart glasses according to any one of claims 1 to 8.

Images