AI glasses and smart phone data interaction system under intelligent scene perception

Abstract

The invention relates to a data interaction system for AI glasses and a smart phone under intelligent scene perception, and relates to the technical field of data interaction, and the system comprises a scene perception module which is used for reading position data, direction data and attitude angle data of the AI glasses and executing scene data initialization of the smart phone; the perception processing module is used for activating a scene perception sensor of the AI glasses and carrying out scene perception; the judgment module is used for executing resource state sensing of the AI glasses and the smart phone and dynamically configuring a sensing task; and the interactive processing module is used for packaging and uploading the low-dimensional scene state vectors identified by the smart phone, and respectively executing scene identification processing. The technical problems that in the prior art, the cooperation efficiency of the AI glasses and the smart phone is low, resource utilization is unbalanced, and the real-time sensing and processing capacity is insufficient in a miscellaneous scene are solved, and the technical effects that efficient cooperation and dynamic load balancing of the AI glasses and the smart phone are achieved, and scene sensing real-time performance and energy efficiency are improved are achieved.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of data interaction, and in particular to an AI glasses and smart phone data interaction system under intelligent scene perception. BACKGROUND

[0002] With the development of augmented reality (AR) and artificial intelligence (AI), AI glasses and smart phones and other intelligent devices have become important tools. AI glasses have great potential in navigation, education, industrial maintenance and medical treatment due to their portability and real-time interaction capability. However, AI glasses are limited by computing resources, battery life and sensor capabilities, and cannot independently complete complex scene perception and data processing tasks. In contrast, smart phones have stronger processing capabilities, rich sensor arrays and mature communication modules, but their mobility and real-time performance are not as good as AI glasses. Therefore, how to realize efficient cooperation between AI glasses and smart phones to improve the overall performance of intelligent scene perception has become a key problem. The data interaction of existing intelligent devices mainly depends on one-way transmission or simple instruction synchronization, and lacks dynamic resource coordination and task allocation mechanisms. For example, AI glasses may only send raw sensor data to smart phones for processing, resulting in high communication delay, high energy consumption and inability to fully utilize the computing advantages of both ends. In addition, scene perception tasks usually require real-time response to environmental changes, and it is difficult to guarantee processing efficiency and accuracy under resource-limited conditions, which further affects the application effect of intelligent devices in complex scenes such as dynamic navigation, multi-target recognition and real-time decision-making.

[0003] Therefore, in the related art at present, there are technical problems of low cooperation efficiency of AI glasses and smart phones, unbalanced resource utilization and insufficient real-time perception and processing capability in miscellaneous scenes. SUMMARY

[0004] The AI glasses and smart phone data interaction system and method under intelligent scene perception provided by the application solve the technical problems of low cooperation efficiency of AI glasses and smart phones, unbalanced resource utilization and insufficient real-time perception and processing capability in miscellaneous scenes in the prior art, and achieve the technical effects of efficient cooperation of AI glasses and smart phones, dynamic load balancing, improved scene perception real-time performance and energy efficiency.

[0005] The application provides an AI glasses and smart phone data interaction system under intelligent scene perception, which comprises a scene perception module, a perception processing module, a discrimination module and an interaction processing module.

[0006] In possible implementation manners, the AI glasses and smart phone data interaction system under intelligent scene perception further performs the following processing: a processor of the AI glasses collects processor utilization, battery capacity and sensor occupancy to establish a resource state description vector of the AI glasses; the smart phone collects computing idle degree and cache occupancy data to establish a resource state description vector of the smart phone; heat generation data of the AI glasses and the smart phone are acquired to establish a resource state penalty factor according to the heat generation data; after the resource state description vectors of the AI glasses and the smart phone are normalized, principal component extraction is performed to acquire resource load feature vectors of the AI glasses and the smart phone respectively; a change rate of the resource load feature vectors is calculated by using a sliding window to establish a time sequence state matrix of resource state trend, the time sequence state matrix is subjected to multi-scale fusion and is penalized and compensated by the resource state penalty factor to establish a resource state perception result.

[0007] In possible implementation manners, the AI glasses and smart phone data interaction system under intelligent scene perception further performs the following processing: the low-dimensional scene state vector is subjected to cluster analysis, the cluster analysis comprises cluster analysis based on spatial position, target semantics and dynamic characteristics to establish a multi-level cluster result; communication bandwidth, delay and power consumption of the double-end communication link are acquired to establish a transmission penalty coefficient; the transmission penalty coefficient is taken as a constraint term, the resource state perception result and the multi-level cluster result are used for adaptive task adaptation analysis of the smart phone and the AI glasses to establish an adaptation analysis result; and the dynamic configuration of the perception task is completed according to the adaptation analysis result.

[0008] In a possible implementation, the AI glasses under the intelligent scene perception and the smart phone data interaction system further perform the following processing: a space pose vector is established according to the position data, direction data and attitude angle data; a local map and an environmental semantic database of a smart phone end are called based on the space pose vector, and an environmental semantic block in a current field of view range is retrieved; environmental complexity analysis of the environmental semantic block is performed, and a first extraction constraint is established; a precision setting of prior perception is obtained, and a second extraction constraint is established according to the precision setting; after fusion of the first extraction constraint and the second extraction constraint, feature compression and structure simplification of the environmental semantic block are performed, and scene framework data constituted by a geometric contour and a semantic label boundary are established.

[0009] In a possible implementation, the AI glasses under the intelligent scene perception and the smart phone data interaction system further perform the following processing: authenticity and timeliness verification is performed on the scene framework data, and a verification result is established; if the verification result is a pass result, a scene perception sensor is directly activated, the scene framework data is used as a prior template, modal alignment and feature decoupling of visual and acoustic sensing signals are performed, semantic consistency fusion of multi-source perception information is performed by using a cross-modal attention mechanism, and a scene semantic feature set is established; feature dimension reduction and state embedding calculation are performed on the scene semantic feature set, and the scene semantic feature set is projected into a low-dimensional scene state vector.

[0010] In a possible implementation, the AI glasses under the intelligent scene perception and the smart phone data interaction system further perform the following processing: if the verification result is a verification failure result, a verification collection instruction is activated; the scene perception sensor is controlled to collect data according to the verification collection instruction, and a collection data set is established; same-position feature extraction of the scene framework data is performed on the collection data set, and feature verification comparison is performed; a linkage feature analysis of the collection data set is performed by using a feature verification comparison result as a framework template, a scene semantic feature set is established, and a low-dimensional scene state vector is established according to the scene semantic feature set.

[0011] In a possible implementation, the AI glasses under the intelligent scene perception and the smart phone data interaction system further include a perception enhancement module configured to perform active gaze perception of a user based on the AI glasses, establish an attention meta event, send the attention meta event to the smart phone, establish scene attention, perform scene trigger analysis of a scene recognition result according to the scene attention, establish enhanced content, and return the enhanced content to the AI glasses and display the enhanced content.

[0012] In a possible implementation, the AI glasses and the smart phone data interaction system under the intelligent scene perception further include a short-time scene prediction module configured to acquire a preset resource library of a user, perform short-time scene prediction according to a scene recognition result and the preset resource library, establish predicted interaction content, and push the predicted interaction content to the AI glasses for display.

[0013] The AI glasses and the smart phone data interaction system under the intelligent scene perception provided in the present application, a scene perception module is configured to read position data, direction data and attitude angle data of the AI glasses and perform scene data initialization of the smart phone; a perception processing module is configured to activate a scene perception sensor of the AI glasses and perform scene perception; a discrimination module is configured to perform resource state perception of the AI glasses and the smart phone and dynamically configure a perception task; and an interaction processing module is configured to pack and upload a low-dimensional scene state vector identified by the smart phone and perform scene recognition processing respectively. The AI glasses and the smart phone are efficiently cooperated, dynamic load balancing is achieved, and the real-time scene perception and processing capability in a mixed scene are improved, thereby achieving the technical effects of efficient cooperation of the AI glasses and the smart phone, dynamic load balancing, and improved real-time scene perception and energy efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments of the present disclosure will be briefly introduced below. In the present application, a flowchart is used to illustrate the operations performed by the platform according to the embodiments of the present application. It should be understood that the foregoing or the following operations are not necessarily performed in sequence. On the contrary, various steps can be processed in reverse order or simultaneously as needed. Meanwhile, other operations can be added to these processes, or a step or several steps can be removed from these processes.

[0015] Figure 1 The AI glasses and the smart phone data interaction system under the intelligent scene perception provided in the present application is shown in the structural schematic diagram.

[0016] Figure 2 The execution process of the discrimination module in the AI glasses and the smart phone data interaction system under the intelligent scene perception provided in the present application is shown in the structural schematic diagram.

[0017] Legend of reference signs: scene perception module 10, perception processing module 20, discrimination module 30, and interaction processing module 40. DETAILED DESCRIPTION

[0018] In order to further illustrate the technical means and effects adopted by the present application to achieve the predetermined invention purposes, the specific embodiments, structures, features and effects according to the present application are described in detail below with reference to the drawings and preferred embodiments.

[0019] The embodiment of the application provides an AI glasses and smart phone data interaction system under intelligent scene perception, which performs bidirectional data transmission of the AI glasses and the smart phone through a double-end communication link.

[0020] Preferably, the double-end communication link refers to a data transmission channel established between the AI glasses and the smart phone, which is used to perform bidirectional data transmission of the AI glasses and the smart phone, wherein the double-end communication link is implemented based on a short-distance wireless communication protocol, such as Bluetooth, Wi-Fi, etc., to ensure that data can be reliably and orderly transmitted; bidirectional transmission means that the AI glasses and the smart phone both send and receive data streams, and the data stream from the smart phone to the AI glasses can include smart phone initialization data, perception task instructions, predicted augmented / interactive content, etc.; the data stream from the AI glasses to the smart phone can include sensor data, scene feature data, resource state information of the AI glasses side and task execution result data, etc.; the AI glasses and the smart phone perform parallel processing according to task allocation and bidirectionally transmit necessary results, and finally present them to the user at the glasses side.

[0021] As shown in Figure 1 The AI glasses and smart phone data interaction system under intelligent scene perception comprises: A scene perception module 10 is used to read position data, direction data and attitude angle data of the AI glasses, perform scene data initialization of the smart phone and extract scene framework data.

[0022] Preferably, the three-dimensional coordinates read by the GPS, Wi-Fi positioning or Bluetooth beacon components built in the AI glasses are used as the position data for defining the position of the AI glasses in space; the orientation of the glasses provided by the magnetometer inside the AI glasses is used as the direction data of the AI glasses, that is, the azimuth angle of the camera embedded in the glasses; the attitude angle data of the AI glasses, such as the pitch angle, roll angle and yaw angle, are obtained by the inertial measurement unit built in the AI glasses, which are used to accurately describe the tilt and rotation state of the AI glasses in space. Then the smart phone receives the position data, direction data and attitude angle data of the AI glasses, performs scene data initialization of the smart phone, that is, loads and prepares the stored prior environment data in the memory, which usually includes pre-downloaded local map data such as indoor floor plan, building structure diagram or high-precision outdoor map, and environment semantic data containing object information with semantic labels bound to map coordinates, such as the door of A conference room, the coffee machine of B brand and the fire hydrant in C area. Then based on the initialization environment data of the smart phone and the real-time pose of the AI glasses, the smart phone performs data retrieval and simplifies the calculation to generate scene framework data.

[0023] Further, the specific configuration of the scene perception module 10 also includes establishing a spatial pose vector according to the position data, direction data, and attitude angle data; calling the local map and environmental semantic database of the smartphone end based on the spatial pose vector, retrieving the environmental semantic blocks within the current field of view; performing environmental complexity analysis of the environmental semantic blocks to establish the first extraction constraint; obtaining the precision setting of prior perception, and establishing the second extraction constraint according to the precision setting; after fusing the first extraction constraint and the second extraction constraint, performing feature compression and structure simplification of the environmental semantic blocks to establish the scene framework data composed of geometric contours and semantic label boundaries.

[0024] Preferably, the position data, direction data, and attitude angle data of the AI glasses are unified in coordinates and fused in data to integrate into a spatial pose vector to completely and accurately describe the three-dimensional position and three-dimensional orientation of the AI glasses in the global coordinate system; then the spatial pose vector is used as the camera parameter to perform view frustum culling calculation and retrieval in the local map data and environmental semantic data of the smartphone, and only retrieve a plurality of environmental semantic blocks that are currently within the possible range of the AI glasses lens field of view, i.e. all labeled objects or structures within the current theoretical field of view of the AI glasses and their three-dimensional data. Then, the environmental complexity analysis of the environmental semantic blocks is performed, i.e. quantitative analysis is performed on the retrieved plurality of environmental semantic blocks, including calculating the number, density, and type of objects within the current field of view, and generating a complexity index based on the current objective environment as the first extraction constraint, i.e. the higher the complexity, the greater the simplification; the lower the complexity, the more details that can be retained.

[0025] Preferably, the user's preset precision is obtained, such as high precision mode, balanced mode, or low power consumption mode, and the second extraction constraint is configured according to the user's preset precision and resource demand condition, representing the requirement for the degree of detail of the task, for example, the constraint corresponding to the high precision mode is minimization simplification, and the constraint corresponding to the low power consumption mode is maximization simplification; the first extraction constraint and the second extraction constraint are fused by weighted averaging to generate a simplification control instruction, and then feature compression and structure simplification of the environmental semantic blocks are performed according to the simplification control instruction. Specifically, for the geometric contour, a complex three-dimensional mesh is represented by fewer vertices and faces, such as replacing a complex computer host with a cube, or directly extracting the minimum bounding box of an object as its geometric contour; the semantic labels are merged or filtered, for example, in the simplification mode, the independent labels such as keyboard and mouse are ignored, and the overall label of the office table is retained; finally, the scene framework data composed of geometric contours and semantic label boundaries is obtained, wherein the geometric contour is a geometric body representing the spatial occupation range of the object after simplification, and the semantic label boundary is a semantic label closely bound to the geometric body, thereby ensuring that the smartphone delivers a large amount of data to the AI glasses with minimal data and efficiently guides the perception task of the AI glasses.

[0026] The perception processing module 20 is configured to activate the scene perception sensor of the AI glasses after the scene framework data is sent to the AI glasses through the two-end communication link, perform scene perception based on the scene framework data, and establish a low-dimensional scene state vector.

[0027] Preferably, the AI glasses receive the scene framework data of the smartphone through the two-end communication link and use the scene framework data as a control signal to trigger the power management or task scheduling unit of the AI glasses to switch the scene perception sensor in a low-power standby state to an active working mode as needed. Then, the scene perception sensor of the AI glasses performs scene perception using the scene framework data, including quickly comparing and matching the real-time image data obtained by the sensor with the scene framework data, spatially aligning various modal data, focusing on the image corresponding area based on the scene framework data and decoupling relevant features, fusing visual features, depth information, sound clues and other multi-dimensional semantics through a cross-modal attention mechanism and ensuring fusion accuracy, finally integrating the obtained multi-dimensional features into high-dimensional features, performing feature dimension reduction and projecting the high-dimensional features into a preset low-dimensional vector space, each vector corresponding to a specific state of the scene, and finally outputting a low-dimensional scene state vector, thereby ensuring scene perception efficiency and accuracy while reducing communication bandwidth requirements and power consumption.

[0028] Further, the specific configuration of the perception processing module 20 further includes verifying the authenticity and timeliness of the scene framework data and establishing a verification result; if the verification result is a pass result, directly activating the scene perception sensor, using the scene framework data as a prior template, performing modal alignment and feature decoupling of visual and acoustic sensing signals, using a cross-modal attention mechanism to perform semantic consistency fusion of multi-source perception information, and establishing a scene semantic feature set; performing feature dimension reduction and state embedding calculation on the scene semantic feature set, and projecting the scene semantic feature set into a low-dimensional scene state vector.

[0029] Preferably, the authenticity verification of the scene framework data is performed by quickly activating the scene perception sensor of the AI glasses to capture data and extract key features such as edges, corners, and significant semantic features, and quickly matching the key features with the geometric contours and semantic label boundaries in the scene framework data. If the matching degree is higher than a preset threshold, such as the position and shape of the object predicted by the framework data substantially coincide with the position and shape of the detected object in the image, the authenticity verification is passed, ensuring that the scene framework data is consistent with the current physical environment; then, the timeliness verification of the scene framework data is performed, that is, checking whether there is outdated dynamic object information in the scene framework data, for example, the scene framework data indicates that a chair is at this position, but the scene perception sensor of the AI glasses indicates that the position is currently empty. When the number and significance of such cases exceed a preset threshold, the timeliness verification fails, indicating that the environment has changed significantly since the scene framework data was generated; and then outputting a verification result, including a pass result or a fail result.

[0030] Preferably, if the verification result is a pass result, the scene-aware sensor is directly activated to perform modal alignment of visual and acoustic sensing signals using the scene framework data as a prior template. Specifically, the original data of sensors such as cameras and microphones is spatio-temporally aligned and registered to ensure that the image captured by the camera, the sound segment recorded by the microphone, and the pose of the glasses at each moment are under a unified coordinate, and then feature decoupling is performed, that is, for each region of interest marked by the scene framework data, independent features are extracted from different modal data, for example, for the screen region identified by the scene framework data as being in speech, features such as whether the screen content changes or whether there is lip movement are decoupled from the visual modal, and features such as the sound source direction and speech content are decoupled from the acoustic modal. Then, semantic consistency fusion is performed through a cross-modal attention mechanism. When the visual and acoustic features are highly correlated, that is, the lip speech and the voice are synchronous and homologous, a high attention weight is obtained, and then all modal features weighted by the attention weight are integrated to generate a scene semantic feature set containing high-dimensional feature representations of mutual verification and supplementary information between modalities. Then, principal component analysis or an autoencoder is used to reduce the dimension of the features to remove redundant information in the scene semantic feature set, retain the most discriminative core features, and project the reduced features into a pre-set low-dimensional embedding space. Semantic similar scene states are mapped to positions close to each other, and different scenes are mapped to positions far apart, and finally a low-dimensional scene state vector is output, thereby ensuring efficient intelligent scene awareness.

[0031] Further, the specific configuration of the perception processing module 20 further includes activating a verification collection instruction if the verification result is a verification failure result; controlling the scene-aware sensor to collect data according to the verification collection instruction to establish a collection data set; performing feature extraction of the scene framework data at the same position on the collection data set to perform feature verification comparison; using the feature verification comparison result as a framework template to perform linkage feature analysis of the collection data set to establish a scene semantic feature set, and establishing a low-dimensional scene state vector according to the scene semantic feature set.

[0032] If the verification result is a verification failure result, indicating that the scene framework data is unreliable, the verification collection instruction is activated, which is more comprehensive and energy-consuming than the instruction in the normal perception mode, and then the scene perception sensor is controlled according to the verification collection instruction to collect data at a higher resolution, more modalities or longer duration to obtain more complete original multi-modal sensor data such as image sequences, depth maps, point clouds, audio clips, etc., to determine the collection data set; then the same location feature extraction of the scene framework data is performed in the collection data set, that is, the visual features and depth features of the same location described by the failed scene framework data are determined and compared with the features in the scene framework data for feature verification comparison to obtain a feature verification comparison result, including pass / fail determination and specific difference report; then the feature verification comparison result is used as a framework template to perform linked feature analysis of the collection data set, including correlation perception in the collection data set, such as joint analysis of the visual appearance, three-dimensional shape and material of the region, focusing on multi-modal recognition of unknown objects and determining semantic relationships, and then outputting a scene semantic feature set, and finally performing feature dimension reduction and state embedding calculation on the scene semantic feature set to generate a low-dimensional scene state vector that accurately reflects the current situation of the environment.

[0033] The discrimination module 30 is configured to perform resource state perception of the AI glasses and the smart phone, and dynamically configure the perception task based on the resource state perception result and the low-dimensional scene state vector.

[0034] Further, as shown in Figure 2 The discrimination module 30 further includes the following specific configurations: the processor of the AI glasses collects the processor utilization rate, the battery capacity, and the sensor occupancy rate to establish a resource state description vector of the AI glasses; the smart phone collects the computing idle degree and the cache occupancy data to establish a resource state description vector of the smart phone; the AI glasses and the smart phone obtain the heat data, and a resource state penalty factor is established according to the heat data; after the resource state description vectors of the AI glasses and the smart phone are normalized, principal component extraction is performed to obtain the resource load feature vectors of the AI glasses and the smart phone, respectively; the change rate of the resource load feature vector is calculated by using a sliding window to establish a time sequence state matrix of the resource state trend, the time sequence state matrix is subjected to multi-scale fusion, and after being penalized and compensated by the resource state penalty factor, the resource state perception result is established.

[0035] Preferably, the AI glasses and the smart phone are monitored for resources respectively, and the processor utilization, battery capacity and sensor occupancy of the AI glasses are obtained, wherein the processor utilization refers to the usage percentage of CPU / GPU, reflecting the computing load; the battery capacity refers to the percentage of remaining power, reflecting the energy constraint; the sensor occupancy refers to whether the camera and other sensors are occupied by the current task or other tasks, reflecting the availability of hardware channels; then the resource state description vector of the AI glasses is generated by combination, which is used to describe the instantaneous static resource state of the AI glasses. The computing idle degree and cache occupancy data of the smart phone are collected, wherein the computing idle degree represents the available computing capacity of the phone, which is complementary to the processor utilization; the cache occupancy data refers to the occupancy of memory or special cache, reflecting the potential bottleneck of data processing; the resource state description vector of the smart phone is also constructed by combination.

[0036] Preferably, the temperature sensor is used to obtain the heating data of the AI glasses and the smart phone, and excessive temperature may cause the system to reduce the frequency, so that the actual performance is reduced even if other indicators seem good, and then the resource state penalty factor is established according to the heating data, which is positively correlated with the temperature, for example, the higher the temperature, the greater the resource state penalty factor, which is used to discount the effective performance of the AI glasses and the smart phone; the resource state description vector of the AI glasses and the smart phone is normalized, including mapping different dimensions and ranges of indicators to the same scale through minimum-maximum normalization or Z-score standardization, so that different indicators can be compared and weighted fused; then the principal component analysis is used to extract the principal components from the normalized resource state vector, that is, to extract the principal components that are independent of each other and can represent most of the resource state information from the multiple original indicators that may be correlated, and then the resource load feature vectors of the AI glasses and the smart phone are obtained respectively.

[0037] Preferably, a sliding window is preset, and the change rate of the resource load feature vector arranged in time sequence according to the sliding window is calculated, that is, the first derivative of the data principal component in each sliding window is calculated as the change rate, so as to quantify the change direction and speed of the resource load, for example, whether the CPU load is rapidly rising or slowly falling; then the resource load feature vector at the current time and its change rate vector in the sliding window are integrated into a time sequence state matrix, which contains the current state and recent trend; different weights are given to different time scale trends of the time sequence state matrix for multi-scale fusion, and then the resource state penalty factor is applied to the fused time sequence state matrix for penalty compensation, for example, the load value in the matrix is multiplied by the penalty factor, and finally the resource state perception result is generated.

[0038] Further, the specific configuration of the discrimination module 30 also includes clustering analysis on the low-dimensional scene state vector, which includes clustering analysis based on spatial position, target semantics, and dynamic characteristics, and establishes a multi-level clustering result; obtains the communication bandwidth, delay, and power consumption of the double-end communication link, and establishes a transmission penalty coefficient; uses the transmission penalty coefficient as a constraint term, and performs adaptive task adaptation analysis of the smartphone and the AI glasses using the resource state perception result and the multi-level clustering result, and establishes an adaptation analysis result; and completes the dynamic configuration of the perception task according to the adaptation analysis result.

[0039] Preferably, based on the spatial position, target semantics, and dynamic characteristics, the low-dimensional scene state vector representing the entire scene is subjected to clustering analysis, specifically, objects or regions in space that are adjacent to each other are clustered into one category for local centralized processing; objects that are similar in semantics are clustered into one category to call a dedicated recognition model; objects that are similar in motion state are clustered into one category to distinguish between static environments and dynamic targets; and further, sub-regions or sub-targets with similar characteristics are identified to obtain a multi-level clustering result, i.e., the low-dimensional scene is divided into multiple different subsets from different angles, for example, it is simultaneously divided into a static background cluster, a dynamic pedestrian cluster, a nearby desktop cluster, and a distant wall cluster. The communication bandwidth, delay, and power consumption of the double-end communication link are obtained, which respectively determine the rate of data transmission, the real-time performance of data transmission, and the energy cost of data transmission, and a transmission cost function is constructed according to the weighted combination of the communication bandwidth, delay, and power consumption, which is used to evaluate the transmission penalty coefficient. When the bandwidth is low, the delay is high, and the power consumption is large, the transmission penalty coefficient is high, i.e., the data transmission cost is high.

[0040] Preferably, the adaptive task adaptation analysis of the smartphone and the AI glasses is performed based on the resource state awareness result and the multi-level clustering result, wherein the transmission penalty coefficient quantifies the cost of communication, the resource state awareness result quantifies the computational cost and availability of the glasses and the smartphone, and the multi-level clustering result quantifies the characteristics and structure of the scene task itself; then the transmission penalty coefficient is taken as a constraint term, and the adaptive task adaptation analysis is performed based on a linear programming or heuristic optimization algorithm. Specifically, for each multi-level clustering result, the total cost of processing on the glasses and on the smartphone is respectively evaluated, wherein the glasses processing cost is determined based on the resource load of the glasses and the computational complexity of the task, and the smartphone processing cost is determined based on the resource load of the smartphone, the computational complexity of the task, and the product of the transmission penalty coefficient and the data volume of the task cluster; then the glasses processing cost and the smartphone processing cost are compared, and the processing end with lower total cost is selected for each level clustering result, and finally the adaptation analysis result is determined, including an explicit task allocation mapping table, and specific configuration instructions are generated according to the adaptation analysis result and sent to the AI glasses end or the smartphone end, wherein the static background and specific pedestrian tracking are processed locally by the AI glasses, and complex character recognition and large-scale semantic segmentation are processed by the smartphone, ensuring that the perception task is dynamically optimally configured to the AI glasses and the smartphone.

[0041] The interaction processing module 40 is configured to segment the low-dimensional scene state vector according to the perception task, pack the low-dimensional scene state vector identified by the smartphone, and upload it to the smartphone through the double-end communication link, and perform scene recognition processing by the smartphone and the AI glasses respectively.

[0042] Preferably, the low-dimensional scene state vector is logically segmented based on the dynamic configuration result of the perception task, including identifying the task cluster in the low-dimensional scene state vector that needs to be processed by the smartphone, packing the low-dimensional scene state vector identified by the smartphone into a data packet, including data sequence number, task ID, timestamp, etc., and uploading it to the smartphone through the double-end communication link, and then the AI glasses and the smartphone work in parallel according to the allocated tasks to perform scene recognition processing. Specifically, the smartphone receives the data, reads and parses the corresponding low-dimensional scene state vector, calls the artificial intelligence model to perform deep analysis and high-precision recognition on these vectors, which may include complex object recognition, scene semantic segmentation, or information retrieval by accessing the cloud database, and generates high-precision scene recognition results on the smartphone; at the same time, the AI glasses locally process the part of the low-dimensional scene state vector that is not uploaded, including performing tasks with high real-time requirements and relatively small computational complexity, such as basic object tracking, spatial plane detection, or simple gesture recognition, and generating fast and lightweight scene recognition results on the AI glasses end, thereby ensuring that computationally intensive tasks are executed on the smartphone end and real-time tasks are executed on the AI glasses end, and improving the real-time performance and energy efficiency of scene recognition perception.

[0043] Further, the AI glasses and smart phone data interaction system under the intelligent scene perception further comprises a perception enhancement module, which is configured to perform active gaze perception of a user based on the AI glasses, establish a focus meta event, send the focus meta event to a smart phone, establish scene focus, perform scene trigger analysis of a scene recognition result based on the scene focus, establish enhanced content, and return the enhanced content to the AI glasses and display the enhanced content.

[0044] Preferably, the perception enhancement module realizes active visual enhancement cooperation by introducing a scene trigger. The AI glasses monitor the user's eye movement in real time by using an internal eye movement tracking sensor, accurately calculate the user's gaze point and gaze duration, generate a structured focus meta event when detecting that the user continuously gazes at a specific object or area for more than 3 seconds, and contain the spatial coordinates of the gaze target, the start time and duration of the gaze, and the preliminary identification of the gazed object. The structured meta event is sent to the smart phone through a double-end communication link. The smart phone fuses it with the current scene recognition result being processed to clearly identify the focus of the user's current interest. Then, based on the focus meta event and scene semantics, the smart phone performs intelligent push or enhanced content return, such as displaying detailed instructions, AR labels, instant translation results, etc. Finally, the generated enhanced content is returned to the AI glasses. The display driving component of the AI glasses receives the enhanced content and accurately maps the virtual enhanced content to the real object gazed at by the user, and displays the virtual enhanced content through the optical lens, realizing the fusion of virtual and reality.

[0045] Further, the AI glasses and smart phone data interaction system under the intelligent scene perception further comprises a short-time scene prediction module, which is configured to obtain a preset resource library of a user, perform short-time scene prediction based on a scene recognition result and the preset resource library, establish predicted interaction content, and push the predicted interaction content to the AI glasses for display.

[0046] Preferably, the short-time scene prediction module reads a preset resource library, i.e., a digital user model and a content database, from the storage space of the smartphone, which can include operations frequently performed or information frequently viewed by the user in a specific scene, personal preference settings of the user, pre-associated digital content, and a scene-action rule library; then based on the scene recognition result and the preset resource library, the short-time scene prediction of the environment where the AI glasses are located is performed through the computing power resources of the mobile phone, i.e., the rule-based reasoning engine matches and analyzes the current scene recognition result with the preset resource library, outputs the short-time scene prediction result, and determines the predicted interaction content, for example, the mobile phone predicts that the user will enter a meeting scene soon, and pushes the meeting schedule to the AI glasses in advance for display; when the AI glasses detect scene switching signals such as acoustic echo, sudden change of illumination, etc., the pre-fetch content is quickly activated for seamless switching, and the predicted interaction content is pushed to the graphic cache of the AI glasses through the double-end communication link, so as to realize the predictive preloading and rendering engine of the zero-waiting AR experience, thereby improving the fluency of the interaction system.

[0047] The above is only a preferred embodiment of the present application, and does not limit the present application in any form. Although the present application has been disclosed as above with a preferred embodiment, it is not intended to limit the present application. Any person skilled in the art can make some changes or modifications to the above disclosed technical content to obtain equivalent embodiments with equivalent changes, without departing from the scope of the technical solution of the present application. Any modification, equivalent change and modification of the above embodiments according to the technical essence of the present application are still within the scope of the technical solution of the present application.

Claims

1. An AI glasses and smart phone data interaction system under intelligent scene perception, characterized in that, The system performs bidirectional data transmission of the AI glasses and the smart phone through a double-end communication link, and the system comprises: a scene perception module, which reads position data, direction data and attitude angle data of the AI glasses, performs scene data initialization of the smart phone, and extracts scene framework data; a perception processing module, which activates scene perception sensors of the AI glasses after the scene framework data is sent to the AI glasses through the double-end communication link, performs scene perception based on the scene framework data, and establishes a low-dimensional scene state vector; a discrimination module, which performs resource state perception of the AI glasses and the smart phone, and dynamically configures a perception task based on a resource state perception result and the low-dimensional scene state vector; an interaction processing module, which segments the low-dimensional scene state vector according to the perception task, packs the low-dimensional scene state vector identified by the smart phone, and uploads the low-dimensional scene state vector to the smart phone through the double-end communication link, and performs scene recognition processing by using the smart phone and the AI glasses respectively.

2. The AI glasses and smart phone data interaction system under intelligent scene awareness according to claim 1, wherein, In the discrimination module, the resource state perception of the AI glasses and the smart phone comprises: collecting processor utilization, battery capacity and sensor occupancy of the AI glasses to establish a resource state description vector of the AI glasses; collecting calculation idle degree and cache occupancy data of the smart phone to establish a resource state description vector of the smart phone; obtaining heating data of the AI glasses and the smart phone, and establishing a resource state penalty factor according to the heating data; performing principal component extraction after normalizing the resource state description vectors of the AI glasses and the smart phone to obtain resource load feature vectors of the AI glasses and the smart phone respectively; calculating a change rate of the resource load feature vectors by using a sliding window to establish a time sequence state matrix of a resource state trend, performing multi-scale fusion on the time sequence state matrix, and performing penalty compensation on the time sequence state matrix through the resource state penalty factor to establish a resource state perception result.

3. The AI glasses and smart phone data interaction system under intelligent scene awareness according to claim 2, wherein, In the discrimination module, the dynamic configuration of the perception task based on the resource state perception result and the low-dimensional scene state vector comprises: performing cluster analysis on the low-dimensional scene state vector, which comprises cluster analysis based on spatial position, target semantics and dynamic characteristics to establish a multi-level cluster result; obtaining communication bandwidth, delay and power consumption of the double-end communication link to establish a transmission penalty coefficient; using the resource state perception result and the multi-level cluster result to perform adaptive task adaptation analysis of the smart phone and the AI glasses by taking the transmission penalty coefficient as a constraint term to establish an adaptation analysis result; dynamically configuring the perception task according to the adaptation analysis result.

4. The AI glasses and smart phone data interaction system under intelligent scene awareness according to claim 1, wherein, In the scene perception module, the extraction of the scene framework data comprises: establishing a spatial pose vector according to the position data, the direction data and the attitude angle data; calling a local map and an environment semantic database of the smart phone based on the spatial pose vector to retrieve an environment semantic block within a current field of view; performing environment complexity analysis of the environment semantic block to establish a first extraction constraint; obtaining a precision setting of prior perception to establish a second extraction constraint according to the precision setting; After the first extraction constraint and the second extraction constraint are fused, feature compression and structure simplification of the environmental semantic block are performed to establish a scene framework data composed of geometric contours and semantic label boundaries.

5. The AI glasses and smart phone data interaction system under intelligent scene awareness according to claim 1, wherein, In the perception processing module, the scene perception sensor of the AI glasses is activated, scene perception is performed based on the scene framework data, a low-dimensional scene state vector is established, and the low-dimensional scene state vector includes: The scene framework data is verified for authenticity and timeliness, and a verification result is established. If the verification result is a pass result, the scene perception sensor is directly activated, the scene framework data is used as a prior template, modal alignment and feature decoupling of visual and acoustic sensing signals are performed, semantic consistency fusion of multi-source perception information is performed using a cross-modal attention mechanism, and a scene semantic feature set is established. Feature dimension reduction and state embedding calculation are performed on the scene semantic feature set, and the scene semantic feature set is projected into a low-dimensional scene state vector.

6. The AI glasses and smart phone data interaction system under intelligent scene awareness according to claim 5, wherein, In the perception processing module, the verification result is established, and the verification result further includes: If the verification result is a verification failure result, a verification collection instruction is activated. The scene perception sensor is controlled to collect data according to the verification collection instruction, and a collection data set is established. The same position feature extraction of the scene framework data is performed on the collection data set, and feature verification comparison is performed. Using the feature verification comparison result as a framework template, the collection data set is subjected to linkage feature analysis, a scene semantic feature set is established, and a low-dimensional scene state vector is established based on the scene semantic feature set.

7. The AI glasses and smart phone data interaction system under intelligent scene awareness according to claim 1, wherein, The system further includes: A perception enhancement module is configured to perform active gaze perception of a user based on the AI glasses, establish a focus meta event, send the focus meta event to a smartphone, establish scene focus, perform scene trigger analysis of a scene recognition result based on the scene focus, establish enhanced content, and return the enhanced content to the AI glasses and display the enhanced content.

8. The AI glasses and smart phone data interaction system under intelligent scene awareness according to claim 1, wherein, The system further includes: A short-time scene prediction module is configured to obtain a preset resource library of a user, perform short-time scene prediction based on a scene recognition result and the preset resource library, establish predicted interactive content, and push the predicted interactive content to the AI glasses for display.

Images