Control method, apparatus, computing device, and system for smart glasses

Abstract

The application relates to a control method and device of intelligent glasses, a computing device and a system. The method comprises the following steps: the intelligent glasses acquire a voice instruction issued by a user, and convert the voice instruction into a target text instruction; the intelligent glasses determine a target interface screenshot, and send the target interface screenshot and the target text instruction to a cloud server after encryption; the cloud server decrypts the encrypted transmission information, determines the target interface screenshot and the target text instruction, and determines an operation step according to the target interface screenshot and the target text instruction through a cloud multimodal large model; the cloud server assembles the operation step into a standardized instruction sequence, and sends the standardized instruction sequence to the intelligent glasses; and the intelligent glasses determine instruction operation information according to the standardized instruction sequence, and execute the instruction operation information. The above scheme can improve the intelligent degree and adaptability of an AR glass control method.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a control method, apparatus, computing device, and system for smart glasses. Background Technology

[0002] With the increasing functionality of AR (Augmented Reality) wearable devices and the growing user demand, how to achieve automated control of AR smart glasses through intelligent means has become a hot topic in the industry. For example, existing AR glasses typically operate through fixed voice commands or simple gestures, and some remote collaboration applications and translation functions are also beginning to appear on AR platforms. However, existing AR glasses control solutions still have many shortcomings in practical applications. For example, many existing solutions require directly uploading voice commands or camera images collected by the AR glasses to cloud servers for processing. Network round-trip communication and cloud computing often cause significant response delays, failing to meet the real-time requirements of AR scenarios. Existing technologies often require uploading sensitive data such as user voice and visual images to the cloud for analysis and processing, which may lead to privacy risks if transmission and storage are not handled properly. Many AR devices currently interact based on pre-set fixed command sets or simple logic, and the devices cannot truly "understand" the user's intentions and the content of the environment. Traditional voice assistants typically only perform limited functions based on voice keywords, lacking the utilization of camera visual information, thus limiting their decision-making intelligence. The system struggles to handle freely expressed commands and the associative understanding of visual scenes in complex environments, resulting in a low level of intelligence in interaction. Existing solutions are often customized for specific applications or scenarios, lacking versatility. Therefore, reducing response latency for AR glasses, protecting user privacy, improving the intelligence level of AR glasses, and enhancing their environmental adaptability are problems that need to be addressed. Summary of the Invention

[0003] Therefore, it is necessary to provide a control method, device, computing device, and system for smart glasses that can reduce the response latency of AR glasses, protect the privacy of AR glasses users, improve the intelligence level of AR glasses, and enhance the environmental adaptability of AR glasses, in order to address the above-mentioned technical problems.

[0004] In a first aspect, this application provides a control method for smart glasses, wherein the control method is implemented by a smart glasses control system, the smart glasses control system including smart glasses and a cloud server, and the method includes:

[0005] The smart glasses acquire the voice commands issued by the user and convert the voice commands into target text commands;

[0006] The smart glasses identify a screenshot of the target interface, encrypt the screenshot and the target text command, determine the encrypted transmission information, and send the encrypted transmission information to the cloud server.

[0007] The cloud server decrypts the encrypted transmission information, determines the target interface screenshot and the target text instruction, and uses a cloud-based multimodal big data model to determine the operation steps required to realize the user's intent based on the target interface screenshot and the target text instruction.

[0008] The cloud server assembles the operation steps into a standardized instruction sequence and sends the standardized instruction sequence to the smart glasses;

[0009] The smart glasses determine the instruction operation information based on the standardized instruction sequence and execute the instruction operation information.

[0010] In one embodiment, the training method for the cloud-based multimodal large model includes:

[0011] The cloud server combines pre-trained visual language models, text perceptrons, image perceptrons, and spatial perceptrons to determine a multimodal fusion model;

[0012] Using the sample interface screenshots of the smart glasses and the sample description text corresponding to the sample interface screenshots, the multimodal fusion model is trained to determine the first multimodal model to be trained.

[0013] Based on the sample interface screenshot, the sample annotation information in the sample interface screenshot, and the sample interaction elements corresponding to the sample interface screenshot, the first multimodal model to be trained is trained to determine the second multimodal model to be trained; the sample annotation information includes sample prompts, sample icons, and sample virtual controls in the sample interface screenshot.

[0014] Based on the sample interface screenshot, sample instruction sequence, and sample text instruction corresponding to the sample interface screenshot, the second multimodal model to be trained is trained to determine the large multimodal model in the cloud.

[0015] In one embodiment, the second multimodal model to be trained is trained based on the sample interface screenshot, the sample command sequence, and the sample text command corresponding to the sample interface screenshot, to determine the large multimodal model in the cloud, including:

[0016] Based on the sample interface screenshot, sample command sequence, and sample text command corresponding to the sample interface screenshot, the second multimodal model to be trained is trained to determine the candidate multimodal large model;

[0017] The candidate multimodal large model is reinforced and trained using a group relative strategy optimization algorithm to determine the cloud-based multimodal large model.

[0018] In one embodiment, the candidate multimodal large model is reinforced and trained using a group relative policy optimization algorithm to determine the cloud-based multimodal large model, including:

[0019] The candidate multimodal large model is used to perform a model testing task. A population relative strategy optimization algorithm is used to determine the reward function based on the task execution result of the candidate multimodal large model.

[0020] The model reward score is determined based on the reward function using the group relative strategy optimization algorithm, and the model parameters of the candidate multimodal large model are optimized based on the model reward score to determine the cloud-based multimodal large model.

[0021] In one embodiment, using a cloud-based multimodal large model, the operation steps required to achieve the user's intent are determined based on the target interface screenshot and the target text instruction, including:

[0022] The target interface screenshot and the target text command are input into the cloud-based multimodal large model. The target prompt words in the target interface screenshot are extracted by the text perceptron in the cloud-based multimodal large model.

[0023] The target icon and target virtual control in the target interface screenshot are determined by the graphics perceptron in the cloud multimodal large model.

[0024] The spatial relationship between the target icon and the target virtual control is determined by the spatial perceptron in the cloud-based multimodal large model.

[0025] Using the target prompts, target icons, and target virtual controls as target interactive elements, and through the visual language model in the cloud-based multimodal large model, the operation steps required to realize the user's intent are determined based on the target spatial relationships, the target interactive elements, and the target text instructions.

[0026] In one embodiment, the control method for the smart glasses further includes:

[0027] When executing the instruction operation information, the smart glasses use a dynamic programming algorithm to determine instruction correction information based on the target text instruction, the historical task execution data of the sub-tasks already executed during the execution of the instruction operation information, and the real-time interface screenshot of the smart glasses after executing the sub-tasks.

[0028] From the standardized instruction sequence, determine the instruction sequence to be adjusted corresponding to the subtask that was not executed in the instruction operation information;

[0029] Based on the instruction correction information, the instruction sequence to be adjusted is adjusted.

[0030] In one embodiment, the control method for the smart glasses further includes:

[0031] After the execution of the instruction operation information is completed, the smart glasses generate operation result feedback information, and send the operation result feedback information, the intermediate data generated when executing the instruction operation information, the screenshot of the operation interface corresponding to the intermediate data, and the screenshot of the task completion interface of the smart glasses after the execution of the instruction operation information to the cloud server.

[0032] The cloud server retrains the cloud-based multimodal large model based on the operation result feedback information, intermediate data, operation interface screenshots, and task completion interface screenshots.

[0033] Secondly, this application also provides a control device for smart glasses, the device comprising:

[0034] A voice conversion module, deployed in smart glasses, is used to acquire voice commands issued by the user and convert the voice commands into target text commands;

[0035] A screenshot acquisition module is deployed in smart glasses to determine a target screen screenshot, encrypt the target screen screenshot and the target text command, determine the encrypted transmission information, and send the encrypted transmission information to a cloud server.

[0036] The cloud-based large model inference module, deployed on a cloud server, is used to decrypt the encrypted transmission information, determine the target interface screenshot and the target text instruction, and determine the operation steps required to realize the user's intent based on the target interface screenshot and the target text instruction through the cloud-based multimodal large model.

[0037] An action instruction parsing module, deployed on a cloud server, is used to assemble the operation steps into a standardized instruction sequence and send the standardized instruction sequence to the smart glasses.

[0038] An instruction execution module, deployed in smart glasses, is used to determine instruction operation information based on the standardized instruction sequence and execute the instruction operation information.

[0039] Thirdly, this application also provides a computing device, including a memory and a processor. The memory stores a computer program. When the computing device is a smart glasses, it performs the following steps: acquiring a voice command issued by a user and converting the voice command into a target text command; determining a target interface screenshot and encrypting the target interface screenshot and the target text command, determining encrypted transmission information, and sending the encrypted transmission information to a cloud server; determining instruction operation information based on a standardized instruction sequence fed back by the cloud server, and executing the instruction operation information.

[0040] When the computing device is a cloud server, the following steps are performed: decrypt the encrypted transmission information, determine the target interface screenshot and the target text instruction, and determine the operation steps required to realize the user's intent based on the target interface screenshot and the target text instruction through a cloud-based multimodal big data model; assemble the operation steps into a standardized instruction sequence, and send the standardized instruction sequence to the smart glasses.

[0041] Fourthly, this application also provides a control system for smart glasses, which includes smart glasses and a cloud server.

[0042] The aforementioned control method, device, computing device, and system for smart glasses involve the smart glasses acquiring voice commands from the user and converting them into target text commands. The smart glasses then determine a target interface screenshot, encrypt both the screenshot and the target text command, determine encrypted transmission information, and send this encrypted transmission information to a cloud server. The cloud server decrypts the encrypted transmission information, determines the target interface screenshot and the target text command, and, using a cloud-based multimodal large model, determines the operation steps required to achieve the user's intent based on the target interface screenshot and the target text command. The cloud server assembles these operation steps into a standardized command sequence and sends the standardized command sequence to the smart glasses. The smart glasses determine the command operation information based on the standardized command sequence and execute the command operation information. This solves the problems of high response latency, high privacy risks, low intelligence, and poor adaptability in current AR device automated control methods. The above solution constructs an intelligent agent system that tightly coordinates AR glasses and a cloud server. The AR glasses quickly process user voice commands through voice recognition technology, converting them into target text commands and uploading them to the cloud server. This eliminates the need to upload audio to the cloud server, reducing interaction latency between the AR glasses and the cloud server. Encrypting the target text commands and target interface screenshots before sending them to the cloud server protects user privacy. Simultaneously, using both target text commands and target interface screenshots as input data for a multimodal large-scale model provides a clear view of the AR glasses' real-time environment and virtual interface state during command execution. The cloud-based multimodal large-scale model deployed on the cloud server analyzes the target interface screenshots and target text commands to determine the necessary steps to achieve the user's intent, improving the intelligence of AR glasses control. Determining the necessary steps through the multimodal large-scale model allows for analysis of target interface screenshots and target text commands based on the massive training data of the large-scale model, thereby enhancing the environmental adaptability of the AR glasses control method. Assembling the operation steps into a standardized command sequence before feeding it back to the AR glasses further reduces interaction latency between the cloud server and the AR glasses. Attached Figure Description

[0043] Figure 1 This is a flowchart illustrating a control method for smart glasses in one embodiment;

[0044] Figure 2 This is a flowchart illustrating a cloud-based multimodal large model training method in one embodiment;

[0045] Figure 3 This is a flowchart illustrating a method for determining operation steps in one embodiment;

[0046] Figure 4 This is a structural block diagram of the control device for smart glasses in one embodiment;

[0047] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0049] In one embodiment, such as Figure 1 As shown, a control method for smart glasses is provided. This control method is implemented by a smart glasses control system, which includes smart glasses and a cloud server. In this embodiment, the method includes the following steps:

[0050] S110: The smart glasses acquire the user's voice commands and convert them into target text commands.

[0051] Among them, smart glasses, also known as AR glasses, refer to glasses that function as smart terminal devices. AR glasses are wearable computing devices equipped with cameras, microphones, display modules, and local computing chips. AR glasses can overlay digital content onto the user's real-world field of vision, achieving an interactive experience that combines the virtual and real worlds. The display module can be a micro-projector or a waveguide lens.

[0052] Specifically, the process involves acquiring voice commands input from the user's microphone via AR glasses, and then converting these voice commands into text-based commands (target text commands) using speech recognition and audio encoding technologies. Methods for converting voice commands into target text commands can include: voice command preprocessing, which involves noise reduction and echo cancellation to improve audio quality and thus increase the accuracy of speech recognition; feature extraction from the preprocessed voice commands to determine speech features, which can be Mel-frequency cepstral coefficients or linear prediction coefficients of the preprocessed voice commands; and finally, a speech recognition algorithm to determine the target text command based on the speech features. This speech recognition algorithm can be a pre-trained deep learning algorithm or a pre-trained convolutional neural network.

[0053] For example, if a user says to the glasses, "Connect to my supervisor for remote assistance," the AR glasses can use the voice recognition module to recognize the user's voice command as the text "Connect to my supervisor for remote assistance."

[0054] It should be noted that converting voice commands into target text commands and then transmitting them to the cloud server can improve command transmission efficiency and reduce the interaction latency between smart glasses and the cloud server.

[0055] The S120 smart glasses determine the target interface screenshot, encrypt the target interface screenshot and target text command, determine the encrypted transmission information, and send the encrypted transmission information to the cloud server.

[0056] Specifically, the AR glasses use a camera to capture images of the user's surroundings, combine these images with the virtual interface elements of the AR glasses to determine a screenshot of the target interface. The target interface screenshot and the target text commands are then encrypted to determine the encrypted transmission information, which is then sent to the cloud server.

[0057] For example, a key server can pre-generate a key pair according to the protocol between the AR glasses and the cloud server. The public key of the key pair is distributed to the AR glasses, and the private key is allocated to the cloud server. The AR glasses use the public key to encrypt the target interface screenshot and the target text command, thus determining the encrypted transmission information. After obtaining the encrypted transmission information, the cloud server can decrypt it using the private key to obtain the target interface screenshot and the target text command.

[0058] For example, AR glasses capture the image of the scene seen by the user through AR glasses as the target interface screenshot. If the AR glasses are in the function menu virtual interface when the screenshot is taken, and the virtual interface contains the remote collaboration application icon, or the user can see the user interface prompt area, then the obtained target interface screenshot contains the user interface prompt area, the remote collaboration application icon, and the function menu virtual interface. The target interface screenshot also contains the layout of the virtual controls in the function menu virtual interface and key information about the user's surrounding scene.

[0059] It's important to note that directly encrypting voice commands is costly. Therefore, when controlling AR glasses directly via voice commands, these commands are often sent directly to the cloud server without encryption, posing a privacy risk. Encrypting target text commands, on the other hand, is less costly and faster than encrypting voice commands. Therefore, encrypting target text commands further reduces latency between the smart glasses and the cloud server.

[0060] S130: The cloud server decrypts the encrypted transmission information, determines the target interface screenshot and target text command, and uses the cloud multimodal big model to determine the operation steps required to realize the user's intent based on the target interface screenshot and target text command.

[0061] The steps required to achieve the user's intent refer to the steps that the AR glasses need to complete, determined based on the target text instruction and the target interface screenshot. For example, if the target text instruction is "click," then the steps required to achieve the user's intent are: to click the virtual control corresponding to the click operation on the AR glasses' virtual interface.

[0062] S140 and the cloud server assemble the operation steps into a standardized instruction sequence and send the standardized instruction sequence to the smart glasses.

[0063] The standardized instruction sequence is a standardized JSON instruction sequence. This sequence defines a series of standardized operations to be executed sequentially, such as clicking virtual buttons, swiping views, entering text, waiting, and displaying information. A normalized relative coordinate system ranging from 0 to 1000 is used to represent the virtual interface of the AR glasses or the target location within the user's field of vision where the operation needs to be performed.

[0064] Specifically, after the cloud server determines the operation steps, it converts the data corresponding to the operation steps into a JSON format string, thereby determining a standardized instruction sequence. This standardized instruction sequence is then sent to the smart glasses.

[0065] S150: The smart glasses determine the instruction operation information according to the standardized instruction sequence and execute the instruction operation information.

[0066] The instruction operation information includes the instruction operation to be executed and the target location corresponding to the instruction operation. The target location is located on the virtual interface of the AR glasses.

[0067] Specifically, the smart glasses are equipped with an action command parsing module, which parses standardized command sequences in JSON format. This module determines the command operation information based on the standardized command sequence and calls the virtual control interface or graphical interaction control layer of the AR glasses system line by line to execute the corresponding command operation information, thereby converting the standardized JSON command sequence into actual interface interaction behavior.

[0068] It's important to note that virtual control interfaces are a technique used in automated testing to define and manipulate areas of the interface that cannot be identified through conventional methods. By encapsulating specific interface areas as virtual controls, operations such as clicks, text recognition, and image comparison can be implemented, making it suitable for scenarios involving custom buttons and complex interface elements. The graphical user interface (GUI) control layer is the controller layer in the MVC (Model-View-Controller) architecture. It is responsible for handling the interaction logic between the user and the graphical interface, receiving user input commands, processing data in the model layer, and ultimately updating the view layer to display the results.

[0069] In the aforementioned control method for smart glasses, the smart glasses acquire voice commands issued by the user and convert them into target text commands. The smart glasses then determine a target interface screenshot, encrypt both the screenshot and the text command, and send this encrypted transmission information to a cloud server. The cloud server decrypts the encrypted transmission information, identifies the target interface screenshot and the text command, and, using a cloud-based multimodal big data model, determines the necessary operation steps to achieve the user's intent based on the screenshot and command. The cloud server assembles these operation steps into a standardized command sequence and sends it to the smart glasses. The smart glasses then determine and execute the command operation information based on the standardized command sequence. This method solves the problems of high response latency, high privacy risks, low intelligence, and poor adaptability in current AR device automation control methods. The above solution constructs an intelligent agent system that tightly coordinates AR glasses and a cloud server. The AR glasses quickly process user voice commands through voice recognition technology, converting them into target text commands and uploading them to the cloud server. This eliminates the need to upload audio to the cloud server, reducing interaction latency between the AR glasses and the cloud server. Encrypting the target text commands and target interface screenshots before sending them to the cloud server protects user privacy. Simultaneously, using both target text commands and target interface screenshots as input data for a multimodal large-scale model provides a clear view of the AR glasses' real-time environment and virtual interface state during command execution. The cloud-based multimodal large-scale model deployed on the cloud server analyzes the target interface screenshots and target text commands to determine the necessary steps to achieve the user's intent, improving the intelligence of AR glasses control. Determining the necessary steps through the multimodal large-scale model allows for analysis of target interface screenshots and target text commands based on the massive training data of the large-scale model, thereby enhancing the environmental adaptability of the AR glasses control method. Assembling the operation steps into a standardized command sequence before feeding it back to the AR glasses further reduces interaction latency between the cloud server and the AR glasses.

[0070] In one embodiment, such as Figure 2 As shown, the training methods for cloud-based multimodal large models include:

[0071] The S210 cloud server combines pre-trained visual language models, text perceptrons, image perceptrons, and spatial perceptrons to determine a multimodal fusion model.

[0072] The visual language model comprises two main components: the large language model and the visual encoder. The large language model, trained using deep learning techniques, learns the patterns of human language by analyzing massive amounts of text data, enabling it to generate or understand natural language text. The visual encoder is a core component that transforms images or videos into machine-understandable numerical data. It primarily connects image and text understanding, allowing the language model to recognize features such as themes, colors, and locations within images, and to perform complex reasoning and interactions. The text perceptron is a perceptron model used in artificial intelligence to process text data, belonging to the sensory organs of AI. It can understand and process textual information, extracting key features by analyzing text content to provide data support for subsequent decisions or tasks. The image perceptron is a fundamental binary classification model in machine learning, simulating the structure of biological neurons to make linear classification decisions. It classifies input features by adjusting weights and thresholds, primarily used to handle linearly separable problems. The spatial perceptron is a fusion technology combining spatial perception and the concept of a perceptron, mainly referring to a technical component that acquires environmental data through sensors and transforms it into usable information. A spatial perceptron needs to possess three-dimensional spatial environment perception capabilities, including locating its own position, recognizing surrounding objects, building environmental models, and planning movement paths. Its core is to acquire environmental data through sensors and process it using algorithms to generate executable instructions. The constructed multimodal fusion model has the ability to process target interface screenshots and target text instructions.

[0073] S220. Using screenshots of the smart glasses' sample interface and corresponding sample description text, the multimodal fusion model is trained to determine the first multimodal model to be trained.

[0074] The sample interface screenshots can be randomly selected screenshots of the virtual interface generated during the historical operation of AR glasses. The corresponding sample description text can be image description information added manually based on the sample interface screenshots. For example, the sample description text can include descriptions of UI elements in the sample interface screenshots and descriptions of the interface structure of the virtual interface. The first multimodal model to be trained can determine the image description text corresponding to the input image based on the input image. UI elements refer to visual elements used in interface design, such as a five-pointed star representing a collection in the virtual interface.

[0075] It should be noted that by using screenshots of the smart glasses' interface and the corresponding sample description text, the multimodal fusion model is trained. The trained multimodal fusion model is used as the first multimodal model to be trained. This allows the visual language model in the multimodal fusion model to be trained, enabling the first multimodal model to extract visual content from the graphical user interface and to understand image semantics.

[0076] S230. Based on the sample interface screenshot, the sample annotation information in the sample interface screenshot, and the sample interaction elements corresponding to the sample interface screenshot, train the first multimodal model to be trained and determine the second multimodal model to be trained.

[0077] The sample annotation information includes sample prompts, sample icons, sample virtual controls, and the position coordinates of the sample icons, sample virtual controls, and sample prompts in the sample interface screenshot. Sample interactive elements refer to the operational functions shown in the sample interface screenshot, such as, but not limited to, swiping up, swiping down, and clicking. Sample prompts are the prompts in the sample interface screenshot, serving as the command carrier for users interacting with the augmented reality system. They express task requirements in a structured way, guiding the system to provide operational instructions or information. Sample icons can include UI elements in the sample interface screenshot, or road icons in the user's surrounding environment. Sample prompts in the sample interface screenshot can be identified using optical character recognition (OCR) technology.

[0078] Specifically, the first multimodal model to be trained is trained based on sample interface screenshots of the smart glasses and the sample annotation information of the sample interface screenshots. This enables the trained first multimodal model to locate and recognize various virtual controls, icons, and prompts in the virtual interface of the AR glasses. Then, the trained first multimodal model is trained again based on sample interface screenshots and the corresponding sample interaction elements to determine the second multimodal model to be trained. This second multimodal model enables the second multimodal model to recognize elements in the graphical user interface of the AR glasses.

[0079] S240. Based on the sample interface screenshot, sample instruction sequence, and sample text instruction corresponding to the sample interface screenshot, train the second multimodal model to be trained and determine the large multimodal model in the cloud.

[0080] Specifically, supervised model training is performed on the second multimodal model to be trained based on sample interface screenshots, sample command sequences, and corresponding sample text commands. The sample command sequences serve as the supervised data. The sample command sequences can be standardized JSON-formatted commands generated during historical AR glasses control processes collected manually, while the sample text commands can be converted from user commands issued during historical AR glasses control processes. The sample interface screenshots, sample command sequences, and corresponding sample text commands can be used as the sample dataset. This dataset is divided into a model training dataset and a model testing dataset. The model training dataset is used to train the second multimodal model to be trained, and the model testing dataset is used to test the trained model. The model loss function, which can be the cross-entropy loss function, is calculated based on the predicted command sequences output by the trained second multimodal model during model testing and the sample command sequences in the model testing dataset. The training status of the second multimodal model to be trained is determined based on the model loss function. If the training is completed, the trained second multimodal model to be trained is used as the cloud-based multimodal large model. If the training is not completed, the second multimodal model to be trained is iteratively trained to update its model parameters.

[0081] The above solution provides a method for training a multimodal fusion model to determine a large cloud-based multimodal model. This method enables the trained large cloud-based multimodal model to locate and identify virtual controls, icons, and prompts in the virtual interface of AR glasses, recognize elements in the graphical user interface of AR glasses, analyze complex virtual interfaces containing real-world scenes, and plan tasks based on the virtual interface and target text instructions of AR glasses. This provides accurate semantic description information for subsequent decision-making.

[0082] In one embodiment, the second multimodal model to be trained is trained based on a sample interface screenshot, a sample command sequence, and the sample text command corresponding to the sample interface screenshot, to determine a large multimodal model in the cloud, including:

[0083] Based on the sample interface screenshots, sample command sequences, and sample text commands corresponding to the sample interface screenshots, the second multimodal model to be trained is trained to determine the candidate multimodal large model; the candidate multimodal large model is reinforced and trained using a population relative strategy optimization algorithm to determine the cloud-based multimodal large model.

[0084] Among them, group relative policy optimization is a reinforcement learning algorithm that aims to improve the performance of large language models in complex reasoning tasks.

[0085] The above solution can further enhance the decision-making robustness and long-term planning capabilities of cloud-based multimodal large models in unknown scenarios.

[0086] In one embodiment, a candidate multimodal large model is reinforced and trained using a group relative policy optimization algorithm to determine the cloud-based multimodal large model, including:

[0087] A model testing task is performed using candidate multimodal large models. A population relative policy optimization algorithm is used to determine the reward function based on the task execution results of the candidate multimodal large models. The model reward score is then determined based on the reward function using the population relative policy optimization algorithm. The model parameters of the candidate multimodal large models are then optimized based on the model reward score to determine the cloud-based multimodal large model.

[0088] Specifically, based on sample interface screenshots, sample command sequences, and corresponding sample text commands, the second multimodal model to be trained is trained to determine candidate multimodal large models. The candidate multimodal large models are then used to execute test tasks for automated control of AR glasses. During the execution of these test tasks, the candidate multimodal large models can generate various test action sequences based on the test interface screenshots and test text commands corresponding to the test tasks. Using a group relative policy optimization algorithm, the test action sequences are selected based on the reward function feedback after each test action sequence is executed. Test action sequences with reward functions greater than a preset threshold are selected as policy feedback sequences. These policy feedback sequences, the corresponding test interface screenshots, and the test text commands are then used to train the candidate multimodal large models to determine the cloud-based multimodal large model.

[0089] The reward function can include an action format reward function, an action type reward function, an action parameter reward function, and a task completion reward function. For example, the action format reward function is determined as follows: if the output JSON format test instruction sequence does not conform to the specification, a reward of -1 is immediately assigned; if it conforms, a reward of +1 is assigned. The action type reward function is determined as follows: if the executed operation falls within the truth box when the test instruction is executed, a reward of +1 is assigned; if the executed operation does not fall within the truth box, a reward of -1 is assigned. Test instructions refer to the instruction operation information determined by the AR glasses based on the test instruction sequence. The action parameter reward function is determined as follows: if the executed operation correctly hits the target control when the test instruction is executed, a reward of +1 is assigned; if the executed operation fails to hit the target control when the test instruction is executed, a reward of -1 is assigned. The task completion reward function is determined as follows: if the instruction execution result is correct after executing the test instruction, a reward of +1 is assigned; if the instruction execution result is incorrect after executing the test instruction, a reward of -1 is assigned.

[0090] The above approach not only reinforces previously learned correct operating patterns, but also teaches the user to improve strategies through trial and error, feedback and adjustment when faced with complex or unfamiliar interface situations, demonstrating a stronger adaptive decision-making ability.

[0091] In one embodiment, such as Figure 3 As shown, using a cloud-based multimodal large model, based on the target interface screenshot and target text instructions, the operation steps required to achieve the user's intent are determined, including:

[0092] S310. Input the target interface screenshot and target text command into the cloud multimodal large model, and extract the target prompt words from the target interface screenshot through the text perceptron in the cloud multimodal large model.

[0093] S320: Using the graphics perceptron in the cloud-based multimodal large model, determine the target icon and target virtual control in the target interface screenshot.

[0094] S330: Determine the target spatial relationship between the target icon and the target virtual control through the spatial perceptron in the cloud-based multimodal large model.

[0095] S340. Using target prompts, target icons, and target virtual controls as target interaction elements, and through the visual language model in the cloud-based multimodal large model, determine the operation steps required to realize the user's intent based on the target spatial relationships, target interaction elements, and target text instructions.

[0096] The above solution utilizes the reasoning capabilities of a visual language model to generate the operational steps required to achieve the user's intent based on the target spatial relationships, target interactive elements, and target text instructions. This improves the reliability of the determined operational steps.

[0097] In one embodiment, the control method for the smart glasses further includes:

[0098] When executing command operation information, smart glasses use a dynamic programming algorithm to determine command correction information based on the target task command, historical task execution data of subtasks already executed during the execution of the command operation information, and real-time screenshots of the smart glasses interface after the execution of subtasks. From the standardized command sequence, the command sequence to be adjusted corresponding to the subtasks that were not executed in the command operation information is determined. Based on the command correction information, the command sequence to be adjusted is adjusted.

[0099] Understandably, a standardized instruction sequence refers to a sequence of multiple instructions, and each instruction in the standardized instruction sequence can be considered a subtask. Historical execution data refers to the data generated during the execution of subtasks within the executed standardized instruction sequence; this data can be recorded in log files and retrieved by reading the log files. Real-time screenshots of the smart glasses' interface after executing subtasks refer to screenshots of the smart glasses' virtual interface obtained after each subtask is executed, allowing for an understanding of the subtask's execution status. Dynamic programming algorithms can generate instruction correction information based on the target task instructions, historical task execution data, and real-time interface screenshots.

[0100] The above scheme can dynamically modify the standardized instruction sequence based on the execution result of the instruction operation information when executing the instruction operation information, which improves the adaptability of the method and thus improves the completion rate of complex tasks.

[0101] In one embodiment, the control method for the smart glasses further includes:

[0102] After the smart glasses complete the execution of the command operation information, they generate operation result feedback information and send the operation result feedback information, intermediate data generated during the execution of the command operation information, screenshots of the operation interface corresponding to the intermediate data, and screenshots of the task completion interface of the smart glasses after the execution of the command operation information to the cloud server. The cloud server retrains the multimodal large model in the cloud based on the operation result feedback information, intermediate data, operation interface screenshots, and task completion interface screenshots.

[0103] For example, after all instruction operations are completed, the AR glasses generate operation result feedback information through a pre-deployed task result feedback module. For instance, the feedback method could be: an announcement via voice saying "You have been connected to your supervisor," or a notification displayed in the user's field of vision to inform them of successful task completion. If the execution result of any step during the operation fails to achieve the expected effect, such as the target application failing to launch or the specified contact not being found, the task result feedback module can capture the abnormal information and promptly provide feedback to the user via voice or text regarding the reason for the operation failure, along with corresponding suggested actions. It can also re-capture a screenshot of the AR glasses' virtual interface and resend the virtual state screenshot and the target text instruction to the cloud server. The cloud server then uses a cloud-based multimodal large model to re-determine the operation steps to ensure the task can be completed successfully.

[0104] The above solution can provide feedback to the user after the instruction operation information is executed, and based on the feedback information, intermediate data, operation interface screenshots, and task completion interface screenshots, it can retrain the cloud-based multimodal large model, continuously optimize the cloud-based multimodal large model, and ensure that the task can be completed smoothly.

[0105] For example, based on the above embodiments, the control method for smart glasses includes:

[0106] The system acquires voice commands input from the user's microphone via the AR glasses, converts these commands into text format (target text commands) using speech recognition and audio encoding technologies. The AR glasses then capture images of the user's surrounding environment using a camera, combining this environmental image with virtual interface elements to determine a target interface screenshot. Both the target interface screenshot and the target text commands are encrypted to determine the encrypted transmission information, which is then sent to the cloud server.

[0107] The cloud server combines pre-trained visual language models, text perceptrons, image perceptrons, and spatial perceptrons to determine a multimodal fusion model. Using sample interface screenshots from smart glasses and corresponding descriptive text, the multimodal fusion model is trained to determine the first multimodal model to be trained. The first multimodal model is then trained using the sample interface screenshots and their annotation information, enabling it to locate and recognize virtual controls, icons, and prompts within the AR glasses' virtual interface. Finally, the first multimodal model is trained again using the sample interface screenshots and their corresponding interactive elements to determine the second multimodal model to be trained.

[0108] Based on sample interface screenshots, sample command sequences, and corresponding sample text commands, a second multimodal model to be trained is used to determine candidate multimodal large models. The candidate multimodal large models are then used to execute test tasks for automated control of AR glasses. During the execution of these test tasks, the candidate multimodal large models can generate various test action sequences based on the test interface screenshots and test text commands corresponding to the test tasks. A group relative policy optimization algorithm is used to select test action sequences based on the reward function feedback after each test action sequence is executed. Test action sequences with reward functions greater than a preset threshold are selected as policy feedback sequences. These policy feedback sequences, the corresponding test interface screenshots, and the test text commands are then used to train the candidate multimodal large models to determine the cloud-based multimodal large model.

[0109] The target interface screenshot and target text command are input into a cloud-based multimodal model. The text perceptron in the cloud-based multimodal model extracts the target prompt words from the screenshot. The graphics perceptron in the cloud-based multimodal model identifies the target icon and target virtual control in the screenshot. The spatial perceptron in the cloud-based multimodal model determines the target spatial relationship between the target icon and target virtual control. Using the target prompt words, target icon, and target virtual control as target interaction elements, the visual language model in the cloud-based multimodal model, based on the target spatial relationship, target interaction elements, and target text command, determines the operation steps required to achieve the user's intent.

[0110] When executing command operation information, smart glasses use a dynamic programming algorithm to determine command correction information based on the target task command, historical task execution data of subtasks already executed during the execution of the command operation information, and real-time screenshots of the smart glasses interface after the execution of subtasks. From the standardized command sequence, the command sequence to be adjusted corresponding to the subtasks that were not executed in the command operation information is determined. Based on the command correction information, the command sequence to be adjusted is adjusted.

[0111] After the smart glasses complete the execution of the command operation information, they generate operation result feedback information and send the operation result feedback information, intermediate data generated during the execution of the command operation information, screenshots of the operation interface corresponding to the intermediate data, and screenshots of the task completion interface of the smart glasses after the execution of the command operation information to the cloud server. The cloud server retrains the multimodal large model in the cloud based on the operation result feedback information, intermediate data, operation interface screenshots, and task completion interface screenshots.

[0112] For example, if the target text instruction is a click instruction, the AR glasses' action instruction parsing module will locate the corresponding virtual control position in the current AR glasses virtual interface based on the provided target identifier or coordinates, and simulate a selection or click operation; if the target text instruction is an input text instruction, the AR glasses can automatically inject the corresponding characters into the specified text input area; if the target text instruction is a wait instruction, the AR glasses can pause for a specified duration; if the target text instruction is a prompt instruction, it can trigger the AR glasses' voice broadcast or interface message display function. In this way, the high-level actions described in the standardized instruction sequence in JSON format are implemented one by one into specific control instructions for the AR glasses.

[0113] For example, the AR glasses invoke the action command parsing module, which can sequentially parse and execute the command information in the standardized command sequence. The action command parsing module reads each command information in the standardized command sequence, locates the corresponding element in the AR glasses' virtual interface based on the control identifier or coordinate information in the read command information, and simulates the execution of the corresponding command operation information through the AR glasses' virtual control interface.

[0114] For example, the standardized instruction sequence determined by the cloud server could be: [

[0116] {"action": "click", "target": "remote collaboration application icon", "x": 500, "y":}

[0117] 200},

[0118] {"action": "wait", "duration": 3000},

[0119] {"action": "click", "target": "Call Supervisor Button", "x": 300, "y":}

[0120] 150} ]

[0122] The standardized instruction sequence described above outlines three steps to be executed sequentially: Step 1: Click the remote collaboration application icon, located at approximately field-of-view coordinates 500, 200. Step 2: Wait 3 seconds for the application to launch. Step 3: Click the "Call Supervisor" button, located at approximately field-of-view coordinates 300, 150. Each instruction includes the action type and a description or location parameter of the target element. In practical applications, the standardized instruction sequence can also include more auxiliary fields, such as "target" to specify the text label or ID (Identity document) of the UI (User Interface) control, "type" to indicate the control type, or "speakText" to indicate the content to be read aloud.

[0123] The AR glasses' action command parsing module locates the remote collaboration application icon in the AR glasses' virtual interface based on coordinates and executes a "click"; then it reads the "wait" command and pauses for about 3 seconds; next, it locates the "Call Supervisor" button and executes a "click". All of the above operations are completed automatically on the device side by calling the AR glasses' system interface, without any manual intervention from the user. From the user's perspective, the AR glasses' virtual interface sequentially displays effects such as the application being opened and a remote call being made. The entire execution process occurs on the AR glasses device, simulating actual interactive operations using a graphical user interface.

[0124] In the aforementioned control method for smart glasses, the smart glasses acquire voice commands issued by the user and convert them into target text commands. The smart glasses then determine a target interface screenshot, encrypt both the screenshot and the text command, and send this encrypted transmission information to a cloud server. The cloud server decrypts the encrypted transmission information, identifies the target interface screenshot and the text command, and, using a cloud-based multimodal big data model, determines the necessary operation steps to achieve the user's intent based on the screenshot and command. The cloud server assembles these operation steps into a standardized command sequence and sends it to the smart glasses. The smart glasses then determine and execute the command operation information based on the standardized command sequence. This method solves the problems of high response latency, high privacy risks, low intelligence, and poor adaptability in current AR device automation control methods. The above solution constructs an intelligent agent system that tightly coordinates AR glasses and a cloud server. The AR glasses quickly process user voice commands through voice recognition technology, converting them into target text commands and uploading them to the cloud server. This eliminates the need to upload audio to the cloud server, reducing interaction latency between the AR glasses and the cloud server. Encrypting the target text commands and target interface screenshots before sending them to the cloud server protects user privacy. Simultaneously, using both target text commands and target interface screenshots as input data for a multimodal large-scale model provides a clear view of the AR glasses' real-time environment and virtual interface state during command execution. The cloud-based multimodal large-scale model deployed on the cloud server analyzes the target interface screenshots and target text commands to determine the necessary steps to achieve the user's intent, improving the intelligence of AR glasses control. Determining the necessary steps through the multimodal large-scale model allows for analysis of target interface screenshots and target text commands based on the massive training data of the large-scale model, thereby enhancing the environmental adaptability of the AR glasses control method. Assembling the operation steps into a standardized command sequence before feeding it back to the AR glasses further reduces interaction latency between the cloud server and the AR glasses.

[0125] Based on the same inventive concept, this application also provides a control device for smart glasses that implements the control method for smart glasses described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more embodiments of the control device for smart glasses provided below can be found in the limitations of the control method for smart glasses described above, and will not be repeated here.

[0126] In one embodiment, such as Figure 4 As shown, a control device for smart glasses is provided, including: a voice conversion module 401, an interface screenshot acquisition module 402, a cloud-based large model inference module 403, an action command parsing module 404, and a command execution module 405, wherein:

[0127] The voice conversion module 401, deployed in smart glasses, is used to acquire voice commands issued by the user and convert the voice commands into target text commands;

[0128] The interface screenshot acquisition module 402 is deployed in smart glasses to determine the target interface screenshot, encrypt the target interface screenshot and the target text command, determine the encrypted transmission information, and send the encrypted transmission information to the cloud server.

[0129] The cloud-based large model inference module 403, deployed on a cloud server, is used to decrypt the encrypted transmission information, determine the target interface screenshot and the target text instruction, and determine the operation steps required to realize the user's intent based on the target interface screenshot and the target text instruction through the cloud-based multimodal large model.

[0130] The action instruction parsing module 404, deployed on a cloud server, is used to assemble the operation steps into a standardized instruction sequence and send the standardized instruction sequence to the smart glasses.

[0131] The instruction execution module 405, deployed in the smart glasses, is used to determine instruction operation information based on the standardized instruction sequence and execute the instruction operation information.

[0132] For example, the control device for the above-mentioned smart glasses further includes:

[0133] A model training module, deployed on a cloud server, combines pre-trained visual language models, text perceptrons, image perceptrons, and spatial perceptrons to determine a multimodal fusion model. Using screenshots of the smart glasses' interface and corresponding sample description text, the multimodal fusion model is trained to determine a first multimodal model to be trained. Based on the screenshots, sample annotation information, and corresponding sample interaction elements, the first multimodal model is trained to determine a second multimodal model to be trained. The sample annotation information includes sample prompts, sample icons, and sample virtual controls in the screenshots. Based on the screenshots, sample instruction sequences, and corresponding sample text instructions, the second multimodal model is trained to determine a large cloud-based multimodal model.

[0134] Furthermore, the aforementioned model training module is also specifically used for:

[0135] Based on the sample interface screenshot, sample instruction sequence, and sample text instruction corresponding to the sample interface screenshot, the second multimodal model to be trained is trained to determine the candidate multimodal large model; the candidate multimodal large model is reinforced and trained using a population relative strategy optimization algorithm to determine the cloud-based multimodal large model.

[0136] Furthermore, the aforementioned model training module is also specifically used for:

[0137] The candidate multimodal large model is used to perform a model testing task. A population relative policy optimization algorithm is used to determine the reward function based on the task execution result of the candidate multimodal large model. The model reward score is determined based on the reward function using the population relative policy optimization algorithm. The model parameters of the candidate multimodal large model are optimized based on the model reward score to determine the cloud-based multimodal large model.

[0138] For example, the cloud-based large model inference module 403 is specifically used for:

[0139] The target interface screenshot and the target text command are input into the cloud-based multimodal large model. The target prompt words in the target interface screenshot are extracted by the text perceptron in the cloud-based multimodal large model.

[0140] The target icon and target virtual control in the target interface screenshot are determined by the graphics perceptron in the cloud multimodal large model.

[0141] The spatial relationship between the target icon and the target virtual control is determined by the spatial perceptron in the cloud-based multimodal large model.

[0142] Using the target prompts, target icons, and target virtual controls as target interactive elements, and through the visual language model in the cloud-based multimodal large model, the operation steps required to realize the user's intent are determined based on the target spatial relationships, the target interactive elements, and the target text instructions.

[0143] For example, the control device for the above-mentioned smart glasses further includes:

[0144] An instruction planning module, deployed in smart glasses, is used to determine instruction correction information based on the target text instruction, historical task execution data of subtasks already executed during the execution of the instruction operation information, and real-time screenshots of the smart glasses interface after the execution of the subtasks, using a dynamic programming algorithm when executing the instruction operation information; determine the instruction sequence to be adjusted corresponding to the unexecuted subtasks in the instruction operation information from the standardized instruction sequence; and adjust the instruction sequence to be adjusted based on the instruction correction information.

[0145] For example, the control device for the above-mentioned smart glasses further includes:

[0146] A feedback information determination module, deployed in smart glasses, is used to generate operation result feedback information after the execution of the instruction operation information, and send the operation result feedback information, intermediate data generated when executing the instruction operation information, a screenshot of the operation interface corresponding to the intermediate data, and a screenshot of the task completion interface of the smart glasses after the execution of the instruction operation information to the cloud server.

[0147] The model retraining module, deployed on a cloud server, is used to retrain the cloud-based multimodal large model based on the operation result feedback information, intermediate data, operation interface screenshots, and task completion interface screenshots.

[0148] The various modules in the control device of the aforementioned smart glasses can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0149] In one embodiment, a computing device is provided, including a memory and a processor. The memory stores a computer program. When the computing device is smart glasses, the following steps are implemented: acquiring a voice command issued by a user and converting the voice command into a target text command; determining a target interface screenshot and encrypting the target interface screenshot and the target text command, determining encrypted transmission information, and sending the encrypted transmission information to a cloud server; determining instruction operation information based on a standardized instruction sequence fed back by the cloud server, and executing the instruction operation information.

[0150] When the computing device is a cloud server, the following steps are performed: decrypt the encrypted transmission information, determine the target interface screenshot and the target text instruction, and determine the operation steps required to realize the user's intent based on the target interface screenshot and the target text instruction through a cloud-based multimodal big data model; assemble the operation steps into a standardized instruction sequence, and send the standardized instruction sequence to the smart glasses.

[0151] For example, in one embodiment, the computer device described above may be a terminal, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a positioning method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0152] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0153] In one embodiment, a control system for smart glasses is provided, the control system comprising smart glasses and a cloud server, wherein:

[0154] The smart glasses terminal is used to acquire voice commands issued by the user and convert them into target text commands; determine the target interface screenshot, encrypt the target interface screenshot and the target text commands, determine the encrypted transmission information, and send the encrypted transmission information to the cloud server; acquire the standardized command sequence sent by the cloud server, determine the command operation information based on the standardized command sequence, and execute the command operation information;

[0155] The cloud server is used to obtain encrypted transmission information sent by the smart glasses, decrypt the encrypted transmission information, determine the target interface screenshot and the target text instruction, and determine the operation steps required to realize the user's intention based on the target interface screenshot and the target text instruction through the cloud multimodal big model; assemble the operation steps into a standardized instruction sequence, and send the standardized instruction sequence to the smart glasses.

[0156] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0157] Step 1: The smart glasses acquire the voice commands issued by the user and convert the voice commands into target text commands;

[0158] Step 2: The smart glasses identify the target interface screenshot, encrypt the target interface screenshot and the target text command, determine the encrypted transmission information, and send the encrypted transmission information to the cloud server;

[0159] Step 3: The cloud server decrypts the encrypted transmission information, determines the target interface screenshot and target text command, and uses the cloud multimodal big model to determine the operation steps required to realize the user's intent based on the target interface screenshot and target text command.

[0160] Step 4: The cloud server assembles the operation steps into a standardized instruction sequence and sends the standardized instruction sequence to the smart glasses;

[0161] Step 5: The smart glasses determine the instruction operation information based on the standardized instruction sequence and execute the instruction operation information.

[0162] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0163] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0164] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0165] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A control method of smart glasses, characterized by, The control method of the smart glasses is realized by a smart glasses control system, the smart glasses control system comprising smart glasses and a cloud server, and the control method of the smart glasses comprising: The smart glasses obtain a voice instruction issued by a user and convert the voice instruction into a target text instruction; The smart glasses determine a target interface screenshot, encrypt the target interface screenshot and the target text instruction, determine encrypted transmission information, and send the encrypted transmission information to the cloud server; The cloud server decrypts the encrypted transmission information, determines the target interface screenshot and the target text instruction, and determines operation steps required to achieve a user intent according to the target interface screenshot and the target text instruction through a cloud multimodal large model; The cloud server assembles the operation steps into a standardized instruction sequence and sends the standardized instruction sequence to the smart glasses; The smart glasses determine instruction operation information according to the standardized instruction sequence and execute the instruction operation information; The training method of the cloud multimodal large model comprises: The cloud server combines a pre-trained visual language model, a text perceiver, a graph perceiver, and a spatial perceiver to determine a multimodal fusion model; The multimodal fusion model is trained using a sample interface screenshot of the smart glasses and a sample description text corresponding to the sample interface screenshot to determine a first to-be-trained multimodal model; The first to-be-trained multimodal model is trained according to a sample interface screenshot, sample annotation information of the sample interface screenshot, and a sample interactive element corresponding to the sample interface screenshot to determine a second to-be-trained multimodal model; the sample annotation information comprises a sample prompt word, a sample icon, and a sample virtual control in the sample interface screenshot; The second to-be-trained multimodal model is trained according to the sample interface screenshot, a sample instruction sequence, and a sample text instruction corresponding to the sample interface screenshot to determine a cloud multimodal large model.

2. The method of claim 1, wherein, The second to-be-trained multimodal model is trained according to the sample interface screenshot, a sample instruction sequence, and a sample text instruction corresponding to the sample interface screenshot to determine a cloud multimodal large model, comprising: The second to-be-trained multimodal model is trained according to the sample interface screenshot, a sample instruction sequence, and a sample text instruction corresponding to the sample interface screenshot to determine a candidate multimodal large model; The candidate multimodal large model is reinforced through a group relative strategy optimization algorithm to determine the cloud multimodal large model.

3. The method of claim 2, wherein, The candidate multimodal large model is reinforced through a group relative strategy optimization algorithm to determine the cloud multimodal large model, comprising: A model test task is executed through the candidate multimodal large model, a reward function is determined according to a task execution result of the candidate multimodal large model executing the model test task using a group relative strategy optimization algorithm; The population-based strategy optimization algorithm is used to determine a model reward score based on the reward function, and model parameters of the candidate multi-modal large model are optimized based on the model reward score to determine a cloud multi-modal large model.

4. The method of claim 1, wherein, Through the cloud multi-modal large model, the operation steps required to achieve the user intent are determined according to the target interface screenshot and the target text instruction, including: The target interface screenshot and the target text instruction are input into the cloud multi-modal large model, and a target prompt word of the target interface screenshot is extracted through a text perceiver in the cloud multi-modal large model; Through a graph perceiver in the cloud multi-modal large model, a target icon and a target virtual control in the target interface screenshot are determined; Through a space perceiver in the cloud multi-modal large model, a target spatial relationship between the target icon and the target virtual control is determined; The target prompt word, the target icon, and the target virtual control are taken as target interactive elements, and through a visual language model in the cloud multi-modal large model, operation steps required to achieve the user intent are determined according to the target spatial relationship, the target interactive elements, and the target text instruction.

5. The method of claim 1, wherein, Further comprising: The intelligent glasses determine instruction correction information through a dynamic programming algorithm according to the target text instruction, historical task execution data of a subtask that has been executed in the instruction operation information execution process, and a real-time interface screenshot of the intelligent glasses after the subtask is executed when executing the instruction operation information; From the standardized instruction sequence, a to-be-adjusted instruction sequence corresponding to a subtask not executed in the instruction operation information is determined; Based on the instruction correction information, the to-be-adjusted instruction sequence is adjusted.

6. The method of claim 1, wherein, Further comprising: The intelligent glasses generate operation result feedback information after the execution of the instruction operation information ends, and send the operation result feedback information, intermediate data generated when the user operation is executed, an operation interface screenshot corresponding to the intermediate data, and a task completion interface screenshot of the intelligent glasses after the execution of the instruction operation information ends to the cloud server; The cloud server performs model retraining on the cloud multi-modal large model according to the operation result feedback information, the intermediate data, the operation interface screenshot, and the task completion interface screenshot.

7. A control device of smart glasses, characterized by, The control device of the intelligent glasses comprises: A voice conversion module deployed on the intelligent glasses, configured to acquire a voice instruction issued by a user and convert the voice instruction into a target text instruction; An interface screenshot acquisition module deployed on the intelligent glasses, configured to determine a target interface screenshot, encrypt the target interface screenshot and the target text instruction, determine encrypted transmission information, and send the encrypted transmission information to the cloud server; A cloud large model inference module deployed on the cloud server, configured to decrypt the encrypted transmission information, determine the target interface screenshot and the target text instruction, and determine operation steps required to achieve the user intent according to the target interface screenshot and the target text instruction through the cloud multi-modal large model. An action instruction analysis module is arranged in a cloud server, configured to assemble the operation steps into a standardized instruction sequence, and send the standardized instruction sequence to the smart glasses; An instruction execution module is arranged in the smart glasses, configured to determine instruction operation information according to the standardized instruction sequence, and execute the instruction operation information; A model training module is arranged in the cloud server, configured to combine a pre-trained visual language model, a text perceiver, a graph perceiver and a space perceiver to determine a multi-modal fusion model; use a sample interface screenshot of the smart glasses and sample description text corresponding to the sample interface screenshot to train the multi-modal fusion model to determine a first to-be-trained multi-modal model; use the sample interface screenshot, sample annotation information of the sample interface screenshot and sample interactive elements corresponding to the sample interface screenshot to train the first to-be-trained multi-modal model to determine a second to-be-trained multi-modal model; the sample annotation information includes sample prompt words, sample icons and sample virtual controls in the sample interface screenshot; use the sample interface screenshot, a sample instruction sequence and sample text instructions corresponding to the sample interface screenshot to train the second to-be-trained multi-modal model to determine a cloud multi-modal large model.

8. A computing device comprising a memory and a processor, the memory storing a computer program, characterized in that, When the computing device is the smart glasses, the following steps are implemented: obtaining a voice instruction issued by a user, and converting the voice instruction into target text instructions; determining a target interface screenshot, and encrypting the target interface screenshot and the target text instructions to determine encrypted transmission information, and sending the encrypted transmission information to the cloud server; determining instruction operation information according to a standardized instruction sequence fed back by the cloud server, and executing the instruction operation information; When the computing device is the cloud server, the following steps are implemented: decrypting the encrypted transmission information to determine the target interface screenshot and the target text instructions, and determining operation steps required to achieve a user's intention according to the target interface screenshot and the target text instructions through a cloud multi-modal large model; assembling the operation steps into a standardized instruction sequence, and sending the standardized instruction sequence to the smart glasses; The training method of the cloud multi-modal large model includes: The cloud server combines a pre-trained visual language model, a text perceiver, a graph perceiver and a space perceiver to determine a multi-modal fusion model; The sample interface screenshot of the smart glasses and the sample description text corresponding to the sample interface screenshot are used for model training of a multi-modal fusion model to determine a first to-be-trained multi-modal model; the sample interface screenshot, sample annotation information of the sample interface screenshot, and sample interactive elements corresponding to the sample interface screenshot are used for model training of the first to-be-trained multi-modal model to determine a second to-be-trained multi-modal model; the sample annotation information includes sample prompt words, sample icons, and sample virtual controls in the sample interface screenshot; the sample interface screenshot, a sample instruction sequence, and sample text instructions corresponding to the sample interface screenshot are used for model training of the second to-be-trained multi-modal model to determine a cloud multi-modal large model.

9. A control system of smart glasses, characterized by, The control system of the smart glasses includes the smart glasses of claim 8 and a cloud server.

Images