Joint learning for smart head-mounted devices
Through the joint learning mechanism of local neural networks and global neural networks, the resource and privacy issues of intelligent head-mounted devices are solved when enhancing active functions, personalized control and function promotion are realized, and efficient user experience and data privacy protection are provided. .
Patent Information
- Application Number
- CN202410825986.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-06-27
- Filing Date
- 2024-06-25
- Publication Date
- 2024-12-27
AI Technical Summary
Existing smart head-mounted devices face high demands for energy, memory and data storage resources when enhancing active functions, and data privacy and bandwidth limitations become challenges, making it difficult to implement personalized and promote algorithms or models.
The joint learning mechanism of local neural networks and global neural networks is adopted to collect wearer information through input modules, optimize active functions, and realize personalized control and function promotion through signal transmission and weight replacement mechanisms to reduce data transmission needs.
Effectively manage the proactive functions of smart head-mounted devices, provide a tailored user experience, reduce energy and storage resource requirements, solve data privacy issues, and promote algorithms or models among multiple users.
Smart Images
Figure CN119204165A_ABST
Abstract
Description
Technical Field
[0001] The present invention generally relates to intelligent head-mounted devices, and more particularly to a communication system, a head-mounted device, a method for controlling a head-mounted device, and a corresponding non-transitory computer-readable storage medium. Background Art
[0002] Smart eyewear has evolved to include a variety of active features designed to improve wearer comfort, visual acuity, and ease of use. These active features can be personalized based on the wearer's interactions and preferences, providing a unique, customized user experience. To achieve this, eyewear typically requires extensive data collection, processing, and storage capabilities, which can be challenging given the inherent resource constraints in wearable devices.
[0003] Typically, to enhance active functions, data from eyewear is transmitted to a smartphone and then further to a cloud server for processing. The processed data is then sent back to the eyewear, typically in the form of an updated algorithm or model. However, this approach has several limitations. It requires significant energy and memory resources, not only for the eyewear and smartphone, but also for cloud storage and processing. Additionally, bandwidth limitations can make it challenging to transmit raw data to the cloud, while the need to store raw data in the cloud raises concerns about data privacy.
[0004] Some alternative approaches have been proposed, such as processing all data directly on the smartphone, which could alleviate some of the issues related to bandwidth and data privacy. However, these approaches have their own limitations. They still require large amounts of local data storage and lack the ability to generalize algorithms or models across multiple users, which could limit the functionality and adaptability of eyewear.
[0005] Therefore, there is a need for a system and method that can effectively control the active functions of smart eye wear based on the wearer's interactions and preferences while overcoming the limitations of current methods. Summary of the invention
[0006] The invention is defined by the accompanying independent claims. Additional features and advantages of the concepts disclosed herein are set out in the following description.
[0007] The present disclosure aims to improve this situation.
[0008] In particular, the different aspects outlined in the present disclosure collectively contribute to effectively managing one or more active functions of a smart head-mounted device based on the wearer's interactions and / or preferences, thereby addressing the shortcomings associated with traditional approaches. Each of these aspects plays a key role in the personalization of active functions tailored to the wearer, while further providing the ability to generalize across populations of wearers. In addition, these aspects help minimize the necessary energy, memory, and data storage resources, thereby reducing the need for large amounts of raw data transmission. Importantly, the implementation of these aspects effectively addresses concerns about data privacy.
[0009] To this end, the present disclosure describes a communication system for a head mounted device with active functionality, the communication system comprising:
[0010] - an interface capable of sending and receiving signals between the local neural network and the global neural network,
[0011] in:
[0012] - the local neural network enables control of the active function using information collected by the input module about the wearer of the head mounted device or the wearer's surroundings,
[0013] -The sent and received signals help the local neural network to participate in a joint learning process with the global neural network.
[0014] The information collected by the input module ensures a comprehensive understanding of the wearer's needs and preferences.
[0015] The local neural network enables precise control of active functions of the head-mounted device. By utilizing information collected by the input module, the local neural network allows the functions of the head-mounted device to be optimized according to the needs and preferences of the wearer. In addition, the local neural network can use the information collected by the input module to actively predict the needs of the wearer. For example, the local neural network can dynamically optimize the intensity of the hue based on changing lighting conditions, or actively adjust the optical focal length to reduce visual fatigue.
[0016] As wearers with similar use cases and preferences contribute to a global network, the collective wisdom accumulated through the federated learning process can be used to improve the active functions of the headset and provide tailored solutions for specific scenarios. Whether it is adapting to new lighting environments, optimizing for different visual tasks, or meeting the requirements of specific wearers, the ability to learn and evolve through federated learning ensures that the communication system is relevant and effective in different situations.
[0017] Optionally, the signal from the local neural network encapsulates a set of N local weights, N>1, which represents the relationship between training data and corresponding labels of the training data, where the training data is information collected by the input module and the labels correspond to various states of the active function.
[0018] In some cases, the characteristics or preferences of a particular wearer may be derived from the training data collected by the input module and may be reflected by the set of local weights. Thus, the utilization of local weights may enable personalized control of active functions, allowing for a more customized and tailored user experience.
[0019] Further, as indicated, the local set of weights represents a relationship between training data and corresponding labels of the training data. Thus, by encapsulating this relationship in a single signal, the communication system optimizes communication efficiency and reduces overall data transmission requirements.
[0020] Optionally, the signal from the global neural network encapsulates M global weights, N>M≥1, and each received global weight is used to replace the corresponding weight in the set of local weights.
[0021] In this weight replacement mechanism, since the number of global weights M encapsulated in the signal from the global neural network is less than the number of local weights N, only a subset of local weights are replaced while other local weights remain unchanged. This selective replacement provides significant advantages. For example, it allows the head-mounted device to retain local weights that reflect personalized wearer preferences or unique characteristics that are not common or unnecessary on a global scale. Therefore, it helps to maintain the personalized user experience provided by the device while integrating enhanced features learned from the global network. In addition, this mechanism allows global improvements to be adopted gradually, providing a smoother transition from local learned behavior to global learned behavior. This may be particularly useful in eye wear applications, where sudden changes in active functions (such as automatic adjustment of optical power or tint intensity) may cause discomfort or confusion to the wearer.
[0022] The present disclosure also describes a head-mounted device with active functions, the head-mounted device comprising:
[0023] - an input module configured to collect information related to a wearer of the head mounted device or the wearer's surroundings,
[0024] - a local neural network that enables the use of information collected by the input module to control active functions, and
[0025] - an interface capable of sending and receiving signals between the local neural network and the global neural network,
[0026] Among them, the sent and received signals help the local neural network to participate in the joint learning process with the global neural network.
[0027] Optionally, the input module comprises a human-machine interface adapted to receive wearer input from a wearer of the head mounted device.
[0028] The human-machine interface allows the wearer to provide input directly, enabling intuitive and user-friendly control of the head-mounted device. This enhances the overall user experience and simplifies the interaction between the wearer and the device.
[0029] These inputs can include wearer-specific data, such as age, gender, laterality, preferences related to transmission intensity, optical power adjustment level, or desired defocus setting. In some cases, these inputs can also enable the wearer to voluntarily control active functions. Such voluntary modifications of active functions by the wearer can be interpreted as a way to express their satisfaction or dissatisfaction. By analyzing these inputs, the local neural network can fine-tune the active functions to provide a personalized and optimized visual experience. This level of customization ensures that the device adapts to the wearer's unique characteristics and visual requirements, thereby improving the wearer's satisfaction and comfort.
[0030] Optionally, the input module comprises a sensing module adapted to sense:
[0031] - environmental data, for example in a list comprising one or more of: brightness data, distance data and frame frame data, and / or
[0032] - Wearer data, for example in a list comprising one or more of: activity data, sensitivity data, laterality data and posture data.
[0033] Posture data may refer to head orientation data.
[0034] For example, by collecting wearer-specific data, such as activity level, the device can dynamically adjust its active features to provide optimal performance during different activities. Whether the wearer is exercising, working at a computer, or just relaxing, the headset can adjust its features to enhance visual comfort and support the wearer's specific needs in a variety of situations.
[0035] In addition, by collecting environmental data (such as brightness or distance), the headset can intelligently optimize its active functions to provide the best visual experience in various environments. This ensures that the wearer gets the best visual clarity and comfort in bright lighting or at different distances.
[0036] Optionally, the head-mounted device includes an optical lens, and the active function is an optical function of the optical lens.
[0037] Optionally, the optical function is a transmission function, a power function or a defocus function. The transmission function may be a hue function.
[0038] By incorporating optical lenses as part of the head-mounted device, active functions become directly related to the optical properties of the lenses. This integration allows precise control and optimization of optical functions, thereby enhancing the wearer's visual experience. Whether it is adjusting tint intensity for different lighting conditions, modifying optical power for personalized vision correction, or enabling defocus functions for specific tasks, the device can become a comprehensive solution for a variety of optical needs.
[0039] Optionally, the head mounted device includes a screen, and the active function is a function of the screen. The screen may be a light field display. The function of the screen may be to amplify some feature displayed on the screen, such as font, brightness or color… The function of the screen may also be a picture displayed within a mobile headset, such as the pictures seen in various types of head mounted devices implementing augmented reality, virtual reality, mixed reality or extended reality technology.
[0040] The present disclosure also describes a method for controlling a head mounted device having an active function, the method comprising:
[0041] - using the input module to collect information related to the wearer of the head mounted device or the wearer's surroundings,
[0042] enabling the use of local neural networks and the information collected to control active functions, and
[0043] -Sending and receiving signals between the local neural network and the global neural network through the interface, wherein the sent and received signals facilitate the local neural network to participate in a joint learning process with the global neural network.
[0044] The present disclosure also describes a computer-readable storage medium, optionally a non-transitory computer-readable storage medium, having stored thereon a computer program comprising instructions that, when executed by a processor, cause the processor to perform the methods described herein.
[0045] The storage medium may include a hard drive, a solid state drive, a CD, a USB drive, etc. In this context, the storage medium is used to refer to any medium storing a computer program for running the method.
[0046] The present disclosure also describes a computer program comprising instructions accessible to a processor and which, when executed by the processor, causes the processor to perform the method described herein.
[0047] The present disclosure also describes a device equipped with a processor operably connected to a memory and a communication interface, the device being configured to perform any of the methods described herein.
[0048] The present disclosure also describes a communication interface configured to send and receive signals between a local neural network module and a global neural network module,
[0049] in:
[0050] - the local neural network module is capable of generating or refining a personalized model for a wearer of a head mounted device, wherein the head mounted device comprises an optical lens having an optical function as an active function,
[0051] - the personalized model enables the use of information collected by the input module about the wearer of the head mounted device or the wearer's surroundings to control active functions,
[0052] -The sent and received signals facilitate the local neural network module to participate in a joint learning process with the global neural network module, wherein said participation provides the ability to generalize personalized models across a population of wearers.
[0053] Optionally, the signal from the local neural network module encapsulates a set of N local weights, N>1, which represents the relationship between training data and corresponding labels of the training data, where the training data is information collected by the input module and the labels correspond to various states of the active function.
[0054] Optionally, the signal from the global neural network module encapsulates M global weights, N>M≥1, and each received global weight is used to replace the corresponding weight in the local weight set.
[0055] Optionally, the federated learning process is a personalized federated learning process defined by:
[0056] -the global neural network module is able to access wearer data indicating that the wearer of the head mounted device belongs to a particular wearer group, and
[0057] -The signal from the global neural network module to the local neural network module is tailored for the specific wearer group and, optionally, does not include any information related to the group identity of the specific wearer group.
[0058] The present disclosure also describes a head mounted device, the head mounted device comprising an optical lens having an optical function as an active function, the head mounted device comprising:
[0059] - an input module configured to collect information related to a wearer of the head mounted device or the wearer's surroundings,
[0060] - a local neural network module capable of generating or improving a personalized model for the wearer,
[0061] a personalization model that enables the use of information collected by the input module to control active functions, and
[0062] - an interface configured to send and receive signals between the local neural network module and the global neural network module,
[0063] The sent and received signals facilitate the local neural network module to participate in a joint learning process with the global neural network module, wherein the participation provides the ability to generalize personalized models across a population of wearers.
[0064] Optionally, the input module comprises a human-machine interface adapted to receive wearer input from a wearer of the head mounted device.
[0065] Optionally, the input module comprises a sensing module adapted to sense:
[0066] - environmental data, for example in a list comprising one or more of: brightness data, distance data and frame frame data, and / or
[0067] - Wearer data, for example in a list comprising one or more of: activity data, sensitivity data, laterality data and posture data.
[0068] Optionally, the optical function is a transmission function, a power function or a defocus function.
[0069] The present disclosure also describes a computer-implemented method for controlling a head-mounted device including an optical lens having an optical function as an active function, the method comprising:
[0070] - using the input module to collect information related to the wearer of the head mounted device or the wearer's surroundings,
[0071] - enabling the control of active functions using a personalized model generated or refined for the wearer by the local neural network module (120) and the information collected, and
[0072] -Sending and receiving signals between the local neural network module and the global neural network module (130) via interfaces (124, 132), wherein the sent and received signals facilitate the local neural network module to participate in a joint learning process with the global neural network module, wherein said participation provides the ability to generalize the personalized model across a population of wearers.
[0073] The present disclosure also describes a non-transitory computer-readable storage medium having stored thereon a computer program including instructions which, when executed by a processor, cause the processor to perform the above-described computer-implemented method. BRIEF DESCRIPTION OF THE DRAWINGS
[0074] Figure 1 A general overview of a system suitable for controlling active functions of a head mounted device in an exemplary embodiment is depicted.
[0075] Figure 2 The general structure of an artificial neural network in an exemplary embodiment is depicted.
[0076] Figure 3 Depicting an exemplary embodiment Figure 1 The communication scheme between the elements shown. DETAILED DESCRIPTION
[0077] The focus of this disclosure is on methods and systems that facilitate adaptive, intelligent control of active functions in head-mounted devices through personalized machine learning models. The main elements disclosed therein involve head-mounted devices, controllers, input modules, local neural networks, global neural networks, and communication interfaces between the two networks.
[0078] The input module collects relevant data from the wearer's interactions and the environment, and the local neural network utilizes the data to build a local model and generate instructions for the controller to control one or more active functions of the device.
[0079] The local networks participate in a federated learning process with the global neural network, allowing the two neural networks to refine and update their respective models. Federated learning is a machine learning approach in which models are trained on multiple decentralized devices or servers that hold local data samples without exchanging that data. This approach maintains data privacy and reduces the need for data centralization.
[0080] One of the significant advantages of this approach is its ability to enable personalization. By leveraging machine learning, the active features of the headset can be adapted to the specific needs and preferences of individual wearers. Furthermore, through a federated learning process, improvements in the model can benefit all users without compromising their personal data, as only model updates (also known as weights) are shared rather than raw data.
[0081] The elements of the present disclosure have broad applicability to various types of head-mounted devices. For example, they can be used to automatically adjust the tint of eyewear based on lighting conditions, fine-tune the focus of near vision tasks, or manage myopia control for children's eyewear. Therefore, the present disclosure has the potential to greatly enhance the user experience in a range of applications in the field of wearable technology.
[0082] refer to Figure 1 , which provides a general overview of a system designed to control active functions of a head mounted device in an exemplary embodiment.
[0083] A head mounted device may be any device worn on the head. In the context of the present disclosure, this may include, but is not limited to, glasses, headphones, helmets, virtual reality devices, augmented reality devices, etc. A head mounted device may include one or more optical lenses. An optical lens is a piece of transparent material with curved sides that is used to gather or disperse light.
[0084] Active functionality can be any dynamically modifiable aspect of a headset. This can include optical functions, such as adjusting the tint or focus of glasses, the volume or sound balance of headphones, or the display settings of a VR or AR device, to name a few examples.
[0085] The system includes an input module (110), which may include various input sources, such as sensors or a human-machine interface.
[0086] A sensor or sensing module is a device or part of a device that is capable of detecting and responding to some type of input from the physical environment. In the context of the present disclosure, sensors may include ambient light sensors for lighting condition analysis, proximity sensors for nearby object detection, accelerometers for measuring the direction and speed of head movement, eye trackers (particularly infrared eye trackers) for identifying gaze direction, time-of-flight sensors for sensing gaze distance, cameras for capturing images of the wearer or the environment, etc.
[0087] A human-machine interface is a device or software that allows humans to interact with machines. In the context of the present disclosure, a human-machine interface may be suitable for receiving, for example, tactile input or auditory input (such as voice commands). Other examples of suitable human-machine interfaces include gesture recognition systems and graphical user interfaces on connected devices. For example, some human-machine interfaces may allow for the provision of declarative data about the wearer (such as their age, gender, laterality, facial features, prescription, etc.), or the retrieval of compiled data about the frames of the head-mounted device (referred to as "frame frame data"), all of which may be provided as input to the system.
[0088] Some of the input sources listed above may be incorporated into the head mounted device, while others may be provided on a connected device for user input.
[0089] The system further includes a local neural network module (120) having an input interface (122) designed to accommodate a different range of inputs from the input module. It also has an output interface (126) responsible for sending instructions to a controller (140) that manages one or more active functions of the head-mounted device. The local network module can take a variety of forms, such as dedicated processing circuitry built directly into the head-mounted device, or even integrated into another device used by the wearer (such as a smartphone or tablet).
[0090] In addition, the system also has a global neural network module (130) that can communicate with the local neural network modules via a communication link. This communication link is symbolically represented by a pair of communication interfaces (124, 132). The global neural network module can be hosted in a remote server or as an application in a cloud-based environment. The global neural network module can be configured to communicate with other data sources (150), such as other local neural networks, a cloud database storing anonymous data from multiple wearers, or an external API providing real-time weather data.
[0091] Reference now Figure 2 , which provides the general structure of an artificial neural network in an exemplary embodiment. This general structure applies to both local and global neural networks. It is a simplified representation intended to convey the key elements of the network structure, rather than an accurate description of the complexity of the network in actual applications.
[0092] The artificial neural network shown includes multiple layers of neurons. At the ends, there is an input layer (210) and an output layer (250), with one or more hidden layers (230) sandwiched between them. Each of these layers contains a number of neurons (212) interconnected with each neuron in the adjacent layer. These interconnections are called "weights".
[0093] Artificial neural networks are often used to create and tune models. The training phase involves providing labeled training data as input to the input layer. In machine learning, "training data" refers to the raw data from which the model learns, and "labels" refer to the corresponding outcomes or categories that the model is trained to "predict." Weights are numerical parameters in the neural network that are adjusted during the training phase to improve the predictions of the output layer output. After the training phase is complete, the artificial neural network enters the production phase to receive raw data that is not necessarily labeled as input and output corresponding predictions.
[0094] To show the clarity, Figure 2 Only a few layers and a handful of neurons are shown. In reality, neural networks contain many more layers, densely packed with large numbers of interconnected neurons, creating countless weights or connections, far beyond what can be represented in a single schematic diagram.
[0095] In this disclosure, the term "local weight" refers to the connection between two neurons within a local neural network, and "global weight" similarly refers to the connection within the global neural network. Typically, there is a one-to-one correspondence between local weights and global weights, meaning that the structure of the local neural network (including the number of layers, the number of neurons within each layer, and the function of each neuron) reflects the structure of the global neural network. However, it should be noted that this one-to-one correspondence is not strictly necessary for the operation of the system, as long as the local and global networks share a compatible structure that allows the global weights to be mapped or converted to the local network environment.
[0096] Reference now Figure 3 , which provides an exemplary embodiment of Figure 1 The communication scheme between the elements shown.
[0097] It can be considered that at a given moment, the head mounted device is worn by the wearer and the input module (110) has previously collected relevant data through interaction with the human-machine interface or normal operation of the sensor.
[0098] The accumulated data is transmitted (302) to the input of the local neural network module (120). The local neural network module then uses the transmitted data to generate or refine a personalized model of the wearer. The personalized model is used to control active functions of the device, such as color tone changes, optical power adjustments, defocus function modifications, or display settings for AR / VR content, all of which are calibrated according to the wearer's usage patterns and preferences.
[0099] Simultaneously or periodically, the personalized model is promoted across a larger group of wearers.
[0100] The promotion process involves a series of basic actions.
[0101] First, weights inferred from the local neural network are sent (306) to the global neural network module (130), which encapsulates the correlation between the accumulated training data and the labels corresponding to the training data. The global neural network module (130) also receives (304) relevant data from other sources (150), such as a centralized database storing demographic information about the wearer, environmental data, or information collected from other head-mounted devices in the network.
[0102] The global neural network module (130) formulates a generalized model. This may involve selecting a subgroup of wearers for the generalization step and averaging the neural network weights from the selected devices to build a global neural network model. Based on the principle of federated learning (FL), the model represents the collective intelligence of the selected subgroup of wearers.
[0103] The global weights from the global model are then transmitted back (308) to each wearable device. Depending on the specific architecture of the system, these weights may be sent directly to the head mounted device or may be initially sent to a connected smartphone or other computing device.
[0104] Upon receipt, these global weights are integrated into the local model, forming an updated local model. This can involve selectively replacing certain local weights while retaining other local weights. Various approaches can be used to this end, such as only sending global weights that are designated to replace local weights, or passing a broader set of global weights and using a filtering mechanism at the local neural network module to select specific weights for replacement. Regardless, the modified local model constitutes a personalized adjustment to the generalized model, thereby retaining aspects unique to the wearer while integrating insights from the global neural network.
[0105] Finally, active functions are adjusted based on the updated local model (310). The model guides the active functions of the device to provide a wearer experience tailored to the wearer's individual needs and preferences while being informed by collective learning from the larger group of wearers. This approach allows the device to dynamically adapt to new situations while remaining tuned to specific local preferences, thereby leveraging collective learning accumulated from the wider collection of wearers.
[0106] This general description is intended to present exemplary embodiments of the present invention. Changes, modifications, and substitutions will be apparent to those skilled in the art, and may be made without departing from the scope of the present invention. The specific configurations of components and the manner in which they interact are merely illustrative, and alternative configurations and interactions are within the scope of the appended claims.
[0107] With this general description, the following specific embodiments are used to further illustrate the proposed invention. These embodiments correspond to different use cases, and each embodiment presents a unique method of personalized smart eyewear operation. Each embodiment includes Figure 3 The same general process already depicted in , which involves data collection, model creation, model generalization, and practical application of the model to the functionality of the head-mounted device.
[0108] One embodiment revolves around a pair of glasses equipped with optical lenses having an adjustable tint, such as electrochromic glasses.
[0109] Initial data collection involves adjusting the tint based on a deterministic model whose output is completely determined by the input, leaving no room for random variation. The model can be universally applicable, personalized, or segmented, depending on the measurements captured in a professional optometry setting.
[0110] The model has been successfully tested operating primarily on the ambient light sensor (ALS) values provided by the corresponding sensor. Future tests will be conducted by further incorporating other input data, such as the wearer's sensitivity to light, measured by infrared (IR) sensors or by spectrophotometry.
[0111] A personalized model is then developed through local computation based on the wearer's specific interactions. The model integrates inputs such as ALS, meteorological conditions, and wearer activity. Future refinements may consider other inputs such as indoor / outdoor environment, IR, positioning, sun angle, etc.
[0112] The neural network weights are uploaded to a cloud-based environment, and a collection of wearers with similar sensitivity levels are selected to create a generalized model using a federated learning strategy. This generalized model is then further personalized by learning certain layers of the neural network based on the wearer's specific data.
[0113] Finally, the tint of the glasses is adjusted based on the final model to reflect the predicted wearer preference, which may be affected by various factors including meteorological conditions, season, GPS location, ALS, spectrophotometry, IR, etc.
[0114] Another embodiment is directed to an eyewear with an adjustable focus feature intended to assist with near vision tasks. A general process similar to that described above is implemented, but using different parameters specific to near vision assistance. During the initial data collection phase, the E-Focus focal power is adjusted based on a primary model, for example, personalized or segmented by near vision activity distance measurements at an ECP store. The personalized model uses parameters such as time of flight (ToF) sensor, pitch head angle, and activity state. After model generalization and focus adjustment based on predicted wearer preferences, this can result in a smart eyewear device that provides excellent near vision support.
[0115] One unique embodiment focuses on eyewear with adjustable defocus that is specifically tailored for myopia management in children. The initial model can manipulate the defocus based on factors such as the distance of near vision activity, reading head angle, age of the wearer, and specific data related to the eyewear frame fitment.
[0116] It is important to note that in real scenarios, the defocus function is not always active. The defocus function is only called when necessary. In addition, each activation or adjustment of the defocus function requires some adaptation time for the wearer's eyes. Therefore, the data captured during this eye adaptation period may not be completely relevant. Therefore, the input module can be configured to send only the sensed data after a sufficiently long period of time has passed after the defocus function is activated or modified to the local neural network module.
[0117] Once the data is sent, a personalized model is created and then generalized so that the defocus can be adjusted based on the predictions made by the personalized model.
[0118] Ultimately, personalized and generalized models predict the wearer's preferences and adjust the defocus accordingly.
[0119] Each of these examples illustrates how the proposed invention allows head-mounted devices (particularly eyewear) to adapt to the needs of the wearer while also taking into account the collective intelligence from a wider collection of wearers. The actions detailed above can be modified and adjusted to best meet the needs of the wearer, resulting in a more flexible, efficient, and satisfying user experience.
Claims
1. A communication system for a head mounted device with active functionality, the communication system comprising: - an interface capable of sending and receiving signals between the local neural network and the global neural network, in: - the local neural network enables control of the active functions using information collected by the input module about the wearer of the head mounted device or the wearer's surroundings, -The transmitted and received signals help the local neural network to participate in a joint learning process with the global neural network.
2. The communication system according to claim 1, wherein: The signal from the local neural network encapsulates a set of N local weights, N>1, and the set represents the relationship between training data and corresponding labels of the training data, wherein the training data is the information collected by the input module, and the labels correspond to various states of the active function.
3. The communication system according to claim 2, wherein: The signal from the global neural network encapsulates M global weights, N>M≥1, and each received global weight is used to replace the corresponding weight in the local weight set.
4. The communication system according to any one of claims 1 to 3, wherein: The federated learning process is a personalized federated learning process defined by: - the global neural network has access to wearer data indicating that the wearer of the head mounted device belongs to a particular wearer group, and - The signal from the global neural network to the local neural network is tailored for the specific group of wearers and optionally does not include any information related to the group identity of the specific group of wearers.
5. A head-mounted device with active functions, and the head-mounted device comprises: - an input module configured to collect information related to a wearer of the head mounted device or the wearer's surroundings, - a local neural network enabling control of said active functions using said information collected by said input module, and - an interface capable of sending and receiving signals between the local neural network and the global neural network, The sent and received signals help the local neural network to participate in a joint learning process with the global neural network.
6. The head mounted device according to claim 5, wherein: The input module includes a human-machine interface adapted to receive wearer input from a wearer of the head mounted device.
7. The head mounted device according to claim 5 or 6, wherein: The input module comprises a sensing module, wherein the sensing module is adapted to sense: - environmental data, for example in a list comprising one or more of: brightness data, distance data and frame frame data, and / or - Wearer data, for example in a list comprising one or more of: activity data, sensitivity data, laterality data and posture data.
8. The head mounted device according to any one of claims 5 to 7, wherein: The head mounted device comprises an optical lens, and the active function is an optical function of the optical lens.
9. The head mounted device of claim 8, wherein: The optical function is a transmission function, a power function or a defocus function.
10. The head mounted device according to any one of claims 5 to 7, wherein: The head mounted device comprises a screen, and the active function is a function of the screen.
11. A method for controlling a head mounted device having active functions, the method comprising: - using an input module to collect information related to a wearer of the head mounted device or the wearer's surroundings, enabling use of the local neural network and the collected information to control the active function, and - sending and receiving signals between the local neural network and the global neural network through an interface, wherein the sent and received signals facilitate the local neural network to participate in a joint learning process with the global neural network.
12. A non-transitory computer-readable storage medium having stored thereon a computer program comprising instructions which, when executed by a processor, cause the processor to perform the method according to claim 11.