Color vision assisting method and device based on intelligent glasses, medium and intelligent glasses

Abstract

The invention discloses a color vision assisting method and device based on intelligent glasses, a medium and the intelligent glasses, the method is applied to the intelligent glasses with a built-in camera and a processor, in the implementation process of the method, real-time video streams in the visual field of a user wearing the intelligent glasses are collected through the camera, the real-time video streams are sent to the processor, and the real-time video streams are sent to the processor; the processor is used for identifying at least part of objects involved in the real-time video stream through the processor so as to determine object information of the at least part of objects, the object information at least comprises category name information and color information, and then color vision auxiliary prompting is carried out on a user according to the object information.

Description

Technical Field

[0001] This invention relates to the fields of smart devices and artificial intelligence, and in particular to a color vision assistance method, device, medium, and smart glasses based on smart glasses. Background Technology

[0002] Color vision deficiency (commonly known as color blindness or color weakness) is a common visual perception impairment affecting tens of millions of people worldwide. It primarily manifests as difficulty distinguishing specific colors, such as red and green, and in some cases, difficulty identifying blue and yellow. This presents numerous challenges to individuals' daily lives, such as recognizing traffic lights, selecting ripe fruit, matching clothing colors, and limiting their participation in occupations and artistic activities that rely on color recognition.

[0003] To bring convenience to users with color vision deficiencies, the following two assistive methods are currently provided: The first method: special filter glasses. The principle is to use special coated filters in the lenses to filter out some overlapping spectra, thereby enhancing the contrast between easily confused colors such as red and green, and helping some color-blind patients to distinguish them better. The second method involves mobile auxiliary applications that use the smartphone's camera to capture images, identify colors in the scene through software algorithms, and then inform the user in the form of text or voice. This is equivalent to adding a "color recognition" function to the phone.

[0004] While both methods can provide some assistance, they both have certain limitations. Specifically, the first method cannot allow users with color vision deficiencies to see colors they would not normally perceive, and it offers little help to those with complete color blindness or certain types of color blindness. The second method, while providing color information, requires users to actively take out their smartphones, point them at the target object, and view the screen. This process is cumbersome, interrupts natural and continuous visual observation, and fails to provide a real-time, seamless assisted experience.

[0005] Therefore, how to provide effective and convenient color vision assistance methods for users with color vision disorders is a technical problem that urgently needs to be solved. Summary of the Invention

[0006] This specification implements a color vision assistance method, device, medium, and smart glasses based on smart glasses, in order to partially solve the problem that the prior art cannot provide effective and convenient color vision assistance to users with color vision disorders.

[0007] The embodiments in this specification adopt the following technical solutions: This specification provides an embodiment of a color vision assistance method based on smart glasses. The method is applied to smart glasses, which have a built-in camera and a processor, including: The camera captures a real-time video stream from the field of view of the user wearing the smart glasses. The real-time video stream is sent to the processor so that the processor can identify at least some of the objects involved in the real-time video stream to determine the object information of the at least some of the objects, the object information including at least category name information and color information; Based on the object information, color vision assistance prompts are provided to the user.

[0008] Furthermore, in some embodiments, before identifying at least a portion of the objects involved in the real-time video stream by the processor, the method further includes: Acquire the user's eye movement data during the acquisition of the real-time video stream; The real-time video stream is sent to the processor, which then identifies at least some of the objects in the real-time video stream to determine the object information of the at least some of the objects, specifically including: The real-time video stream and the eye-tracking data are sent to the processor, which processes the eye-tracking data to determine the target object being gazed at by the user, and identifies the target object involved in the real-time video stream to determine the object information of the target object.

[0009] Furthermore, in some embodiments, the processor identifies at least some of the objects involved in the real-time video stream, specifically including: The processor uses a built-in recognition model to identify at least some of the objects in the real-time video stream.

[0010] Furthermore, in some embodiments, the smart glasses are equipped with a communication module; The processor identifies at least some of the objects in the real-time video stream, specifically including: The processor invokes the communication module to send the real-time video stream to a recognition model in the cloud for identification of at least some of the objects in the real-time video stream.

[0011] Furthermore, in some embodiments, the recognition model includes an object recognition model and a color recognition model; Identifying at least some of the objects involved in the real-time video stream, specifically including: The real-time video stream is input into the object recognition model to identify at least some of the objects involved in the real-time video stream, and to determine the category name information corresponding to the at least some objects and the position information of the images of the at least some objects in the real-time video stream. Based on the location information, the color of at least some of the objects is identified using the color recognition model to determine the color information corresponding to at least some of the objects.

[0012] Furthermore, in some embodiments, color vision assistance prompts are provided to the user based on the object information, specifically including: Based on the object information, a prompt message is generated; The prompt information is displayed in the user's field of vision through the display device built into the smart glasses, so as to mark at least some of the objects through the prompt information.

[0013] Furthermore, in some embodiments, color vision assistance prompts are provided to the user based on the object information, specifically including: Based on the object information, a prompt message is generated; The prompts are played through the smart glasses to provide color vision assistance to the user.

[0014] This specification provides an embodiment of a color vision assistive device based on smart glasses. The device has a built-in camera and a processor, including: The acquisition module is used to acquire real-time video streams from the field of view of the user wearing the device via the camera; The recognition module is used to send the real-time video stream to the processor so that the processor can recognize at least some of the objects involved in the real-time video stream to determine the object information of the at least some of the objects, wherein the object information includes at least category name information and color information; The prompting module is used to provide color vision assistance prompts to the user based on the object information.

[0015] This specification provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned color vision assistance method based on smart glasses.

[0016] This specification provides an embodiment of smart glasses, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the aforementioned color vision assistance method based on smart glasses.

[0017] The above-mentioned technical solutions adopted in this specification can achieve the following beneficial effects: The color vision assistance method based on smart glasses provided in the embodiments of this specification is applied to smart glasses with built-in cameras and processors. During the implementation of the method, the camera captures a real-time video stream in the field of vision of the user wearing the smart glasses, and sends the real-time video stream to the processor. The processor identifies at least some of the objects involved in the real-time video stream to determine the object information of the at least some objects. The object information includes at least category name information and color information. Based on the object information, color vision assistance prompts are then provided to the user.

[0018] As can be seen from the above method, the smart glasses in this specification can provide color vision assistance to users. By collecting real-time video streams from the user's field of vision, objects appearing in the user's field of vision can be identified to determine the category name and color information of at least some objects in the user's field of vision. This information can then be used to provide color vision assistance to users with color vision impairments. This not only provides effective assistance to users with various types of color vision impairments but also eliminates the need for users to perform cumbersome operations, thus greatly facilitating their daily lives. Attached Figure Description

[0019] Figure 1 A flowchart illustrating a color vision assistance method based on smart glasses, provided as an embodiment of this specification; Figure 2 This diagram illustrates how the smart glasses provided in this manual call upon a recognition model deployed in the cloud to complete a recognition task. Figure 3 This is a schematic diagram illustrating the use of AR to provide color vision assistance to users, as provided in this instruction manual. Figure 4 A schematic diagram of a color vision assist device based on smart glasses provided in the embodiments of this specification; Figure 5 An embodiment provided in this specification corresponds to Figure 1 A schematic diagram of smart glasses. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0021] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0022] Figure 1 This is a flowchart illustrating a color vision assistance method based on smart glasses, provided as an embodiment of this specification.

[0023] S100: The camera captures a real-time video stream from the field of vision of the user wearing the smart glasses.

[0024] To provide better color vision assistance for users with color vision deficiencies and to offer them greater convenience in their daily lives, this manual provides a set of smart glasses. These smart glasses incorporate a high-definition camera and a processor. The camera can capture real-time video streams of the user's field of vision, and subsequently, based on these video streams, it can identify objects appearing in the user's field of vision and their color information.

[0025] The aforementioned smart glasses also have a built-in processor that processes the real-time video stream captured by the camera to obtain the recognition results of objects appearing in the user's field of vision, and based on this, can generate corresponding prompts for the user.

[0026] In addition to these, smart glasses also incorporate other components, such as communication modules like WiFi or Bluetooth, to enable internet connectivity and interaction with other smart devices like smartphones. Furthermore, smart glasses also include built-in batteries and charging ports (including wired charging ports such as Type-C, magnetic charging ports with magnetic contacts, and wireless charging ports) for charging. Other common components built into smart glasses will not be listed here.

[0027] S102: The real-time video stream is sent to the processor so that the processor can identify at least some of the objects involved in the real-time video stream to determine the object information of the at least some of the objects.

[0028] After acquiring the aforementioned real-time video streams, they can be sent to the processor in the smart glasses. The processor can then process these real-time video streams to identify at least some of the objects appearing in the user's field of vision.

[0029] This specification demonstrates various methods for object recognition within the user's field of vision. The processor may incorporate a built-in recognition model, which can be an artificial intelligence model such as a neural network model or a convolutional neural network model. Considering the need for lightweight design in smart glasses, the processor embedded within them is a microprocessor; correspondingly, the recognition model embedded in the microprocessor can be a lightweight model obtained through methods such as knowledge distillation.

[0030] The recognition model built into the processor can be written directly into the processor before the smart glasses leave the factory, or it can be downloaded from the network and written into the processor after the smart glasses are connected to the network.

[0031] Of course, assuming the smart glasses have internet connectivity, the aforementioned recognition model can also be deployed in the cloud. Therefore, after acquiring the real-time video stream, the recognition task can be completed using the cloud-deployed recognition model, such as... Figure 2 As shown.

[0032] Figure 2 This diagram illustrates how the smart glasses provided in this manual call upon a recognition model deployed in the cloud to complete a recognition task.

[0033] exist Figure 2 In the process, after acquiring a real-time video stream, the processor can call the communication module in the smart glasses. Figure 2 (Not shown in the image) The real-time video stream is uploaded to the cloud-based recognition model via the communication module. The cloud-based recognition model identifies the objects in the real-time video stream and returns the identification results (i.e., the identified object information) to the smart glasses via the communication module.

[0034] In practical applications, smart glasses can connect to the network in various ways. For example, smart glasses can connect directly to a WiFi network through a built-in WiFi module (i.e., a communication module); another example is that they can establish a communication connection with smart devices such as smartphones through a Bluetooth module (i.e., a communication module). In this case, the smart glasses can first send the real-time video stream they have collected to the connected smart device through the established communication connection, and then the smart device can upload it to the recognition model in the cloud to obtain the recognition result.

[0035] Furthermore, the aforementioned recognition model can possess both object category recognition and object color recognition capabilities. In this case, when the acquired real-time video stream is input into the recognition model, the model can identify both the category and color of objects appearing in the user's field of vision.

[0036] Of course, object category recognition and color recognition can also be accomplished by two different recognition models. In this case, the aforementioned recognition models can include an object recognition model and a color recognition model. The object recognition model is responsible for identifying the category of objects appearing in the user's field of vision, while the color recognition model is responsible for identifying the color of objects appearing in the user's field of vision.

[0037] In this specification, a real-time video stream can first be input into an object recognition model to identify at least some of the objects involved in the real-time video stream, determining the category name information corresponding to these at least some objects and the position information of the images of these at least some objects in the real-time video stream. Then, based on this position information, a color recognition model can be used to identify the colors of these at least some objects to determine the color information corresponding to these at least some objects.

[0038] The aforementioned location information is used to indicate the image position of at least a portion of the object within one or more frames of the real-time video stream. Therefore, the aforementioned location information and the real-time video stream can be sent to the color recognition model, allowing the color recognition model to first determine the image region of the object requiring color recognition in the real-time video stream based on the location information, and then perform color recognition on these image regions to obtain the color information of the object to be recognized.

[0039] Of course, you can first determine the image area in the real-time video stream where the object to be identified appears based on the above location information, and then only use this part of the image area (such as a screenshot) as input to the color recognition model so that the color recognition model can obtain the color information of the object to be identified.

[0040] It should be noted that the color information mentioned above can reflect the color of the object to be identified itself (e.g., if the object to be identified is an apple, then the obtained color information directly reflects the color of the apple itself), or it can reflect the color of the current state of the object to be identified (e.g., if the object to be identified is a traffic light, then the obtained color information reflects the color of the traffic light that is currently lit).

[0041] In addition to category name and color information, the above object information may also include other information, such as the location information and status information of the object to be identified (status information can reflect the state of the object; for example, when the object is an apple, the status information can reflect whether the apple is ripe or not).

[0042] In this specification, the processor can obtain object information of all objects in the user's field of vision based on the above recognition model. Subsequently, it can further generate prompt information for all objects in the user's field of vision to prompt the user about the category and color of these objects.

[0043] Of course, in practical applications, users often only focus on a portion of the objects in their field of vision, so it is possible to identify only the object information of those objects that the user is focusing on.

[0044] Therefore, in this specification, the smart glasses can acquire the user's eye movement data while capturing the aforementioned real-time video stream via a built-in eye-tracking module (such as a camera). The real-time video stream and eye movement data can then be sent to a processor in the smart glasses. The processor processes the eye movement data to identify the target object being gazed at by the user within the objects depicted in the real-time video stream, and identifies the target object within the real-time video stream to determine its object information.

[0045] In other words, eye-tracking data can be used to determine the object that the user is focusing on, and then the focus can be placed on or solely on the category name and color information of the target object.

[0046] S104: Provide color vision assistance prompts to the user based on the object information.

[0047] After obtaining the above object information, color vision assistance prompts can be provided to users wearing smart glasses based on this object information.

[0048] Among these technologies, smart glasses can provide color vision assistance to users through augmented reality (AR). The smart glasses can generate prompts based on the object information, which can be in text or icon form (the icons can include clear and easily recognizable simplified images of objects or suggestive text). Then, the smart glasses can display the prompts in AR within the user's field of vision using a built-in display device (such as a miniature display or projector). Users can then use the displayed prompts to mark the objects that need to be identified, such as... Figure 3 As shown.

[0049] Figure 3 This is a schematic diagram illustrating the use of AR to provide color vision assistance to users, as provided in this instruction manual.

[0050] Figure 3 The image displayed on the right is the view seen through one of the lenses of the smart glasses. As you can see, the traffic lights, roads, and buses in this view are real images seen through the lens. The marked boxes and the "red light" displayed in the marked boxes are generated prompts. These prompts are integrated with the real-world images in an AR way to be presented in the user's view.

[0051] Of course, in addition to using AR to assist users' color vision, sound can also be used to assist users' color vision.

[0052] Specifically, the smart glasses generate prompts based on the aforementioned object information, and then play these prompts through the smart glasses to provide color vision assistance to the user. The smart glasses can play these prompts in various ways, such as through a speaker built into the glasses or via bone conduction.

[0053] For audio-based prompts, in addition to displaying the category name and color of the object to be identified, information such as the object's location and current status can also be included. For example, when a user is looking at a traffic light, the prompt could be: "The traffic light directly in front of you is currently red. Please wait at the stop line at the intersection."

[0054] As can be seen from the above method, the smart glasses in this specification can provide color vision assistance to users. By collecting real-time video streams from the user's field of vision, objects appearing in the user's field of vision can be identified to determine the category name and color information of at least some of the objects in the user's field of vision. This information can then be used to provide color vision assistance to users with color vision deficiencies. This not only provides effective assistance to users with various types of color vision deficiencies but also eliminates the need for users to perform cumbersome operations, thus greatly facilitating their daily lives.

[0055] The above describes one or more embodiments of a color vision assisting method based on smart glasses provided in this specification. Based on the same concept, this specification also provides corresponding color vision assisting devices based on smart glasses, such as… Figure 4 As shown.

[0056] Figure 4 This is a schematic diagram of a color vision assistive device based on smart glasses, provided as an embodiment of this specification. The device has a built-in camera and a processor, and includes: Acquisition module 400 is used to acquire real-time video streams from the field of view of a user wearing the device via the camera; The recognition module 402 is used to send the real-time video stream to the processor so that the processor can recognize at least some of the objects involved in the real-time video stream to determine the object information of the at least some of the objects, wherein the object information includes at least category name information and color information; The prompting module 404 is used to provide color vision assistance prompts to the user based on the object information.

[0057] Optionally, before the processor identifies at least some of the objects involved in the real-time video stream, the acquisition module 400 is further configured to acquire the user's eye-tracking data when acquiring the real-time video stream; The recognition module 402 is specifically used to send the real-time video stream and the eye-tracking data to the processor, so that the processor can process the eye-tracking data to determine the target object that the user is looking at, and to identify the target object involved in the real-time video stream to determine the object information of the target object.

[0058] Optionally, the recognition module 402 is specifically used to recognize at least some of the objects involved in the real-time video stream using the recognition model built into the processor.

[0059] Optionally, the smart glasses are equipped with a communication module; The recognition module 402 is specifically used to call the communication module through the processor to send the real-time video stream to the recognition model in the cloud through the communication module, so as to recognize at least some of the objects involved in the real-time video stream.

[0060] Optionally, the recognition model includes an object recognition model and a color recognition model; The recognition module 402 is specifically used to input the real-time video stream into the object recognition model to recognize at least some of the objects involved in the real-time video stream, determine the category name information corresponding to the at least some objects and the position information of the images of the at least some objects in the real-time video stream; and, based on the position information, recognize the color of the at least some objects through the color recognition model to determine the color information corresponding to the at least some objects.

[0061] Optionally, the prompting module 404 is specifically used to generate prompting information based on the object information; and to display the prompting information in the user's field of vision through the display device built into the smart glasses, so as to mark at least some of the objects through the prompting information.

[0062] Optionally, the prompting module 404 is specifically used to generate prompting information based on the object information; and to play the prompting information through the smart glasses to provide color vision assistance prompts to the user.

[0063] The above-described apparatus embodiments correspond to the method embodiments, and detailed descriptions can be found in the description of the method embodiments section, which will not be repeated here. The apparatus embodiments are derived based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments; detailed descriptions can be found in the corresponding method embodiments.

[0064] This specification also provides an embodiment of a computer storage medium that can store multiple instructions adapted to be loaded and executed by a processor as described above. Figure 1The method described in the illustrated embodiment can be found in the following document for a detailed execution process. Figure 1 The specific details of the illustrated embodiments will not be elaborated here.

[0065] This specification also provides a computer program product that stores at least one instruction, said at least one instruction being loaded and executed by the processor as described above. Figure 1 The method described in the illustrated embodiment can be found in the following document for a detailed execution process. Figure 1 The specific details of the illustrated embodiments will not be elaborated here.

[0066] The embodiments in this specification also provide Figure 5 The diagram shows the structure of smart glasses. Figure 5 At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the aforementioned color vision assistance method based on smart glasses.

[0067] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0068] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must also be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0069] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0070] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0071] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0072] Those skilled in the art will understand that embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0073] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0074] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0075] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0076] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0077] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0078] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0079] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0080] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0081] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0082] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0083] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A color vision assistance method based on smart glasses, the method being applied to smart glasses, the smart glasses having a built-in camera and processor, comprising: The camera captures a real-time video stream from the field of view of the user wearing the smart glasses. The real-time video stream is sent to the processor so that the processor can identify at least some of the objects involved in the real-time video stream to determine the object information of the at least some of the objects, the object information including at least category name information and color information; Based on the object information, color vision assistance prompts are provided to the user.

2. The method of claim 1, further comprising, before identifying at least a portion of the objects involved in the real-time video stream by means of the processor: Acquire the user's eye movement data during the acquisition of the real-time video stream; The real-time video stream is sent to the processor, which then identifies at least some of the objects in the real-time video stream to determine the object information of the at least some of the objects, specifically including: The real-time video stream and the eye-tracking data are sent to the processor, which processes the eye-tracking data to determine the target object being gazed at by the user, and identifies the target object involved in the real-time video stream to determine the object information of the target object.

3. The method as described in claim 1, wherein the processor identifies at least a portion of the objects involved in the real-time video stream, specifically including: The processor uses a built-in recognition model to identify at least some of the objects in the real-time video stream.

4. The method as described in claim 1, wherein the smart glasses are provided with a communication module; The processor identifies at least some of the objects in the real-time video stream, specifically including: The processor invokes the communication module to send the real-time video stream to a recognition model in the cloud for identification of at least some of the objects in the real-time video stream.

5. The method as described in claim 3 or 4, wherein the recognition model includes an object recognition model and a color recognition model; Identifying at least some of the objects involved in the real-time video stream, specifically including: The real-time video stream is input into the object recognition model to identify at least some of the objects involved in the real-time video stream, and to determine the category name information corresponding to the at least some objects and the position information of the images of the at least some objects in the real-time video stream. Based on the location information, the color of at least some of the objects is identified using the color recognition model to determine the color information corresponding to at least some of the objects.

6. The method as described in claim 1, wherein providing color vision assistance prompts to the user based on the object information, specifically includes: Based on the object information, a prompt message is generated; The prompt information is displayed in the user's field of vision through the display device built into the smart glasses, so as to mark at least some of the objects through the prompt information.

7. The method as described in claim 1, wherein providing color vision assistance prompts to the user based on the object information specifically includes: Based on the object information, a prompt message is generated; The prompts are played through the smart glasses to provide color vision assistance to the user.

8. A color vision assistive device based on smart glasses, the device having a built-in camera and processor, comprising: The acquisition module is used to acquire real-time video streams from the field of view of the user wearing the device via the camera; The recognition module is used to send the real-time video stream to the processor so that the processor can recognize at least some of the objects involved in the real-time video stream to determine the object information of the at least some of the objects, wherein the object information includes at least category name information and color information; The prompting module is used to provide color vision assistance prompts to the user based on the object information.

9. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any one of claims 1 to 7.

10. A smart glasses, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in any one of claims 1 to 7.

Images