Color blindness auxiliary identification method and device, head-mounted display device, and storage medium

Abstract

The application discloses a color blindness auxiliary identification method and device, equipment and storage medium, relates to the technical field of head-mounted display equipment, and the method is applied to the head-mounted display equipment, and the head-mounted display equipment comprises an inside camera, an outside camera and an imaging module. The method comprises the following steps: determining the color blindness auxiliary type of the head-mounted display equipment, wherein the color blindness auxiliary type corresponds to at least one obstacle color; identifying the fixation point of a target user through the inside camera, and acquiring a scene image through the outside camera; identifying the target object corresponding to the fixation point in the scene image; if the target object exists and belongs to the obstacle color, the target color corresponding prompt information is displayed in the imaging module. The application identifies the target object at the fixation point of the user, and specifically prompts the color of the object observed by the user, avoids the influence on the whole visual picture, avoids the influence of the environment on the color correction, and thus the robustness of the color auxiliary identification is improved.

Description

Technical Field

[0001] This application relates to the field of head-mounted display technology, and in particular to a colorblindness-assisted identification method, device, equipment, and storage medium. Background Technology

[0002] Color blindness, also known as congenital color vision deficiency, is a common visual disorder that prevents patients from distinguishing various colors in the natural spectrum or a particular color.

[0003] In daily life, we often encounter various scenarios that require us to distinguish colors. For people with color blindness, the existing color vision correction methods mostly involve adding certain filters to the lenses or adding filters for color blindness to the displayed image. However, different devices, different lighting environments, or different screen resolutions will affect the effect of the filters, resulting in poor correction results.

[0004] Therefore, how to improve the robustness of color-assisted recognition has become an urgent problem to be solved in this field.

[0005] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0006] The main objective of this application is to provide a colorblindness-assisted identification method, apparatus, device, and storage medium, aiming to solve the technical problem of how to improve the robustness of color-assisted identification.

[0007] To achieve the above objectives, this application proposes a colorblindness-assisted identification method, which is applied to a head-mounted display device. The head-mounted display device includes an inner camera, an outer camera, and an imaging module. The method includes:

[0008] Determine the color blindness assistance type of the head-mounted display device, wherein the color blindness assistance type corresponds to at least one obstacle color;

[0009] The inner camera identifies the target user's gaze point, and the outer camera acquires scene images.

[0010] Identify the target object corresponding to the gaze point in the scene image;

[0011] If the target object contains a target color that corresponds to the obstacle color, then the imaging module displays a prompt message corresponding to the target color.

[0012] In one embodiment, the step of determining the color blindness assistance type of the head-mounted display device includes:

[0013] When the color blindness assistance type of the head-mounted display device is initialized, multiple preset test images are displayed through the imaging module;

[0014] The comparison result is obtained by comparing the feedback information received based on the test image with the color label of the test image;

[0015] If the comparison result shows that the feedback information is different from the color label, then the color label is taken as the obstacle color;

[0016] Determine the color blindness assistance type corresponding to the color of the obstacle in a preset color blindness assistance type database.

[0017] In one embodiment, the step of identifying the target user's gaze point through the inner camera includes:

[0018] The target user's eye movement video is acquired through the inner camera;

[0019] Identify the pupil direction in the eye video;

[0020] The gaze point of the target user is determined based on the pupil direction.

[0021] In one embodiment, the step of displaying the prompt information corresponding to the target color in the imaging module includes:

[0022] Identify the outline position of the target color;

[0023] An overlay layer is generated in the imaging module at the same position as the contour.

[0024] The overlay layer displays a prompt message corresponding to the target color.

[0025] In one embodiment, the step of displaying the prompt information corresponding to the target color in the overlay layer includes:

[0026] Obtain the preset text information corresponding to the target color;

[0027] The preset text information is displayed in the overlay layer.

[0028] In one embodiment, the step of displaying the prompt information corresponding to the target color in the overlay layer further includes:

[0029] Obtain the color mapping relationship corresponding to the preset color blindness assistance type;

[0030] Determine the auxiliary color corresponding to the target color based on the color mapping relationship;

[0031] The auxiliary color is displayed in the overlay layer.

[0032] In one embodiment, after the step of identifying the target object corresponding to the gaze point in the scene image, the method further includes:

[0033] Identify the target outline of the target object;

[0034] The scene image is cropped according to the target contour to obtain the target image of the target object;

[0035] Obtain the color value of each pixel in the target image;

[0036] Target pixels whose color values ​​belong to the same color range are divided into the same region to obtain at least one color region;

[0037] The color corresponding to the color region is used as the color of the target object.

[0038] Furthermore, to achieve the above objectives, this application also proposes a colorblindness-assisted identification device, wherein the device is disposed on a head-mounted display device, the head-mounted display device comprising: an inner camera, an outer camera, and an imaging module; the device comprises:

[0039] A type determination module is used to determine the color blindness assistance type of the head-mounted display device, wherein the color blindness assistance type corresponds to at least one obstacle color;

[0040] The camera module is used to identify the gaze point of the target user through the inner camera and to acquire scene images through the outer camera;

[0041] The object recognition module is used to identify the target object corresponding to the gaze point in the scene image;

[0042] A colorblindness assistance module is used to display prompt information corresponding to the target color in the imaging module if there is a target color in the target object that belongs to the color of the obstacle.

[0043] In addition, to achieve the above objectives, this application also proposes a head-mounted display device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the color blindness-assisted identification method described above.

[0044] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the color blindness-assisted identification method described above.

[0045] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the color blindness-assisted identification method described above.

[0046] This application provides a colorblindness-assisted identification method. First, the colorblindness assistance type of the head-mounted display device is determined. The colorblindness assistance type corresponds to at least one obstacle color that the target user wearing the head-mounted display device cannot accurately identify. Then, the target user's gaze point is identified by an inner camera, and the scene image of the target user's current gaze is acquired by an outer camera. Then, the target object at the target user's gaze point in the scene image can be identified. If the target object contains a target color that the target user cannot accurately identify, the corresponding prompt information of the target color is displayed in the imaging module of the head-mounted display device to assist the user in identifying the target color.

[0047] In summary, this application provides targeted color blindness assistance prompts based on the type of color blindness, which can adapt to different types of color blindness. Furthermore, by identifying the target object at the user's gaze point, it provides targeted color assistance prompts for the object being gazed at. Compared to directly adding filters, this avoids the impact on the overall visual image and avoids the influence of the environment on color correction, thereby improving the robustness of color-assisted recognition. Attached Figure Description

[0048] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0049] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a schematic diagram of the hardware structure of the head-mounted display device involved in the embodiment of the color blindness-assisted identification method of this application;

[0051] Figure 2 This is a flowchart illustrating an embodiment of the colorblindness-assisted identification method of this application.

[0052] Figure 3 This is a flowchart illustrating Embodiment 2 of the colorblindness-assisted identification method of this application.

[0053] Figure 4 This is a schematic diagram of the module structure of the colorblindness-assisted identification device according to an embodiment of this application;

[0054] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the color blindness-assisted identification method in the embodiments of this application.

[0055] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0056] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0057] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0058] The main solution of this application embodiment is: determining the color blindness assistance type of the head-mounted display device, wherein the color blindness assistance type corresponds to at least one obstacle color; identifying the gaze point of the target user through the inner camera and acquiring a scene image through the outer camera; identifying the target object corresponding to the gaze point in the scene image; if there is a target color belonging to the obstacle color in the target object, then displaying the prompt information corresponding to the target color in the imaging module.

[0059] Color blindness, also known as congenital color vision deficiency, is a common visual disorder that prevents patients from distinguishing various colors in the natural spectrum or a particular color.

[0060] In daily life, we often encounter various scenarios that require us to distinguish colors. For people with color blindness, the existing color vision correction methods mostly involve adding certain filters to the lenses or adding filters for color blindness to the displayed image. However, different devices, different lighting environments, or different screen resolutions will affect the effect of the filters, resulting in poor correction results.

[0061] Therefore, how to improve the robustness of color-assisted recognition has become an urgent problem to be solved in this field.

[0062] To address the aforementioned issues, this application provides a colorblindness-assisted identification method. First, the colorblindness assistance type of the head-mounted display device is determined. Each colorblindness assistance type corresponds to at least one obstacle color that the target user wearing the head-mounted display device cannot accurately identify. Then, the target user's gaze point is identified using an inner camera, and the scene image currently being gazed at by the target user is acquired using an outer camera. The target object at the target user's gaze point in the scene image can then be identified. If the target object contains a target color that the target user cannot accurately identify, the corresponding prompt information for that target color is displayed in the imaging module of the head-mounted display device to assist the user in identifying the target color.

[0063] In summary, this application provides targeted color blindness assistance prompts based on the type of color blindness, which can adapt to different types of color blindness. Furthermore, by identifying the target object at the user's gaze point, it provides targeted color assistance prompts for the object being gazed at. Compared to directly adding filters, this avoids the impact on the overall visual image and avoids the influence of the environment on color correction, thereby improving the robustness of color-assisted recognition.

[0064] In this embodiment, for ease of description, the following description uses a head-mounted display device as the execution subject.

[0065] Based on this, this application provides a colorblindness-assisted identification method, which is applied to head-mounted display devices. Please refer to... Figure 1 , Figure 1 This is a schematic diagram of the hardware structure of the head-mounted display device involved in the embodiment of the color blindness-assisted identification method of this application, as shown below. Figure 1 As shown, the head-mounted display device includes a camera image acquisition module, which comprises four cameras: a left inner camera and a right inner camera. The device also includes a power supply module, a SOC main control module, an image processing module, and an optomechanical imaging module. Based on the above hardware structure, please refer to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the colorblindness-assisted identification method of this application.

[0066] In this embodiment, the color blindness-assisted identification method includes steps S10 to S40:

[0067] Step S10: Determine the color blindness assistance type of the head-mounted display device, wherein the color blindness assistance type corresponds to at least one obstacle color;

[0068] In this embodiment, during the initialization or user setup phase, the head-mounted display device determines the user's color blindness assistance type based on the user's color blindness type (such as red-green color blindness, blue-yellow color blindness, etc.).

[0069] Furthermore, in a feasible implementation, step S10 above may include steps S11 to S14:

[0070] Step S11: When the color blindness assistance type of the head-mounted display device is initialized, multiple preset test images are displayed through the imaging module;

[0071] In this embodiment, the head-mounted display device enters an initialization state when it is used for the first time or when the user needs to reset the color blindness assistance type. At this time, the head-mounted display device displays a series of test images to the user through its imaging module. These test images contain various color combinations and patterns, designed to test the user's ability to recognize colors.

[0072] Step S12: Compare the feedback information received based on the test image with the color label of the test image to obtain a comparison result;

[0073] In this embodiment, the head-mounted display device guides the user to provide feedback while displaying these test images. User feedback may include selecting what they perceive as the dominant or specific colors in the image, or indicating their color recognition through other forms of interaction, such as text input or voice messages. The device compares the user's feedback with preset color labels for each test image to assess the user's color recognition accuracy.

[0074] Step S13: If the comparison result is that the feedback information is different from the color label, then the color label is taken as the obstacle color;

[0075] In this embodiment, if the user's feedback is inconsistent with the color label of the test image, it indicates that the user has a recognition difficulty in that color. In this case, the device will record the color label as the user's obstacle color.

[0076] Step S14: Determine the color blindness assistance type corresponding to the obstacle color in a preset color blindness assistance type database.

[0077] In this embodiment, the head-mounted display device has a built-in color blindness assistance type database, which contains the correspondence between different color blindness types and obstacle colors. Based on the determined obstacle color, the corresponding color blindness assistance type can be found in the database.

[0078] For example, when a user first uses the head-mounted display device, the device is in an initial state for colorblindness assistance. This means the device needs to determine the user's specific type of colorblindness in order to provide the most suitable assistance. The device displays a series of preset test images to the user through its built-in imaging module. These test images contain various color combinations and patterns designed to test the user's ability to recognize different colors. After observing the test images, the user provides feedback through the device's interface, indicating the colors or patterns they see. The device compares this feedback with color labels on the test images. These color labels are pre-defined and represent the colors that should be present in different parts of the test images. If the user's feedback does not match the color labels on the test images, the device considers these inconsistent color labels as obstacle colors. These obstacle colors are colors that the user cannot accurately identify. The device then matches these obstacle colors with a preset database of colorblindness assistance types. This database contains various types of colorblindness and their corresponding obstacle colors and assistance methods. Through matching, the device can determine the user's type of colorblindness and automatically adjust its colorblindness assistance function to provide the most accurate color recognition assistance.

[0079] In addition, after the color blindness assistance type of the head-mounted display device is set, the color blindness assistance type can be reset in the interactive interface when the user changes.

[0080] In addition to conducting image tests, users can also directly select some preset color blindness assistance types in the settings through interactive operations.

[0081] By matching and identifying the color blindness type of the target user, it is possible to adapt to users with different color blindness types and improve the application range of head-mounted display devices.

[0082] Step S20: Identify the target user's gaze point using the inner camera and acquire scene images using the outer camera;

[0083] In this embodiment, after the head-mounted display device starts working, the dual-camera system built into the head-mounted display device starts working. The inner camera is responsible for capturing and recognizing the user's eye movements, thereby accurately locating the user's gaze point. At the same time, the outer camera captures images of the user's current environment or scene in real time.

[0084] Furthermore, in a feasible implementation, the step of identifying the target user's gaze point through the inner camera in step S20 above may include steps S21 to S23:

[0085] Step S21: Acquire the target user's eye video through the inner camera;

[0086] In this embodiment, the inner camera of the head-mounted display device starts working, capturing the eye movements of the target user. By continuously shooting and recording videos of the user's eyes, the head-mounted display device can obtain real-time dynamic information about the user's eyes.

[0087] Step S22: Identify the pupil direction in the eye video;

[0088] In this embodiment, the acquired eye video is subjected to in-depth analysis, with particular attention to the movement and positional changes of the pupil. Through image processing technology and algorithms, the head-mounted display device can accurately identify the direction of the pupil.

[0089] Step S23: Determine the gaze point of the target user based on the pupil direction.

[0090] In this embodiment, the gaze point of the target user can be calculated using known eye movement models and pupil direction information.

[0091] For example, when a user wears a head-mounted display device, the camera inside the head-mounted display device acquires images of the user's eyeballs in real time. Through image recognition, the head-mounted display device can identify the user's pupils in the eyeball images. By identifying the pupil images from the two different cameras on the left and right sides, the head-mounted display device can identify the direction of the user's left and right eyes respectively. By combining the directions of the left and right eyes, the user's gaze point can be detected.

[0092] Step S30: Identify the target object corresponding to the gaze point in the scene image;

[0093] In this embodiment, after acquiring the scene image and the gaze point, depth analysis is performed on the acquired scene image, and the target object corresponding to the user's gaze point is identified through a recognition algorithm.

[0094] Step S40: If there is a target color in the target object that belongs to the obstacle color, then display the prompt information corresponding to the target color in the imaging module.

[0095] In this embodiment, after identifying the target object being gazed at by the user, color analysis is performed on the identified target object to check whether it contains obstacle colors that are difficult for the user to identify. Once it is found that the target color belongs to the category of obstacle colors, corresponding prompt information is immediately displayed in the imaging module. This prompt information may include the color name, code, or other forms of color representation, designed to help the user accurately identify and understand these colors.

[0096] This application provides targeted color blindness assistance prompts based on the type of color blindness, which can adapt to different types of color blindness. Furthermore, by identifying the target object at the user's gaze point, it provides targeted color assistance prompts for the object being gazed at. Compared to directly adding filters, this avoids the impact on the overall visual image and avoids the influence of the environment on color correction, thereby improving the robustness of color-assisted recognition.

[0097] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 , Figure 3 This is a schematic diagram of the implementation process involved in the second embodiment of the color blindness-assisted identification method of this application, as shown below. Figure 3 As shown, in one feasible implementation, step S40 above may include steps S41 to S43:

[0098] Step S41: Identify the outline position of the target color;

[0099] In this embodiment, after determining that there are target colors belonging to the obstacle colors in the target object, the head-mounted display device will further analyze the scene image to accurately identify the outline positions of these target colors.

[0100] Step S42: Generate an overlay layer in the imaging module at the same position as the contour position;

[0101] In this embodiment, based on the identified target color outline position, the head-mounted display device generates an overlay layer in its imaging module that completely overlaps with it. This overlay layer is transparent, but it is superimposed on the original scene image to add additional prompt information without affecting the user's observation of the overall scene.

[0102] Step S43: Display the prompt information corresponding to the target color in the overlay layer.

[0103] In this embodiment, within the generated overlay layer, the head-mounted display device accurately displays corresponding prompts at the locations of the target colors. These prompts may include color names, codes, color blocks, or other forms of color representation, designed to help users accurately identify and understand these obstacle colors. The display of the prompts should be both clear and intuitive to ensure that users can easily obtain the information they need.

[0104] In one feasible implementation, step S43 may include steps A10 to A20:

[0105] Step A10: Obtain the preset text information corresponding to the target color;

[0106] In this embodiment, a preset database or mapping table exists within the head-mounted display device, which records various obstacle colors and their corresponding text information. This text information may include the standard name of the color, color code (such as hexadecimal color code), a brief description of the color attribute, etc. When the device recognizes a target color in the scene, it searches for and retrieves the corresponding text information in the database based on the specific value of that color.

[0107] Step A20: Display the preset text information in the overlay layer.

[0108] In this embodiment, after obtaining the preset text information corresponding to the target color, the head-mounted display device directly displays this preset text information in the previously generated overlay layer. The display position of the preset text information should correspond to the outline position of the target color to ensure that the user can intuitively see the association between the color and its corresponding information. At the same time, the font size, color contrast, etc. of the preset text information should also adapt to the color of the target object and the ambient color to ensure clear readability under various lighting conditions.

[0109] For example, if a user wearing an advanced head-mounted display is looking at a red object and that user is red-green colorblind, they will not be able to accurately identify red. To address this, the head-mounted display has pre-determined the user's colorblindness type (red-green colorblindness) using a professional color vision testing module and stores this information in the user's profile.

[0110] When a user focuses their gaze on a red object, the device's built-in precision camera and image recognition algorithm immediately activate. The camera captures the user's eye movements, while the image recognition algorithm performs deep analysis of the scene image to accurately identify the object the user is focusing on and its color attributes. Once it is confirmed that the user is looking at a red object and cannot recognize it due to red-green color blindness, the device immediately activates an auxiliary prompt mechanism.

[0111] At the location of the red object, the device generates a transparent overlay layer. This layer does not obstruct the user's overall view of the object but provides essential information at key locations. Within this layer, the device clearly and accurately marks the object's color attribute in red, while simultaneously outlining the red object using lines that match the preset text color. This design aims to enable users to clearly identify the marked area as red through a combination of visual guidance and preset text information.

[0112] Through the aforementioned intelligent and precise auxiliary functions, head-mounted display devices effectively solve the color recognition difficulties faced by users with red-green color blindness, enhancing their ability to acquire information in complex visual environments. The application of this technology not only reflects the humanistic care inherent in technology but also provides colorblind users with a more equitable and barrier-free visual experience.

[0113] In addition, head-mounted display devices can also mark unrecognizable areas with lines and then display the color of the current area outside the area using preset text, thereby avoiding the preset text from obscuring other content or details in the area.

[0114] In this embodiment, by displaying unrecognizable colors as text, colorblind users can be assisted in accurately identifying the color at the target location.

[0115] In another feasible implementation, step S43 may include steps B10 to B13:

[0116] Step B10: Obtain the preset color mapping relationship corresponding to the color blindness assistance type;

[0117] In this embodiment, the head-mounted display device has multiple preset color blindness assistance types, each corresponding to a specific set of color mapping relationships. These mapping relationships are designed based on the visual characteristics of users with different color blindness types, aiming to map colors that are difficult to distinguish into colors that are easy to distinguish. When the head-mounted display device recognizes the user's color blindness type, it selects the corresponding color mapping relationship from the preset mapping relationships.

[0118] Step B20: Determine the auxiliary color corresponding to the target color based on the color mapping relationship;

[0119] In this embodiment, after obtaining the color mapping relationship, the head-mounted display device will find the corresponding auxiliary color in the mapping relationship based on the specific value of the target color.

[0120] Step B30: Display the auxiliary color in the overlay layer.

[0121] In this embodiment, after determining the auxiliary color, the head-mounted display device will display the target color in the form of the auxiliary color within the previously generated overlay layer. This allows colorblind users to identify and understand the target color in the scene using the auxiliary color. To enhance readability and intuitiveness, the display position of the auxiliary color should correspond to the outline position of the target color, and the brightness and contrast of the auxiliary color also need to be optimized.

[0122] For example, during a user's visual interaction while wearing a head-mounted display device, if the user's gaze is focused on red and green objects in the scene, and the user has been diagnosed with red-green color blindness in a previous test, making it impossible to effectively distinguish between the two colors, the head-mounted display device will perform a series of precise image recognition and auxiliary prompting operations based on the user's specific color vision impairment information, namely red-green color blindness.

[0123] The device first captures an image of the current scene using its built-in high-precision camera, and then uses advanced image recognition algorithms to perform a detailed analysis of the area the user is looking at. Once it detects that the user is focusing on red and green objects, and based on color vision impairment information in the user's profile, the device immediately determines that the user cannot accurately distinguish these two colors, and then triggers a preset auxiliary prompt function.

[0124] To effectively assist users in distinguishing between red and green objects, the head-mounted display generates a semi-transparent auxiliary layer over the target object's area. This layer is designed with colorblind users' visual characteristics in mind, using a fill color that contrasts sharply with colors the user cannot distinguish. For example, yellow is used to cover red areas, or other colors that do not conflict with the user's color vision impairment are chosen to mark green areas. This color filling strategy aims to help users indirectly infer the original object's color attributes based on the difference between the layer's color and the object's actual color by enhancing contrast.

[0125] Furthermore, the auxiliary layer design incorporates edge recognition and contour delineation technology to ensure that users can clearly see the boundaries of the covered area, thereby accurately understanding the color information indicated by the layer. In this way, the head-mounted display device not only provides color recognition assistance but also ensures the consistency and accuracy of the user's visual experience.

[0126] In this embodiment, by using an auxiliary color to cover the target color, it is possible to help colorblind users distinguish colors that are difficult to differentiate, and the experience is relatively imperceptible.

[0127] The above are only two feasible implementations of step S43 provided in this embodiment. This embodiment does not specifically limit the specific implementation of step S43.

[0128] This embodiment provides a colorblindness-assisted identification method that can overlay a layer on a target object and display text or colors for color-assisted identification in the overlay layer to assist colorblind users in identifying colors.

[0129] Based on the first and / or second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to the first and / or second embodiments described above can be referred to the above description and will not be repeated hereafter. Furthermore, after step S30 above, the method may further include steps C10 to C50:

[0130] Step C10: Identify the target outline of the target object;

[0131] In this embodiment, after identifying the target object, the head-mounted display device further analyzes the scene image to accurately identify the outline of the target object. By identifying the target outline, the head-mounted display device can accurately determine the position and shape of the target object in the scene image.

[0132] Step C20: Crop the scene image according to the target contour to obtain the target image of the target object;

[0133] In this embodiment, based on the identified target contour, the head-mounted display device crops the scene image to extract the image of the area where the target object is located. This cropped image is called the target image, and it contains the complete shape and color information of the target object.

[0134] Step C30: Obtain the color value of each pixel in the target image;

[0135] In this embodiment, the target image is analyzed pixel by pixel to obtain the color value of each pixel. The color value typically includes the values ​​of the three channels: red, green, and blue (RGB), or it can be represented by HSV (hue, saturation, brightness). These values ​​together determine the color of the pixel.

[0136] Step C40: Divide the target pixels whose color values ​​belong to the same color range into the same region to obtain at least one color region;

[0137] In this embodiment, pixels with similar color values ​​in the target image are divided into the same region according to a preset color range or threshold.

[0138] Step C50: Use the color corresponding to the color region as the color of the target object.

[0139] In this embodiment, after dividing the color regions, the color characteristics of each color region are analyzed, and the color corresponding to each region is determined. These colors will be regarded as the color characteristics of the target object.

[0140] Through the above steps, not only can target objects be accurately identified in scene images, but also the color characteristics of target objects can be analyzed in depth, providing users with more accurate color recognition assistance.

[0141] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the color blindness-assisted identification method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0142] This application also provides a colorblindness-assisted identification device; please refer to... Figure 4 The device is installed in a head-mounted display device, which includes an inner camera, an outer camera, and an imaging module; the color blindness-assisted recognition device includes:

[0143] The type determination module 10 is used to determine the color blindness assistance type of the head-mounted display device, wherein the color blindness assistance type corresponds to at least one obstacle color;

[0144] The camera module 20 is used to identify the gaze point of the target user through the inner camera and to acquire scene images through the outer camera;

[0145] Object recognition module 30 is used to identify the target object corresponding to the gaze point in the scene image;

[0146] The color blindness assistance module 40 is used to display prompt information corresponding to the target color in the imaging module if there is a target color in the target object that belongs to the color of the obstacle.

[0147] Optionally, the type determination module 10 is also used for:

[0148] When the color blindness assistance type of the head-mounted display device is initialized, multiple preset test images are displayed through the imaging module;

[0149] The comparison result is obtained by comparing the feedback information received based on the test image with the color label of the test image;

[0150] If the comparison result shows that the feedback information is different from the color label, then the color label is taken as the obstacle color;

[0151] Determine the color blindness assistance type corresponding to the color of the obstacle in a preset color blindness assistance type database.

[0152] Optionally, the camera module 20 is also used for:

[0153] The target user's eye movement video is acquired through the inner camera;

[0154] Identify the pupil direction in the eye video;

[0155] The gaze point of the target user is determined based on the pupil direction.

[0156] Optionally, the colorblindness assist module 40 is also used for:

[0157] Identify the outline position of the target color;

[0158] An overlay layer is generated in the imaging module at the same position as the contour.

[0159] The overlay layer displays a prompt message corresponding to the target color.

[0160] Optionally, the colorblindness assist module 40 is also used for:

[0161] Obtain the preset text information corresponding to the target color;

[0162] The preset text information is displayed in the overlay layer.

[0163] Optionally, the colorblindness assist module 40 is also used for:

[0164] Obtain the color mapping relationship corresponding to the preset color blindness assistance type;

[0165] Determine the auxiliary color corresponding to the target color based on the color mapping relationship;

[0166] The auxiliary color is displayed in the overlay layer.

[0167] Optionally, the colorblindness-assisted identification device is also used for:

[0168] Identify the target outline of the target object;

[0169] The scene image is cropped according to the target contour to obtain the target image of the target object;

[0170] Obtain the color value of each pixel in the target image;

[0171] Target pixels whose color values ​​belong to the same color range are divided into the same region to obtain at least one color region;

[0172] The color corresponding to the color region is used as the color of the target object.

[0173] The colorblindness-assisted identification device provided in this application, employing the colorblindness-assisted identification method in the above embodiments, can solve the technical problem of how to improve the robustness of color-assisted identification. Compared with the prior art, the beneficial effects of the colorblindness-assisted identification device provided in this application are the same as those of the colorblindness-assisted identification method provided in the above embodiments, and other technical features in the colorblindness-assisted identification device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0174] This application provides a head-mounted display device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the color blindness-assisted recognition method in Embodiment 1 above.

[0175] The head-mounted display device in this application embodiment may include, but is not limited to, head-mounted display devices such as Mixed Reality (MR) devices (e.g., MR glasses or MR helmets), Augmented Reality (AR) devices (e.g., AR glasses or AR helmets), Virtual Reality (VR) devices (e.g., VR glasses or VR helmets), Extended Reality (XR) devices, or some combination thereof.

[0176] like Figure 5 As shown, the head-mounted display device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the head-mounted display device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the head-mounted display device to communicate wirelessly or wiredly with other devices to exchange data. Although head-mounted display devices with various systems are shown in the figures, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0177] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0178] The head-mounted display device provided in this application employs the color blindness-assisted identification method in the above embodiments, which can solve the technical problem of how to improve the robustness of color-assisted identification. Compared with the prior art, the beneficial effects of the head-mounted display device provided in this application are the same as those of the color blindness-assisted identification method provided in the above embodiments, and other technical features in this head-mounted display device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0179] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0180] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0181] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the color blindness-assisted identification method in the above embodiments.

[0182] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0183] The aforementioned computer-readable storage medium may be included in the head-mounted display device; or it may exist independently and not assembled into the head-mounted display device.

[0184] The aforementioned computer-readable storage medium carries one or more programs that, when executed by the head-mounted display device, cause the head-mounted display device to:

[0185] The color blindness assistance type of the head-mounted display device is determined, wherein the color blindness assistance type corresponds to at least one obstacle color; the gaze point of the target user is identified through the inner camera, and a scene image is acquired through the outer camera; the target object corresponding to the gaze point in the scene image is identified; if there is a target color belonging to the obstacle color in the target object, the prompt information corresponding to the target color is displayed in the imaging module.

[0186] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0187] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0188] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0189] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described color blindness-assisted identification method, thereby solving the technical problem of how to improve the robustness of color-assisted identification. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the color blindness-assisted identification method provided in the above embodiments, and will not be repeated here.

[0190] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the color blindness-assisted identification method described above.

[0191] The computer program product provided in this application can solve the technical problem of how to improve the robustness of color-assisted recognition. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the color blindness-assisted recognition method provided in the above embodiments, and will not be repeated here.

[0192] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A colorblindness-assisted identification method, characterized in that, The method is applied to a head-mounted display device, the head-mounted display device comprising: an inner camera, an outer camera, and an imaging module; the method comprises: Determine the color blindness assistance type of the head-mounted display device, wherein the color blindness assistance type corresponds to at least one obstacle color; The inner camera identifies the target user's gaze point, and the outer camera acquires scene images. Identify the target object corresponding to the gaze point in the scene image; If the target object contains a target color that corresponds to the obstacle color, then the imaging module displays a prompt message corresponding to the target color.

2. The colorblindness-assisted identification method as described in claim 1, characterized in that, The step of determining the color blindness assistance type of the head-mounted display device includes: When the color blindness assistance type of the head-mounted display device is initialized, multiple preset test images are displayed through the imaging module; The comparison result is obtained by comparing the feedback information received based on the test image with the color label of the test image; If the comparison result shows that the feedback information is different from the color label, then the color label is taken as the obstacle color; Determine the color blindness assistance type corresponding to the color of the obstacle in a preset color blindness assistance type database.

3. The colorblindness-assisted identification method as described in claim 1, characterized in that, The step of identifying the target user's gaze point through the inner camera includes: The target user's eye movement video is acquired through the inner camera; Identify the pupil direction in the eye video; The gaze point of the target user is determined based on the pupil direction.

4. The colorblindness-assisted identification method as described in claim 1, characterized in that, The step of displaying the prompt information corresponding to the target color in the imaging module includes: Identify the outline position of the target color; An overlay layer is generated in the imaging module at the same position as the contour. The overlay layer displays a prompt message corresponding to the target color.

5. The colorblindness-assisted identification method as described in claim 4, characterized in that, The step of displaying the prompt information corresponding to the target color in the overlay layer includes: Obtain the preset text information corresponding to the target color; The preset text information is displayed in the overlay layer.

6. The colorblindness-assisted identification method as described in claim 4, characterized in that, The step of displaying the prompt information corresponding to the target color in the overlay layer further includes: Obtain the color mapping relationship corresponding to the preset color blindness assistance type; Determine the auxiliary color corresponding to the target color based on the color mapping relationship; The auxiliary color is displayed in the overlay layer.

7. The colorblindness-assisted identification method as described in any one of claims 1 to 6, characterized in that, After the step of identifying the target object corresponding to the gaze point in the scene image, the method further includes: Identify the target outline of the target object; The scene image is cropped according to the target contour to obtain the target image of the target object; Obtain the color value of each pixel in the target image; Target pixels whose color values ​​belong to the same color range are divided into the same region to obtain at least one color region; The color corresponding to the color region is used as the color of the target object.

8. A colorblindness-assisted identification device, characterized in that, The device is disposed on a head-mounted display device, the head-mounted display device including: an inner camera, an outer camera, and an imaging module; the device includes: A type determination module is used to determine the color blindness assistance type of the head-mounted display device, wherein the color blindness assistance type corresponds to at least one obstacle color; The camera module is used to identify the gaze point of the target user through the inner camera and to acquire scene images through the outer camera; The object recognition module is used to identify the target object corresponding to the gaze point in the scene image; A colorblindness assistance module is used to display prompt information corresponding to the target color in the imaging module if there is a target color in the target object that belongs to the color of the obstacle.

9. A head-mounted display device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the colorblindness-assisted identification method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the color blindness-assisted identification method as described in any one of claims 1 to 7.

Images