Visual aid method, system, head-mounted display device, storage medium and product

Abstract

The application discloses a visual auxiliary method, system, head-mounted display device, storage medium and product, relates to the technical field of head-mounted display devices, and discloses a visual auxiliary method applied to a head-mounted display device, wherein the head-mounted display device comprises vibration prompt points, and the method comprises the following steps: image recognition and distance recognition are performed on user environment objects, and the target direction of a target object is determined from the user environment objects; a first target vibration prompt point corresponding to the target direction is controlled to generate a first vibration prompt, wherein the first vibration prompt is used for indicating the target direction. The application can help the wearer of the head-mounted display device to better perceive the surrounding environment.

Description

Technical Field

[0001] This application relates to the field of head-mounted display technology, and more particularly to visual assistance methods, systems, head-mounted display devices, storage media, and products. Background Technology

[0002] With the rapid development of AR (Augmented Reality) and VR (Virtual Reality) technologies, AR / VR devices and other head-mounted display devices have gradually integrated into people's daily lives and become an important part of modern technological experiences.

[0003] Although AR / VR technology has brought people a brand-new visual experience, people with visual impairments have difficulty recognizing and perceiving their surroundings and cannot enjoy the convenience and help that such head-mounted display devices bring to their lives. At the same time, ordinary users will also experience inconvenience in perceiving their surroundings when using head-mounted display devices in low light conditions.

[0004] In summary, how to help wearers of head-mounted display devices better perceive their surroundings has become a pressing technical problem that needs to be solved in this field. Summary of the Invention

[0005] The main purpose of this application is to provide a visual assistance method, system, head-mounted display device, storage medium, and product designed to help wearers of head-mounted display devices better perceive their surroundings.

[0006] To achieve the above objectives, this application proposes a visual assistance method applied to a head-mounted display device, the head-mounted display device including vibration alert points, and the visual assistance method comprising:

[0007] Perform image recognition and distance recognition on objects in the user environment, and determine the target location of the target object from the objects in the user environment;

[0008] Control the first target vibration prompt point, which corresponds to the target location, among the vibration prompt points to generate a first vibration prompt, wherein the first vibration prompt is used to indicate the target location.

[0009] In one embodiment, the method further includes:

[0010] Determine the current working mode;

[0011] If the current working mode is passive mode, the first target object is determined according to the user's voice command;

[0012] The first target object is used as the target object, and the step of determining the target orientation of the target object from the objects in the user environment is performed.

[0013] In one embodiment, the step of determining the target location of the target object from the objects in the user environment includes:

[0014] If no target object is identified in the user environment objects, the rotation vibration prompt point in each of the vibration prompt points is controlled to generate a second vibration prompt, wherein the second vibration prompt is used to instruct the user to rotate to adjust the screen acquisition angle of the head-mounted display device, so as to obtain a new user environment object in the adjusted screen acquisition angle.

[0015] When the target object is included in a new user environment object, determine the target orientation of the target object.

[0016] In one embodiment, after the step of controlling the first target vibration alert point corresponding to the target orientation among the vibration alert points to generate a first vibration alert, the method further includes:

[0017] Generate a guide route for the current user to move from the current location to the target location;

[0018] A second target vibration prompt point corresponding to the guidance route is determined from the vibration prompt points, and the second target vibration prompt point is controlled to generate a third vibration prompt, wherein the third vibration prompt is used to guide the user's movement route.

[0019] In one embodiment, the step of determining the current operating mode is followed by:

[0020] If the current working mode is active mode, then objects in the user environment with a recognition distance less than a preset safety threshold are identified as the second target objects.

[0021] The second target object is used as the target object, and the step of determining the target orientation of the target object from the user environment objects is performed.

[0022] In one embodiment, the step of determining the current operating mode is followed by:

[0023] If the current working mode is a mixed mode, then the first target orientation of the first target object and the second target orientation of the second target object are determined from the objects in the user environment.

[0024] The third target vibration prompt point corresponding to the first target location among the vibration prompt points is controlled to generate a fourth vibration prompt, and the fourth target vibration prompt point corresponding to the second target location among the vibration prompt points is controlled to generate a fifth vibration prompt, wherein the vibration frequencies of the fourth vibration prompt and the fifth vibration prompt are different.

[0025] Furthermore, to achieve the above objectives, this application also proposes a visual assistance system for use in a head-mounted display device, the head-mounted display device including vibration alert points, and the visual assistance system comprising:

[0026] The object recognition module is used to perform image recognition and distance recognition on objects in the user environment, and to determine the target location of the target object from the objects in the user environment.

[0027] A vibration module is used to control a first target vibration prompt point corresponding to the target orientation among the vibration prompt points to generate a first vibration prompt, wherein the first vibration prompt is used to indicate the target orientation.

[0028] In addition, to achieve the above objectives, this application also proposes a head-mounted display device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the visual assistance method as described above.

[0029] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the visual assistance method described above.

[0030] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the visual assistance method described above.

[0031] This application proposes a visual assistance method applied to a head-mounted display device, which includes various vibration alert points. First, the head-mounted display device performs image recognition and distance recognition on objects in the user's environment to determine the user's surroundings and the distances between the user and objects within that environment. Then, it determines the target orientation of a target object among the identified objects in the user's environment. The target object can be determined based on the user's intent and / or preset object recognition rules within the device. Based on the target orientation of the target object, the device controls a first target vibration alert point corresponding to that target orientation to generate a first vibration alert. This first vibration alert occurs at a specific frequency and pattern, enabling the user to clearly perceive the target orientation of the target object.

[0032] In summary, this application utilizes vibration alerts from head-mounted display devices, combined with image recognition and distance recognition technologies, to achieve precise indication of the location of target objects. Users can understand and perceive objects in their surroundings simply by feeling the vibration alerts from different locations on the head-mounted display device. This not only expands the application scope of head-mounted display devices but also helps wearers better perceive their surroundings. Attached Figure Description

[0033] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0034] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a flowchart illustrating an embodiment of the visual assistance method of this application.

[0036] Figure 2 This is a schematic diagram of the structure of the head-mounted display device provided in Embodiment 1 of the visual assistance method of this application;

[0037] Figure 3 This is another structural schematic diagram of the head-mounted display device provided in Embodiment 1 of the visual assistance method of this application;

[0038] Figure 4 This is a schematic diagram of the module structure of the visual assistance system according to an embodiment of this application;

[0039] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the visual assistance method in the embodiments of this application.

[0040] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0041] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0042] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0043] The main solution of this application embodiment is: to provide a head-mounted display device including various vibration prompt points, to perform image recognition and distance recognition on objects in the user's environment, and to determine the target orientation of the target object from the objects in the user's environment; to control the first target vibration prompt point corresponding to the target orientation among the various vibration prompt points to generate a first vibration prompt, wherein the first vibration prompt is used to indicate the target orientation.

[0044] Although AR / VR technology has brought people a brand-new visual experience, people with visual impairments have difficulty recognizing and perceiving their surroundings and cannot enjoy the convenience and help that such head-mounted display devices bring to their lives. At the same time, ordinary users will also experience inconvenience in perceiving their surroundings when using head-mounted display devices in low light conditions.

[0045] This application provides a solution that utilizes vibration alerts from head-mounted display devices, combined with image recognition and distance recognition technologies, to achieve precise indication of the location of target objects. Users can understand and perceive objects in their surroundings simply by feeling the vibration alerts from different locations on the head-mounted display device. This not only expands the application scope of head-mounted display devices but also helps wearers better perceive their surroundings.

[0046] It should be noted that the execution subject of this embodiment is a head-mounted display device. In this embodiment, the head-mounted display device includes, but is not limited to, Mixed Reality (MR) devices (e.g., MR glasses or MR helmets), Augmented Reality (AR) devices (e.g., AR glasses or AR helmets), Virtual Reality (VR) devices (e.g., VR glasses or VR helmets), Extended Reality (XR) devices, or some combination thereof. The following description uses a head-mounted display device as the execution subject to illustrate this embodiment and the subsequent embodiments.

[0047] Based on this, the embodiments of this application provide a visual assistance method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the visual assistance method of this application.

[0048] In this embodiment, the method is applied to a head-mounted display device, which includes vibration alert points. The visual assistance method includes steps S10 to S20:

[0049] Step S10: Perform image recognition and distance recognition on objects in the user environment, and determine the target location of the target object from the objects in the user environment.

[0050] Head-mounted display devices capture images of the user's environment using their built-in cameras or other sensors. They then use image recognition technology to identify objects in the captured images and calculate the distances between these objects and the wearer. Based on the user's intent and / or the object recognition rules preset in the device, they determine the target object that the user is currently interested in or needs to find from the identified objects in the user's environment and determine the target location of that target object.

[0051] Step S20: Control the first target vibration prompt point corresponding to the target orientation among the vibration prompt points to generate a first vibration prompt, wherein the first vibration prompt is used to indicate the target orientation.

[0052] It should be noted that the vibration alerts are integrated into the head-mounted display device and can transmit information to the wearer through physical vibration. Specifically, the head-mounted display device contains multiple vibration alerts that indicate different directions. These multiple vibration alerts are distributed in different positions on the device. When any of the vibration alerts indicating a direction vibrates, it means that the target object is in the direction indicated by that vibration alert.

[0053] Based on the determined target location of the target object, the head-mounted display device controls the first target vibration prompt point corresponding to the target location to generate a first vibration prompt. The vibration prompt is performed at a specific frequency so that the user can clearly perceive the position of the target object based on the vibration.

[0054] Furthermore, in one feasible embodiment, the structural block diagram of the head-mounted display device is as follows: Figure 2 As shown, the head-mounted display device includes a visual sensor unit and a system set unit. The visual sensor unit specifically includes four cameras, RC1, RC2, LC1, and LC2, used to capture images of the user's environment from multiple angles, as well as image recognition and distance recognition. The configured System set unit includes a central processing unit (responsible for data processing and control), a power supply (providing stable and reliable power support for the entire device), and network and data transmission components (realizing communication and data exchange between the system and the external environment). The System set unit provides users with stable, efficient, and powerful computing and information processing capabilities through the efficient coordination of these components.

[0055] Another structural block diagram of a head-mounted display device is as follows: Figure 3As shown, the head-mounted display device includes a linear motor vibration unit composed of nine vibration indicator points: LT, LB, RT, RB, MP, LF, LR, RF, and RR. Specifically, in a living environment, any point in space can be decomposed into a set of spatial coordinates. When a user is in any position, they place themselves at the (0, 0, 0) coordinate origin. Through the action of the nine vibration indicator points, the head-mounted display device in this embodiment can achieve 4Dof (Four Degrees of Freedom) position indication. Among the vibration indicator points, LF indicates the left-front direction, LR the left-rear direction, LT the left-up direction, LB the left-down direction, RT the right-up direction, RB the right-down direction, RF the right-front direction, RR the right-rear direction, and MP is the rotation and stop guidance system. By importing the MP guidance system, when the MP is matched with the above direction systems, the following can be achieved: LF+MP instructs the user to rotate in the left-forward direction, LR+MP instructs the user to rotate in the left-rear direction, RF+MP instructs the user to rotate in the right-forward direction, RR+MP instructs the user to rotate in the right-rear direction, LF+LT instructs the left-forward and upward direction, and LF+RF instructs the direction directly in front. Other direction combinations can also be matched according to actual needs. When the MP vibrates alone, it indicates that the action has stopped or the action has been completed.

[0056] It should be noted that in actual application scenarios, the device can of course be set with more or fewer vibration alerts to help users perceive the surrounding environment, and this embodiment does not make specific limitations on this.

[0057] In summary, this embodiment of the application utilizes the vibration prompts of the head-mounted display device, combined with image recognition and distance recognition technologies, to achieve accurate indication of the location of target objects. Users can understand and perceive objects in their surroundings simply by feeling the vibration prompts at different locations on the head-mounted display device. This not only expands the application scope of the head-mounted display device but also helps the wearer of the head-mounted display device to better perceive their surroundings.

[0058] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description and will not be repeated hereafter. Furthermore, the visual assistance method may further include steps A10 to A30:

[0059] Step A10: Determine the current working mode;

[0060] When the head-mounted display device is first powered on, initialized, or when the wearer issues a command to switch modes, the device will first determine the working mode in order to provide the corresponding functions according to the wearer's needs.

[0061] Step A20: If the current working mode is passive mode, then determine the first target object according to the user's voice command;

[0062] If the current working mode is determined to be passive mode, the head-mounted display device will enter a specific operation process. In this mode, the device will wait for and listen to the wearer's voice commands. Upon receiving the wearer's voice commands, the device will immediately activate the voice recognition function to parse the command content in order to determine the first target object specified by the user.

[0063] Step A30: Select the first target object as the target object and perform the step of determining the target orientation of the target object from objects in the user environment.

[0064] Once the first target object is successfully identified, the head-mounted display device will regard it as the primary object of interest in the current user environment, i.e., the target object. Subsequently, the device will determine the precise location of the target object from the user environment, i.e., the target orientation, and perform subsequent operation steps based on the target orientation.

[0065] In this way, head-mounted display devices can accurately identify and locate target objects specified by users in passive mode based on their voice commands, thereby providing users with a more convenient and intelligent interactive experience and making daily life easier.

[0066] Based on this, step S10, "determining the target location of the target object from objects in the user's environment," includes:

[0067] Step S101: If no target object is identified in the user environment, the rotation vibration prompt point in each vibration prompt point is controlled to generate a second vibration prompt. The second vibration prompt is used to instruct the user to rotate to adjust the screen acquisition angle of the head-mounted display device and obtain a new user environment object in the adjusted screen acquisition angle.

[0068] It should be noted that, in addition to the vibration alerts used to indicate orientation, the head-mounted display device also includes vibration alerts used to indicate rotation, i.e., rotation vibration alerts.

[0069] When the head-mounted display fails to identify the target object specified by the user in the current user environment, the rotation vibration point among the vibration prompts generates a second vibration prompt. The purpose of this vibration prompt is to convey a clear instruction to the user that the body / head needs to be rotated to adjust the viewing angle of the head-mounted display. By rotating the direction, the user can change the image captured by the head-mounted display, thereby obtaining a new, adjusted viewing angle, which may contain the target object that was not previously identified.

[0070] Step S102: When the target object is contained in the new user environment object, determine the target orientation of the target object.

[0071] After the user rotates their body / head according to the vibration prompt to adjust the device's image capture angle, the head-mounted display device will reanalyze the new user environment objects. If the target object is successfully identified in the new user environment objects, the target object's location will be determined immediately.

[0072] For example, when the head-mounted display is in passive mode, it listens to the user's voice commands in real time and prompts the user to perform actions based on the voice commands. In this mode, the visual sensor unit is in standby mode by default. When the user wakes it up by voice, it performs image recognition and distance recognition on objects in the user's environment. When the user uses voice commands to search for objects or perform actions, it notifies the user of the direction of movement through vibration of the vibration prompt point.

[0073] Specifically, when a user wakes up the head-mounted display device with their voice and requests to pick up object A, the device first identifies the object based on the environment monitored by the visual sensor unit. When the memory processor finds the object A requested by the user, it will continuously vibrate the vibration prompt point corresponding to the location of object A to remind the user of the relative direction of the object, and calculate the distance between the user's hand and the object in real time. When the user touches the object, it will notify the user that the picking activity has been completed by vibrating the MP separately.

[0074] If the memory processor does not find the object A requested by the user, it will continuously vibrate the MP to remind the user to rotate their body. When object A is exposed in the device's capture screen, the MP will stop vibrating to inform the user that the search activity is complete. The MP will also continuously vibrate the vibration prompt point corresponding to the location of object A to remind the user of the relative direction of the object, and will calculate the distance between the user's hand and the object in real time. When the user touches the object, the MP will vibrate to inform the user that the picking activity is complete.

[0075] In one feasible embodiment, after step S20, the visual assistance method may further include steps S30 to S40:

[0076] Step S30: Generate a guide route for the current user to move from the current location to the target location;

[0077] Once the head-mounted display device successfully determines the target location of the target object, it can generate a guide route from the user's current location to the target location based on the user's current location and the target object's location. This guide route can be a straight line, a curve, or any other path that can efficiently and accurately guide the user to the target object without touching any dangerous obstacles.

[0078] Step S40: Determine the second target vibration prompt point corresponding to the guidance route from the vibration prompt points, and control the second target vibration prompt point to generate a third vibration prompt, wherein the third vibration prompt is used to guide the user's movement route.

[0079] After generating the guidance route, the head-mounted display device will determine the second target vibration cue point corresponding to this guidance route from the various vibration cue points. These vibration cue points provide route guidance information to the user by generating a third vibration cue.

[0080] It should be noted that the device will control these second target vibration prompt points to generate a third vibration prompt. This third vibration prompt is different from the vibration used to indicate the target location of the target object. It is specifically used to guide the user's movement route. Specifically, different frequencies, intensities or modes of vibration can be set so that the user can distinguish the vibration of the location indication from the vibration of the movement route guidance.

[0081] In this way, head-mounted display devices can not only provide users with precise location information of target objects, but also provide users with real-time, dynamic navigation and movement support through vibration prompts, thereby improving the overall user experience and convenience.

[0082] It should also be noted that in practical application scenarios, in order to meet the personalized needs of different users, the functions of the head-mounted display device can be flexibly adjusted and optimized. For example, in passive mode, when a user instructs to find a target object through voice commands, and the device successfully identifies and determines the target object's location, the device can be set to only provide a first vibration prompt indicating the target's location, without generating a third vibration prompt to guide the user to the target object's movement path. In this mode, whenever the user moves, the device will synchronously update the vibration prompt point that generated the first vibration prompt, ensuring that the user can always accurately perceive the direction of the target object and autonomously plan the movement path according to their own needs.

[0083] In addition, the device can also have an intelligent judgment function, which can intelligently determine whether to activate a third vibration alert to provide detailed movement route guidance based on the distance between the user and the target object and the complexity of the surrounding environment. For example, in situations where the distance is far or the environment is complex, the device will automatically activate the third vibration alert to help the user reach the target object more smoothly.

[0084] In addition, to meet users' personalized needs for vibration alerts, users can also specify the specific type of vibration alert via voice commands.

[0085] In one feasible embodiment, after step A10, the visual assistance method may further include steps A40 to A50:

[0086] Step A40: If the current working mode is active mode, then objects in the user environment whose recognition distance is less than the preset safety threshold are identified as the second target objects.

[0087] When the head-mounted display device is in active working mode, the device will automatically perform distance recognition on the objects in the user's environment. If the recognition distance of an object is less than the preset safe distance threshold, the object will be judged as a dangerous obstacle and marked by the device as a second target object.

[0088] Step A50: Select the second target object as the target object and perform the step of determining the target orientation of the target object from objects in the user environment.

[0089] The identified dangerous obstacle, i.e., the second target object, is treated as the target object that the user needs to focus on at present. The target location of the target object is determined from the objects in the user's environment, thereby helping the user to effectively avoid this potential dangerous obstacle and ensuring the user's safety during movement.

[0090] It should be noted that after the step of determining the target location of the target object from the objects in the user's environment is performed, the steps following this step are also performed. The steps to be performed are as described in the above embodiments and will not be repeated here.

[0091] In this way, when the head-mounted display device is in active working mode, it can identify and warn users of dangerous obstacles in the environment in real time, providing users with a safer and more convenient visual assistance experience.

[0092] For example, when the head-mounted display device is in active mode, the device constantly reminds the user of their behavior based on the environment. It monitors the user's environment in real time through a visual sensor unit, performs image recognition and distance recognition on objects in the user's environment, and notifies the user to take cover when it detects that the user is at a passive or active dangerous distance from other objects.

[0093] In one feasible embodiment, after step A10, the visual assistance method may further include steps A60 to A70:

[0094] Step A60: If the current working mode is mixed mode, determine the first target position of the first target object and the second target position of the second target object from the objects in the user environment.

[0095] When the head-mounted display device is in hybrid working mode, it performs a comprehensive analysis of objects in the user's environment to determine the first target location of the first target object and the second target location of the second target object. The first target object is the object to be focused on based on the user's voice command, and the second target object is a nearby dangerous obstacle object identified in the current environment.

[0096] Step A70: Control the third target vibration prompt point corresponding to the first target location among all vibration prompt points to generate a fourth vibration prompt, and control the fourth target vibration prompt point corresponding to the second target location among all vibration prompt points to generate a fifth vibration prompt, wherein the vibration frequencies of the fourth vibration prompt and the fifth vibration prompt are different.

[0097] After determining the location of each target object, the third target vibration prompt point corresponding to the first target location is controlled to generate a fourth vibration prompt, and at the same time, the fourth target vibration prompt point corresponding to the second target location is controlled to generate a fifth vibration prompt.

[0098] The fourth vibration alert has a different vibration frequency than the fifth vibration alert, so that users can quickly identify different target locations by sensing different vibration frequencies, thereby making more accurate reactions and decisions.

[0099] Thus, for people with visual impairments or when the wearer is in low light, the device can help them better understand and perceive their surroundings, reduce safety risks in their activities and daily life, and allow them to enjoy the convenience and assistance brought by head-mounted display devices.

[0100] Furthermore, in one feasible embodiment, the current operating mode of the head-mounted display device can also be a music environment mode, which is typically used when the user is in an environment where music is playing. In this mode, the device monitors the audio signals in the user's environment in real time and adjusts the vibration mode of each vibration prompt point according to the monitored audio signals, thereby giving the user a special musical atmosphere experience. This enhances the user's interaction with the audio rhythm through the head-mounted display device, allowing the user to fully experience the musical atmosphere and enrich their entertainment experience.

[0101] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the visual assistance method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0102] This application also provides a visual assistance system for a head-mounted display device. The head-mounted display device includes vibration alert points. Please refer to... Figure 4 Visual assistance systems include:

[0103] The object recognition module 10 is used to perform image recognition and distance recognition on objects in the user environment, and to determine the target location of the target object from the objects in the user environment.

[0104] The vibration module 20 is used to control the first target vibration prompt point corresponding to the target orientation among the vibration prompt points to generate a first vibration prompt, wherein the first vibration prompt is used to indicate the target orientation.

[0105] Optionally, the vision assistance system also includes a working mode determination module (not shown), which is used for:

[0106] Determine the current working mode;

[0107] If the current working mode is passive mode, the first target object is determined according to the user's voice command;

[0108] The first target object is selected as the target object, and the step of determining the target object's location from objects in the user's environment is performed.

[0109] Optionally, the object recognition module 10 is also used for:

[0110] If no target object is detected in the user's environment, the rotation vibration prompt point in each vibration prompt point is controlled to generate a second vibration prompt. The second vibration prompt is used to instruct the user to rotate to adjust the screen acquisition angle of the head-mounted display device and obtain a new user environment object in the adjusted screen acquisition angle.

[0111] When a target object is contained within a new user environment object, determine the target orientation of the target object.

[0112] Optionally, the vision assistance system also includes a motion guidance module (not shown), which is used for:

[0113] Generate a guide route for the user to move from the current location to the target location;

[0114] The second target vibration prompt point corresponding to the guidance route is determined from the various vibration prompt points, and the second target vibration prompt point is controlled to generate a third vibration prompt, wherein the third vibration prompt is used to guide the user's movement route.

[0115] Optionally, the working mode determination module is also used for:

[0116] If the current working mode is active mode, then objects in the user environment whose recognition distance is less than the preset safety threshold will be identified as the second target objects.

[0117] The second target object is selected as the target object, and the step of determining the target object's location from objects in the user's environment is performed.

[0118] Optionally, the working mode determination module is also used for:

[0119] If the current working mode is mixed mode, then determine the first target position of the first target object and the second target position of the second target object from the objects in the user environment.

[0120] Control the third target vibration alert point corresponding to the first target location among all vibration alert points to generate a fourth vibration alert, and control the fourth target vibration alert point corresponding to the second target location among all vibration alert points to generate a fifth vibration alert, wherein the vibration frequencies of the fourth vibration alert and the fifth vibration alert are different.

[0121] The visual assistance system provided in this application, employing the visual assistance method described in the above embodiments, can help wearers of head-mounted display devices better perceive their surroundings. Compared with the prior art, the beneficial effects of the visual assistance system provided in this application are the same as those of the visual assistance method provided in the above embodiments, and other technical features of the visual assistance system are the same as those disclosed in the visual assistance method of the above embodiments, and will not be repeated here.

[0122] This application provides a head-mounted display device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the visual assistance method in the first embodiment described above.

[0123] The following is for reference. Figure 5 The diagram illustrates a structural schematic suitable for implementing the head-mounted display device of the embodiments of this application. The head-mounted display device in the embodiments of this application may include, but is not limited to, Mixed Reality (MR) devices (e.g., MR glasses or MR helmets), Augmented Reality (AR) devices (e.g., AR glasses or AR helmets), Virtual Reality (VR) devices (e.g., VR glasses or VR helmets), Extended Reality (XR) devices, or some combination thereof, etc. Figure 5 The head-mounted display device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of this application.

[0124] like Figure 5As shown, the head-mounted display device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the head-mounted display device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the head-mounted display device to communicate wirelessly or wiredly with other devices to exchange data. Although head-mounted display devices with various systems are shown in the figures, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0125] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0126] The head-mounted display device provided in this application, employing the visual assistance method described in the above embodiments, can help the wearer of the head-mounted display device better perceive the surrounding environment. Compared with the prior art, the beneficial effects of the head-mounted display device provided in this application are the same as those of the visual assistance method provided in the above embodiments, and other technical features of this head-mounted display device are the same as those disclosed in the visual assistance method of the previous embodiment, and will not be repeated here.

[0127] It should be understood that the various parts disclosed in the embodiments of this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0128] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0129] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the visual assistance method in the above embodiments.

[0130] The computer-readable storage medium provided in this application embodiment may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0131] The aforementioned computer-readable storage medium may be included in the head-mounted display device; or it may exist independently and not assembled into the head-mounted display device.

[0132] The aforementioned computer-readable storage medium carries one or more programs that, when executed by the head-mounted display device, cause the head-mounted display device to: perform image recognition and distance recognition on objects in the user's environment, and determine the target orientation of a target object from the objects in the user's environment; and control a first target vibration prompt point corresponding to the target orientation among the vibration prompt points to generate a first vibration prompt, wherein the first vibration prompt is used to indicate the target orientation.

[0133] Computer program code for performing the operations of the embodiments of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0134] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0135] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0136] The readable storage medium provided in this application embodiment is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for performing the above-described visual assistance method, which can help the wearer of the head-mounted display device to better perceive the surrounding environment. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application embodiment are the same as the beneficial effects of the visual assistance method provided in the above embodiments, and will not be repeated here.

[0137] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the visual assistance method described above.

[0138] The computer program product provided in this application can help wearers of head-mounted display devices better perceive their surroundings. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the visual assistance methods provided in the above embodiments, and will not be repeated here.

[0139] The above are only some embodiments of this application and do not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A visual assistance method, characterized in that, The device is applied to a head-mounted display device, which includes vibration alert points distributed at different locations on the head-mounted display device, and each vibration alert point is used to indicate a different orientation; The method includes: Perform image recognition and distance recognition on objects in the user's environment; If the target object is not detected in the user environment objects, the rotation vibration prompt point among the vibration prompt points is controlled to generate a second vibration prompt. The second vibration prompt is used to instruct the user to rotate to adjust the screen acquisition angle of the head-mounted display device to obtain a new user environment object in the adjusted screen acquisition angle. When the target object is included in the new user environment object, the target orientation of the target object is determined. Control the first target vibration prompt point corresponding to the target location among the vibration prompt points to generate a first vibration prompt, wherein the first vibration prompt is used to indicate the target location; Generate a guide route for the current user to move from the current location to the target location; A second target vibration prompt point corresponding to the guidance route is determined from each of the vibration prompt points, and the second target vibration prompt point is controlled to generate a third vibration prompt, wherein the third vibration prompt is used to guide the user's movement route; The method further includes: Determine the current working mode; If the current working mode is passive mode, then the first target object is determined according to the user's voice command, and the first target object is used as the target object.

2. The method as described in claim 1, characterized in that, Following the step of determining the current working mode, the following is also included: If the current working mode is active mode, then objects in the user environment with a recognition distance less than a preset safety threshold are identified as the second target objects. The second target object is used as the target object, and the step of determining the target orientation of the target object from the user environment objects is performed.

3. The method as described in claim 2, characterized in that, Following the step of determining the current working mode, the following is also included: If the current working mode is a mixed mode, then the first target orientation of the first target object and the second target orientation of the second target object are determined from the objects in the user environment. The third target vibration prompt point corresponding to the first target location among the vibration prompt points is controlled to generate a fourth vibration prompt, and the fourth target vibration prompt point corresponding to the second target location among the vibration prompt points is controlled to generate a fifth vibration prompt, wherein the vibration frequencies of the fourth vibration prompt and the fifth vibration prompt are different.

4. A visual assistance system, characterized in that, Applied to a head-mounted display device, the head-mounted display device including vibration alert points, the system includes: The object recognition module is used to perform image recognition and distance recognition of objects in the user environment. If no target object is recognized in the user environment, the rotation vibration prompt point among the vibration prompt points is controlled to generate a second vibration prompt. The second vibration prompt is used to instruct the user to rotate to adjust the screen acquisition angle of the head-mounted display device to obtain new user environment objects in the adjusted screen acquisition angle. When the target object is included in the new user environment object, the target orientation of the target object is determined. A vibration module is used to control a first target vibration prompt point among the vibration prompt points that corresponds to the target location to generate a first vibration prompt, wherein the first vibration prompt is used to indicate the target location; The system further includes a movement guidance module, which is used to: generate a guidance route for the current user to move to the target location; determine a second target vibration prompt point corresponding to the guidance route from each of the vibration prompt points, and control the second target vibration prompt point to generate a third vibration prompt, wherein the third vibration prompt is used to guide the user's movement route; The system further includes a working mode determination module, which is used to: determine the current working mode; if the current working mode is a passive mode, determine a first target object according to the user's voice command, and use the first target object as the target object.

5. A head-mounted display device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the method as claimed in any one of claims 1 to 3.

6. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 3.

7. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the visual assistance method as described in any one of claims 1 to 3.

Images