Control method and device of near-to-eye display equipment, equipment and storage medium

Abstract

The invention relates to the technical field of equipment interaction, and provides a near-eye display equipment control method and device, equipment and a storage medium, and the method comprises the steps: obtaining environment information of an environment where the near-eye display equipment is located, the environment information comprising at least one of an environment image and environment sound; determining task information related to a preset task of the near-to-eye display equipment according to the environment information; and according to the task information, controlling the near-to-eye display equipment to realize a function corresponding to the preset task, so as to improve the processing convenience of the near-to-eye display equipment on the preset task.

Description

Technical Field

[0001] The present application relates to the field of device interaction technology, and in particular to a control method, apparatus, device and storage medium for a near-eye display device. Background Art

[0002] At present, when users deal with more complex work tasks or life tasks in their daily lives and work, they often need to use terminal devices such as mobile phones and cameras to record information, query information, and remind events. However, the use of these terminal devices often requires frequent manual operations, making it difficult for users to observe the surrounding environmental information. This is not only inconvenient, but also poses certain security risks. Summary of the invention

[0003] The main purpose of this application is to provide a control method, device, equipment and storage medium for a near-eye display device, aiming to solve the technical problem that the device operation is not convenient and safe enough when users handle work or life tasks.

[0004] In a first aspect, the present application provides a control method for a near-eye display device, the control method comprising:

[0005] Acquire environmental information of an environment in which the near-eye display device is located, wherein the environmental information includes at least one of an environmental image and an environmental sound;

[0006] Determining task information related to a preset task of the near-eye display device according to the environmental information;

[0007] According to the task information, the near-eye display device is controlled to implement the function corresponding to the preset task.

[0008] In a second aspect, the present application provides a control device for a near-eye display device, the control device comprising:

[0009] An acquisition module, used to acquire environmental information of an environment in which the near-eye display device is located, wherein the environmental information includes at least one of an environmental image and an environmental sound;

[0010] an information determination module, configured to determine task information related to a preset task of the near-eye display device according to the environmental information;

[0011] A function implementation module is used to control the near-eye display device to implement the function corresponding to the preset task according to the task information.

[0012] In a third aspect, the present application provides a near-eye display device, the near-eye display device comprising a memory and a processor;

[0013] The memory is used to store computer programs;

[0014] The processor is used to execute the computer program and implement the steps of the control method of the near-eye display device as described above when executing the computer program.

[0015] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the control method of the near-eye display device as described above are implemented.

[0016] The present application provides a control method, apparatus, device and storage medium for a near-eye display device, the control method comprising: obtaining environmental information of an environment in which the near-eye display device is located, the environmental information comprising at least one of an environmental image and an environmental sound; determining task information related to a preset task of the near-eye display device based on the environmental information; and controlling the near-eye display device to implement a function corresponding to the preset task based on the task information.

[0017] In the case of obtaining environmental information corresponding to the environment described by the near-eye display device, the near-eye display device can use the environmental information as a basis for making processing decisions on preset tasks. For example, the near-eye display device can determine the task information related to the preset task through environmental information, and implement the function corresponding to the preset task based on the task information to implement the processing decision on the preset task. In the process of the near-eye display device determining and implementing the processing decision on the preset task, the user does not need to lower his head to operate the near-eye display device, nor does he need to frequently manually operate the near-eye display device. Based on this, the user can focus on observing factors in the surrounding environment that may affect his or her own safety without being distracted by determining or implementing processing decisions on preset tasks, which is conducive to improving the convenience of the near-eye display device in processing preset tasks, as well as improving the safety of processing preset tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 is a flow chart of a control method of a near-eye display device provided in an embodiment of the present application;

[0020] Figure 2 is a flow chart of a control method of a near-eye display device according to an embodiment of the present application;

[0021] Figure 3 is a schematic block diagram of a control device for a near-eye display device provided in an embodiment of the present application;

[0022] Figure 4 It is a schematic block diagram of a near-eye display device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0024] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0025] The embodiments of the present application provide a control method, device, equipment and storage medium for a near-eye display device. The control method of the near-eye display device can be applied to a near-eye display device. The near-eye display device may include augmented reality (AR) glasses, mixed reality (MR) glasses, AR helmets, MR helmets, and the like, without limitation. The control method of the near-eye display device can also be applied to a server, which can be a separate server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0026] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0027] See also Figure 1 , Figure 1 It is a flow chart of a control method of a near-eye display device provided in an embodiment of the present application. It should be noted that the control method of a near-eye display device provided in an embodiment of the present application can be used in a near-eye display device or in a server, and is not limited here.

[0028] like Figure 1 As shown, the control method of the near-eye display device includes steps S101 to S103.

[0029] S101. Obtain environmental information of an environment in which a near-eye display device is located, where the environmental information includes at least one of an environmental image and an environmental sound.

[0030] For example, a user may wear a near-eye display device on the head to view at least one of the content displayed by the near-eye display device and the environment in which the near-eye display device is located through the near-eye display device. The content displayed by the near-eye display device may include text, images, videos, animations, virtual special effects, etc., which are not limited here.

[0031] In some embodiments, the near-eye display device may have a shooting function. The near-eye display device with a shooting function is equivalent to the user's eyes, and can be used to assist the user in viewing detailed information of the environment in which the near-eye display device is located. For example, the near-eye display device may be provided with a camera, and the near-eye display device may use the camera to shoot the environment in which it is located to obtain an environmental image. The environmental image may store detailed information of the environment in which the near-eye display device is located in the form of an image.

[0032] In some embodiments, the near-eye display device may also have a recording function. The near-eye display device with a recording function is equivalent to the user's ear, which can assist the user to listen to the detailed information of the environment in which the near-eye display device is located. For example, the near-eye display device may be provided with a microphone, and the near-eye display device may collect the sound of the environment in which it is located through the microphone to obtain the environmental sound. The environmental sound may store the detailed information of the environment in which the near-eye display device is located in the form of audio.

[0033] Accordingly, the near-eye display device can determine at least one of the environmental image and the environmental sound as the environmental information of the environment in which the near-eye display device is located. When the user processes a preset task, if the user ignores the corresponding detailed information of the environment and cannot make a processing decision for the preset task, or is prone to making an erroneous processing decision, the near-eye display device can subsequently determine the task information related to the preset task based on the acquired environmental information. The task information, for example, includes detailed information that the user is prone to ignore. The near-eye display device can then assist the user in making a processing decision for the preset task based on the environmental information, so as to subsequently improve the convenience of the near-eye display device in processing the preset task.

[0034] In some embodiments, the near-eye display device is controlled to capture the surrounding environment according to a preset shooting frequency to obtain an environmental image; and the near-eye display device is controlled to collect the sound of the surrounding environment according to a preset sampling rate to obtain an environmental sound.

[0035] For example, in the process of acquiring the environment image, the near-eye display device can capture the environment at a preset shooting frequency to obtain the environment image. The preset shooting frequency can be pre-set or set by the user, which is not limited here.

[0036] In an exemplary embodiment, since the near-eye display device is worn on the user's head, the device posture of the near-eye display device changes accordingly in response to the change of the user's head posture. During the process of the near-eye display device's device posture changing, the near-eye display device can capture the environment at a preset capture frequency to obtain an environment image.

[0037] Take the preset task as an object search task, and the target object corresponding to the object search task as a key. When a user wears a near-eye display device to search for keys indoors, the near-eye display device can capture different details of the room. For example, the near-eye display device can shoot the room at a preset shooting frequency to obtain a corresponding environmental image while the user is patrolling the space in the house or searching for the space where items are placed. The preset shooting frequency includes, for example, 0.5 seconds / time, 1 second / time, 5 seconds / time, 8 seconds / time, etc., which are not limited here.

[0038] Of course, it is not limited to this. When the near-eye display device posture changes, the near-eye display device can, for example, capture the environment and obtain the environment image when the change amount of the device posture corresponding to the near-eye display device exceeds a preset change amount threshold. This is not limited here.

[0039] For example, in the process of acquiring ambient sound, the near-eye display device can collect the sound of the environment according to the preset collection frequency to obtain the ambient sound. The preset collection frequency can be pre-set or set by the user, which is not limited here.

[0040] In this way, the near-eye display device can obtain environmental images according to the preset shooting frequency without manual setting or issuing commands to the device, which is more convenient and quick. Correspondingly, the near-eye display device can obtain environmental sounds according to the preset collection frequency without manual setting or issuing commands to the device, which is more convenient and quick.

[0041] S102: Determine task information related to a preset task of the near-eye display device according to the environmental information.

[0042] Exemplarily, the preset task may include at least one of an object search task and an object recognition task.

[0043] In some embodiments, the object search task is used to instruct the near-eye display device to search for a target object. The target object may include at least one of a person and an object. Of course, the target object is not limited thereto, and the target object may be pre-set or may be set by the user, which is not limited here.

[0044] For example, the application scenarios of the preset tasks may include finding lost relatives or friends, finding lost objects, finding a destination, finding a route, finding a water source, etc., which are not limited here.

[0045] In some embodiments, the object recognition task is used to indicate the recognition of a preset object involved in the environmental information. The preset object may include at least one of a person, an animal, and weather. Of course, the preset object involved in the environmental information is not limited thereto, and the preset object may be pre-set or may be set by the user, which is not limited here.

[0046] For example, application scenarios of object recognition tasks may include wild food recognition, animal recognition, weather recognition, etc., which are not limited here.

[0047] When the environment information is acquired, the near-eye display device can determine task information related to the preset task based on the environment information. The task information is used to indicate information related to the processing decision of the preset task.

[0048] In some embodiments, after acquiring the environmental information, the near-eye display device determines the mission-critical information related to the preset task based on the environmental information.

[0049] For example, mission-critical information may be selected from mission information.

[0050] In an exemplary embodiment, the near-eye display device may obtain the correlation between the task information and the preset task. When the correlation between the task information and the preset task is greater than or equal to a preset correlation threshold, the near-eye display device may determine the task information whose correlation is greater than or equal to the preset correlation threshold as task-critical information.

[0051] Take the case where the object recognition task included in the preset task is animal identification as an example. In the case where the task information determined according to the environmental information includes tree recognition results and animal recognition results, the near-eye display device can determine that the tree recognition result is not mission-critical information based on the correlation between the tree recognition result and the animal identification task being less than a preset correlation threshold. The near-eye display device can determine that the animal recognition result is mission-critical information based on the correlation between the animal recognition result and the animal identification task being greater than or equal to a preset correlation threshold.

[0052] Of course, it is not limited to this. Take the object search task included in the preset task as an example, which is the search for lost objects. In the case where the task information determined according to the environmental information includes the lost objects included in the environment where the near-eye display device is located and the positions of the non-lost objects in the environment where the near-eye display device is located, the near-eye display device can determine that the position of the lost object in the environment where the near-eye display device is located is the key information of the task based on the correlation between the position of the lost object in the environment where the near-eye display device is located and the search for the lost object is greater than or equal to the preset correlation threshold. The near-eye display device can determine that the position of the non-lost object in the environment where the near-eye display device is located is not the key information of the task based on the correlation between the position of the non-lost object in the environment where the near-eye display device is located and the search for the lost object is less than the preset correlation threshold, which is not limited here. Among them, the preset correlation threshold can be predetermined or determined by the user, which is not limited here.

[0053] The mission critical information is information that can be used by the subsequent near-eye display device to determine the processing decision for the preset task. The mission critical information can be displayed partially or completely in the display interface of the near-eye display device in the form of a list, so as to prompt and remind the user of the mission critical information.

[0054] Exemplarily, the preset task includes an object search task.

[0055] In some implementations, a target object corresponding to the object search task is determined.

[0056] For example, the near-eye display device may respond to the detected object determination instruction corresponding to the object search task and determine that the object indicated by the object determination instruction is the target object corresponding to the object search task. The object determination instruction may be determined based on at least one of a first key operation, a first touch operation, a first gesture, a first voice operation, etc. of the user, which is not limited here. The object determination instruction may be pre-set or may be set by the user, which is not limited here.

[0057] Take the object search task of finding lost relatives and friends as an example. Lost relatives and friends can, for example, communicate with the user through chat applications, telephones, and so on. During the voice communication, if an object determination instruction is detected, the near-eye display device can determine the relatives and friends who are communicating with the user in voice as the target object. Of course, it is not limited to this. The user can also select at least one of the photos of relatives and friends and the historical voice information of relatives and friends through a first button operation, a first touch operation, and so on. The near-eye display device determines that the object determination instruction is detected, thereby determining the relatives and friends as the target object. Accordingly, when determining the target object, the near-eye display device can determine at least one of the task information and task key information related to the object search task based on at least one of the photos, voice information, and historical voice information of relatives and friends, without limitation here.

[0058] Take the object search task of finding a lost object as an example. For example, the user can input the name of the lost object into the near-eye display device through a first touch operation, a first voice operation, etc., and the near-eye display device can determine that the object determination instruction is detected, thereby determining the lost object as the target object. Accordingly, the near-eye display device can retrieve photos of the lost object from the network or cloud client based on the name of the lost object, or search for photos of the lost object from a database stored in the near-eye display device or a database of other terminals that are connected to the near-eye display device for communication.

[0059] Based on this, the near-eye display device can determine at least one of the task information and task key information related to the object search task based on the photo of the lost object, which is not limited here. The photos of the lost objects in the database can be pre-set. Take the lost object as a key as an example. The near-eye display device can take photos of the keys in advance and store the photos of the keys in the database. Among them, the near-eye display device can perform feature extraction processing on the photo of the key to extract the outline of the key, other features of the key, and so on. Accordingly, the near-eye display device can also establish an association relationship between the photo of the key and the identification information of the key. The identification information of the key includes, for example, the name of the key, the outline of the key, and so on, which is not limited here.

[0060] Take the object search task as at least one of destination search and path search as an example. The near-eye display device can, for example, preset identification information corresponding to the destination search and the path search. The identification information may include icons, names, pictures, etc., which are not limited here. When the user's first touch operation on the identification information corresponding to the destination search is detected, the near-eye display device can determine that an object determination instruction is detected, and then determine the corresponding destination as the target object. When the user's first touch operation on the identification information corresponding to the path search is detected, the near-eye display device can determine that an object determination instruction is detected, and then determine the corresponding path as the target object.

[0061] Of course, the method for determining the target object corresponding to the object search task is not limited to this. The target object can be preset or set by the user, which is not limited here.

[0062] In this way, when the near-eye display device determines the target object corresponding to the object search task, the target object can be used as a basis for the near-eye display device to search for the object, which is beneficial to improving the accuracy of the near-eye display device in searching for the target object.

[0063] In some embodiments, the position of the target object is detected in the environmental image, and / or the sound features of the target object are detected in the environmental sound and the position of the sound features is obtained; based on the position of the target object in the environmental image, and / or the position of the sound features, the position of the target object in the environment where the near-eye display device is located is determined.

[0064] In the present application, the position of the target object in the environment of the near-eye display device refers to the direction and / or spatial position of the target object relative to the near-eye display device, with the position of the near-eye display device as the starting point and the position of the target object as the end point.

[0065] Since the object search task requires searching for the target object, the near-eye display device can determine the detailed information related to the target object in the environmental information based on the environmental information to determine at least one of the task information and the key information of the task. The detailed information related to the target object can include at least one of whether the target object is involved in the environmental information and the position of the target object indicated by the environmental information.

[0066] The near-eye display device needs to determine whether the environmental information involves a target object, so that if it is determined that the environmental information involves a target object, the position of the target object in the environment where the near-eye display device is located can be determined in combination with the position of the target object indicated by the environmental information.

[0067] For example, when the environmental information includes an environmental image, the near-eye display device can detect the orientation of the target object in the environmental image. For example, the near-eye display device can identify whether the environmental image includes an image area corresponding to the target object. In an exemplary embodiment, when the first similarity between one of the image areas included in the environmental image and the image of the target object is greater than or equal to a first similarity threshold, the near-eye display device can determine that the environmental image includes an image area corresponding to the target object. The first similarity threshold can be pre-set or set by the user, which is not limited here.

[0068] Accordingly, when the environment image includes an image area corresponding to the target object, the near-eye display device can determine the position of the target object in the environment image based on the position of the image area in the environment image. The near-eye display device can determine the position of the target object in the environment where the near-eye display device is located based on the position of the target object in the environment image. For example, the near-eye display device can convert the position of the target object in the environment into the position of the target object in the environment where the near-eye display device is located in combination with the camera parameters of the near-eye display device, which is not limited here.

[0069] Of course, this is not limited to this. In the case where the environment image does not include the image area corresponding to the target object, the near-eye display device can continue to acquire the environment image of the environment where the near-eye display device is located until the acquired environment image includes the image area corresponding to the target object. This is not limited here.

[0070] For example, when the environmental information includes environmental sounds, the near-eye display device can detect whether the environmental sounds include the sound features of the target object. In an exemplary embodiment, the near-eye display device can detect the second similarity between the sound features of the target object and the sound features included in the environmental sounds. For example, the sound features of the target object may include the spectral features corresponding to the sound of the target object. The near-eye display device can obtain the spectral features corresponding to the environmental sounds to determine the sound features of the environmental sounds. Accordingly, the near-eye display device can determine the second similarity between the sound features of the target object and the sound features included in the environmental sounds based on the similarity between the spectral features corresponding to the sound of the target object and the spectral features corresponding to the environmental sounds. When the second similarity between one of the sound features included in the environmental sounds and the sound features of the target object is greater than or equal to the second similarity threshold, the near-eye display device can determine that the environmental sounds include the sound features of the target object. The second similarity threshold can be pre-set or set by the user, and is not limited here.

[0071] When the sound characteristics of the target object are included in the ambient sound, the near-eye display device can determine the position of the sound characteristics of the target object. For example, when the near-eye display device detects the sound characteristics of the target object in the ambient sound, it can locate the sound source characteristics of the target object to obtain the position of the sound characteristics. The near-eye display device can determine the position of the target object in the environment where the near-eye display device is located based on the position of the sound characteristics of the target object.

[0072] Of course, this is not limited to this. If the ambient sound does not include the sound features of the target object, the near-eye display device may continue to acquire the ambient sound of the environment in which the near-eye display device is located until the acquired ambient sound includes the sound features of the target object. This is not limited here.

[0073] Take the target object corresponding to the object search task as relatives and friends as an example. The near-eye display device performs face recognition processing on the environmental image based on the photos of relatives and friends, and determines the position of relatives and friends in the environmental image. Correspondingly, the near-eye display device can perform voiceprint recognition processing on the environmental sound based on the voice characteristics of relatives and friends to determine the position of the voice characteristics of relatives and friends. Accordingly, the near-eye display device can determine the position of relatives and friends in the environment where the near-eye display device is located by combining at least one of the position of relatives and friends in the environmental image and the position of the voice characteristics of relatives and friends. Of course, it is not limited to this and is not limited here.

[0074] Accordingly, when determining the position of the target object in the environment where the near-eye display device is located, the near-eye display device may determine the position of the target object in the environment where the near-eye display device is located as at least one of task information and task key information related to the object search task, so that the near-eye display device can subsequently determine the processing decision for the object search task.

[0075] In this way, the near-eye display device can determine at least one of the position of the target object in the environmental image and the position of the sound feature of the target object based on environmental information, such as at least one of the environmental image and the environmental sound. Accordingly, the near-eye display device can determine the position of the target object in the environment where the near-eye display device is located based on at least one of the position of the target object in the environmental image and the position of the sound feature of the target object, which is conducive to improving the convenience of determining the position of the target object in the environment where the near-eye display device is located. The position of the target object in the environment where the near-eye display device is located can be used by the near-eye display device to subsequently determine the processing decision for the object search task, which is conducive to subsequently improving the convenience of the near-eye display device in processing the object search task.

[0076] Exemplarily, the preset task includes an object recognition task.

[0077] In some implementations, object recognition processing is performed on at least one of the environmental image and the environmental sound to obtain an object recognition result corresponding to the object recognition task.

[0078] Since the object recognition task requires the recognition of the preset object involved in the environmental information, the near-eye display device can determine the task information according to the object recognition result corresponding to the preset object. Accordingly, the near-eye display device can select the key task information from the task information.

[0079] For example, when the environmental information includes an environmental image, the near-eye display device can detect an image area corresponding to a preset object in the environmental image, and identify the preset object based on the image area corresponding to the preset object to obtain an object recognition result corresponding to the preset object.

[0080] Take the object recognition task of food identification as an example. The near-eye display device can identify all foods included in the environmental image and obtain corresponding object recognition results. The object recognition results may include the classification of food, the name of food, etc., which are not limited here. Among them, the classification of food may include fruits, vegetables, meat, etc., which are not limited here. The name of the food may include the specific name of the fruit, the specific name of the vegetable, the specific name of the meat, etc., which are not limited here.

[0081] For example, when the environmental information includes environmental sounds, the near-eye display device can detect the sound characteristics of a preset object in the environmental sounds, and identify the preset object based on the sound characteristics of the preset object to obtain an object recognition result corresponding to the preset object.

[0082] Taking the object recognition task of animal identification as an example, the near-eye display device can detect the sound characteristics of all preset objects in the ambient sound, and determine the object recognition result corresponding to the preset object based on the sound characteristics of the preset object.

[0083] Take the object recognition task of animal identification as an example. The near-eye display device can recognize the sound characteristics of all animals included in the environmental sound and obtain the corresponding object recognition result. The object recognition result may include the classification of animals, the name of animals, etc., which are not limited here. Among them, the classification of animals may include herbivores, carnivores, etc., which are not limited here. The name of the animal may include the specific name of the herbivore, the specific name of the carnivore, etc., which are not limited here.

[0084] Of course, it is not limited to this. When the environmental information includes environmental images and environmental sounds, the near-eye display device can also integrate the environmental images and environmental sounds to identify preset objects in the environment where the near-eye display device is located.

[0085] Take the object recognition task of weather resolution as an example. The near-eye display device can identify the image areas corresponding to all weather included in the environmental image to obtain the corresponding object recognition results. The object recognition results may include weather classification, weather characteristics, etc., which are not limited here. For example, the near-eye display device can determine that the weather classification of the environment in which the near-eye display device is located is sunny weather based on the brightness of the sunlight in the environmental image being greater than or equal to a preset brightness threshold. For another example, the near-eye display device can determine that the weather characteristics of the environment in which the near-eye display device is located are light winds based on the decibel of the wind sound in the ambient sound being less than or equal to a preset decibel threshold. Correspondingly, the near-eye display device can also determine that the weather characteristics of the environment in which the near-eye display device is located are few clouds based on the density of the clouds in the environmental image being less than a preset density threshold. Of course, it is not limited to this and is not limited here.

[0086] When the near-eye display device performs object recognition processing on at least one of the ambient image and the ambient sound to obtain the object recognition results corresponding to each preset object, the near-eye display device can integrate the object recognition results of all the preset objects to determine the object recognition result corresponding to the object recognition task.

[0087] Accordingly, when determining the object recognition result corresponding to the object recognition task, the near-eye display device may determine the object recognition result corresponding to the object recognition task as at least one of task information and task key information related to the object recognition task, so that the near-eye display device can subsequently determine the processing decision for the object recognition task.

[0088] In this way, the near-eye display device can determine the object recognition result corresponding to the object recognition task based on environmental information, such as at least one of the environmental image and the environmental sound, which is conducive to improving the convenience of the object recognition result corresponding to the object recognition task. The object recognition result corresponding to the object recognition task can be used by the near-eye display device to subsequently determine the processing decision for the object recognition task, which is conducive to subsequently improving the convenience of the near-eye display device for the object recognition task.

[0089] In other embodiments of the present application, the near-eye display device can integrate the ambient image and ambient sound to identify and judge the object recognition result corresponding to the object recognition task, which is helpful to improve the accuracy of the object recognition result corresponding to the object recognition task.

[0090] S103. According to the task information, control the near-eye display device to implement a function corresponding to the preset task.

[0091] Exemplarily, the function corresponding to the preset task may include at least one of a display function, a navigation function, and a suggestion function. The functions corresponding to different preset tasks may be different or the same, which is not limited here. Of course, the functions corresponding to the preset tasks are not limited thereto, and the functions corresponding to the preset tasks may be pre-set or set by the user, which is not limited here.

[0092] When at least one of the task information and the task key information associated with the object search task is determined, the near-eye display device can implement the function corresponding to the object search task based on the consideration of assisting the user to find the target object. The function corresponding to the object search task includes, for example, at least one of a display function, a navigation function, and a suggestion function.

[0093] In some embodiments, navigation information of the target object is displayed and / or announced based on the position of the target object in the environment in which the near-eye display device is located.

[0094] For example, when displaying navigation information of a target object, the near-eye display device can add a corresponding virtual special effect to the position of the target object in the environment where the near-eye display device is located by implementing a display function. When the user views the virtual special effect, the user can determine the position of the target object in the environment where the near-eye display device is located.

[0095] For example, when displaying and / or broadcasting the navigation information of the target object, the near-eye display device can realize the navigation function, combined with the position of the target object in the environment where the near-eye display device is located and the current position of the user in the environment where the near-eye display device is located, to navigate how to find the target object. For example, the near-eye display device can display the navigation path corresponding to the target object. For another example, the near-eye display device can broadcast the navigation path corresponding to the target object in real time in combination with the position of the user in the environment where the near-eye display device is located. Accordingly, in the case where there are multiple paths to the position of the target object in the environment where the near-eye display device is located, the near-eye display device can also output the corresponding path selection suggestion for the user's reference by realizing the suggestion function. When outputting the corresponding path selection suggestion, the near-eye display device can be displayed in the form of text, animation, video, etc., or it can be broadcast in the form of voice broadcast, which is not limited here.

[0096] Of course, it is not limited to this. For example, in the process of displaying and / or broadcasting the navigation information of the target object, the near-eye display device may also respond to the change information of at least one of the user's position in the environment where the near-eye display device is located and the target object's position in the near-eye display device. Update the navigation information. For another example, when the user's moving direction is wrong, the near-eye display device can output a direction change prompt to prompt the user to change the moving direction in time. For another example, when the position of the target object in the environment where the near-eye display device is located is not within the shooting range of the near-eye display device, such as when the target object is located behind the near-eye display device, the near-eye display device can output a direction change prompt according to the indication of the position of the sound characteristics of the target object to prompt the user to change the moving direction. For another example, the near-eye display device can obtain the weather forecast and traffic information corresponding to the position of the environment where the near-eye display device is located through the network, so as to update the navigation information in combination with the weather forecast and traffic information. The updated navigation information, direction change prompt, etc. can be displayed in the form of text, animation, video, etc., or can be broadcast in the form of voice broadcast, which is not limited here.

[0097] In this way, the near-eye display device can assist the user in finding the target object when displaying and / or broadcasting the navigation information of the target object, so as to complete the processing decision of the object search task, which is conducive to improving the convenience of the near-eye display device in processing the object search task.

[0098] When at least one of the task information and the task key information associated with the object recognition task is determined, the near-eye display device may implement a function corresponding to the object recognition task based on the consideration of assisting the user in recognizing a preset object involved in the environment in which the near-eye display device is located. The function corresponding to the object recognition task includes, for example, at least one of a display function and a suggestion function.

[0099] In some implementations, based on the object recognition result, risk warning information corresponding to the object recognition result is displayed and / or broadcasted.

[0100] For example, when the object recognition result is determined, the near-eye display device may display the object recognition result. Correspondingly, the near-eye display device may also broadcast the object recognition result.

[0101] For example, when the object recognition result is determined, the near-eye display device can display the risk warning information corresponding to the object recognition result by implementing the suggestion function. Taking the object recognition result including the name of the fruit as an example, the risk warning information output by the near-eye display device can be used to indicate the edible risks corresponding to different fruits. Accordingly, when the user checks the edible risks corresponding to different fruits, he can determine which fruits can be eaten and which fruits cannot be eaten. Taking the object recognition result including the name of the animal as an example, the risk warning information that the near-eye display device can output can be used to indicate the risk factors corresponding to different animals and the risk avoidance suggestions for different animals. Accordingly, when the user checks the risk factors corresponding to different animals and the risk avoidance suggestions for different animals, he can clearly know whether he is in a dangerous situation and promptly refer to the risk avoidance suggestions to avoid animals with higher risk factors. Of course, it is not limited to this. When the object recognition result includes the name of the tree, the near-eye display device can also obtain the current season information, for example. Accordingly, the near-eye display device can combine the seasonal information to determine whether there are fruits on the tree and whether the tree is dangerous, so as to output the corresponding risk warning information. Risk warning information can be displayed in the form of text, animation, video, etc., or it can be broadcast in the form of voice broadcast, without any limitation here.

[0102] In this way, when displaying and / or broadcasting risk warning information corresponding to the object recognition results, the near-eye display device can assist the user in identifying the preset objects involved in the environmental information, and assist the user in escaping the potential dangers in the environment in which the near-eye display device is located, thereby realizing the processing decision of the object recognition task, which is conducive to improving the convenience of the near-eye display device in processing the object recognition task.

[0103] Exemplarily, the number of preset tasks of the near-eye display device may include one or more. In the case where there are multiple preset tasks, based on the user's consideration of the processing requirements of the preset tasks, some preset tasks may be selected from the multiple preset tasks as target tasks, so that the near-eye display device can make corresponding processing decisions on the target tasks.

[0104] In some implementations, in response to a detected task selection operation, a target task corresponding to the task selection operation is determined from preset tasks; and task information related to the target task is determined based on environmental information.

[0105] For example, the task selection operation may be determined based on at least one of a second button operation, a second touch operation, a second gesture, a second voice operation, etc. performed by the user on the near-eye display device, and is not limited here. The task selection operation may be pre-set or may be set by the user, and is not limited here.

[0106] When the near-eye display device detects the task selection operation, it can determine the preset task that the user needs to process from the preset tasks according to the instruction of the task selection operation. Accordingly, the near-eye display device can determine the preset task that the user needs to process as the target task.

[0107] In an exemplary embodiment, the near-eye display device may set different task identifiers corresponding to different preset tasks. The task identifier may include a task icon, a task name, etc., which are not limited here. The near-eye display device may display different task identifiers. Accordingly, when a task selection operation for at least one task identifier is detected, the near-eye display device may determine the preset task corresponding to at least one task identifier selected by the task selection operation as the target task. This is not limited here.

[0108] In another exemplary embodiment, the task selection operation, for example, instructs the near-eye display device to determine the preset tasks under the same theme as the target tasks. The same theme includes, for example, hiking, tourism, wilderness survival, travel navigation, etc., which are not limited here. The target tasks under the same theme may include different preset tasks. For example, for the hiking theme, during the hiking process, the user needs to search for target objects related to hiking and also needs to identify preset objects related to hiking in the environment. The target tasks under the hiking theme may include object search tasks and object recognition tasks. For another example, for the travel navigation theme, during the travel navigation process, the user needs to search for the target objects of the travel navigation, such as the destination, and also needs to identify preset objects related to the travel navigation in the environment, such as road signs, rest stops, gas stations, etc., then the target tasks under the travel navigation theme may include object search tasks and may also include object recognition tasks.

[0109] Accordingly, when determining at least one of the task information and task key information related to the target task, the near-eye display device needs to determine at least one of the task information and task key information related to the target task in combination with the subject of the task selection operation instruction.

[0110] For example, when the target task is an object search task and an object recognition task under the theme of hiking, the near-eye display device can give priority to determining at least one of the task information and key task information related to hiking. For example, for an object search task, the near-eye display device can give priority to determining the positions of hiking companions, water sources, supply points, scenic spots, and other people or objects related to hiking in the environment where the near-eye display device is located. For another example, for an object recognition task, the near-eye display device can give priority to identifying preset objects related to hiking in the environmental information, such as road signs, food, animals, and other preset objects involved in the environment, to obtain corresponding object recognition results. Of course, it is not limited to this and is not limited here.

[0111] When the target task is determined, the near-eye display device can give priority to the target task and suspend processing of other preset tasks except the target task.

[0112] In some embodiments, based on task information related to the target task, the near-eye display device is controlled to implement a function corresponding to the target task.

[0113] Taking the target task under the theme of hiking as an example, the near-eye display device can control the near-eye display device to implement the function corresponding to the object search task based on at least one of the task information related to hiking and the key task information, such as the positions of hiking companions, water sources, supply points, scenic spots, etc., people or objects related to hiking in the environment where the near-eye display device is located. Correspondingly, the near-eye display device can control the near-eye display device to implement the function corresponding to the object recognition task based on the object recognition results of preset objects such as road signs, food, animals, etc. involved in the environment where the near-eye display device is located. Of course, it is not limited to this and is not limited here.

[0114] In this way, the near-eye display device can reduce the number of tasks that need to be processed by the near-eye display device, thereby alleviating the task processing pressure of the near-eye display device and improving the task processing efficiency of the near-eye display device by selecting the target task corresponding to the task selection operation from preset tasks and controlling the near-eye display device to implement the function of the target task according to task information related to the target task.

[0115] In some embodiments, after the near-eye display device is controlled to implement a function corresponding to a preset task according to the task information, at least one of the ambient image and the ambient sound is deleted.

[0116] For example, after the near-eye display device realizes the function corresponding to the preset task, the near-eye display device can determine that the processing decision for the preset task has been completed, and the near-eye display device does not need to continue to use at least one of the originally acquired environmental images and environmental sounds to determine the processing decision for the preset task. Based on this, the near-eye display device can delete at least one of the environmental images and environmental sounds to alleviate the storage pressure of the near-eye display device.

[0117] In this way, the near-eye display device can improve the storage flexibility of at least one of the ambient images and ambient sounds by deleting at least one of the ambient images and ambient sounds, so as to ensure that when the near-eye display device needs to determine the processing decision of other preset tasks in the future, it can have sufficient storage space, and obtain at least one of the ambient images and ambient sounds required for making the processing decision of other preset tasks, which is conducive to improving the convenience of the near-eye display device in processing preset tasks.

[0118] In an exemplary embodiment, Figure 2 As shown, in the process of determining the processing decision of the target task, the near-eye display device can first determine the target task. The near-eye display device can obtain environmental information of the environment in which the near-eye display device is located. For example, the environmental information of the environment in which the near-eye display device is located is obtained in real time through a camera and / or a microphone set on the near-eye display device, and the environmental information includes at least one of an environmental image and an environmental sound. The near-eye display device can monitor and analyze the acquired environmental information in real time to determine at least one of the task information and the key task information related to the target task. The near-eye display device can control the near-eye display device to implement the function corresponding to the target task according to at least one of the task information and the key task information. For example, in the process of implementing the function corresponding to the target task, the near-eye display device can display and / or broadcast at least one of the corresponding display information, navigation information, and suggestion information, and the user can view and / or hear at least one of the display information, navigation information, and suggestion information related to the target task. Correspondingly, the user can determine the processing decision of the target task based on at least one of the display information, navigation information, and suggestion information related to the target task viewed and / or heard, so as to achieve a better decision effect on the target task.

[0119] The control method of the near-eye display device provided in the above embodiment includes: obtaining environmental information of the environment in which the near-eye display device is located, the environmental information including at least one of environmental images and environmental sounds; determining task information related to a preset task of the near-eye display device based on the environmental information; and controlling the near-eye display device to implement the function corresponding to the preset task based on the task information.

[0120] In the case of obtaining environmental information corresponding to the environment described by the near-eye display device, the near-eye display device can use the environmental information as a basis for making processing decisions on preset tasks. For example, the near-eye display device can determine the task information related to the preset task through environmental information, and implement the function corresponding to the preset task based on the task information to implement the processing decision on the preset task. In the process of the near-eye display device determining and implementing the processing decision on the preset task, the user does not need to lower his head to operate the near-eye display device, nor does he need to frequently manually operate the near-eye display device. Based on this, the user can focus on observing factors in the surrounding environment that may affect his or her own safety without being distracted by determining or implementing processing decisions on preset tasks, which is conducive to improving the convenience of the near-eye display device in processing preset tasks, as well as improving the safety of processing preset tasks.

[0121] See also Figure 3 , Figure 3 It is a schematic block diagram of a control device of a near-eye display device provided in an embodiment of the present application. The control device of the near-eye display device can be configured in a near-eye display device or a server to execute the aforementioned control method of the near-eye display device. The near-eye display device may include AR glasses, MR glasses, AR helmets, MR helmets, and the like, which are not limited here. The server may be a separate server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, and big data and artificial intelligence platforms.

[0122] like Figure 3 As shown, the control device of the near-eye display device includes an acquisition module 110 , an information determination module 120 and a function implementation module 130 .

[0123] The acquisition module 110 is used to acquire environmental information of the environment in which the near-eye display device is located, and the environmental information includes at least one of an environmental image and an environmental sound.

[0124] The information determination module 120 is used to determine task information related to a preset task of the near-eye display device according to the environmental information.

[0125] The function implementation module 130 is used to control the near-eye display device to implement the function corresponding to the preset task according to the task information.

[0126] Exemplarily, the preset task includes an object search task.

[0127] The control device further includes an object determination submodule.

[0128] The object determination submodule is used to determine the target object corresponding to the object search task.

[0129] The information determination module 120 includes a first position determination submodule and a second position determination submodule.

[0130] The first position determination submodule is used to detect the position of the target object in the environmental image and / or detect the sound feature of the target object in the environmental sound and obtain the position of the sound feature.

[0131] The second position determination submodule is used to determine the position of the target object in the environment where the near-eye display device is located according to the position of the target object in the environment image and / or the position of the sound feature.

[0132] Exemplarily, the function implementation module 130 includes a first function implementation submodule.

[0133] The first function implementation submodule is used to display and / or broadcast navigation information of the target object according to the position of the target object in the environment where the near-eye display device is located.

[0134] Exemplarily, the preset task includes an object recognition task.

[0135] The information determination module 120 includes an identification submodule.

[0136] The recognition submodule is used to perform object recognition processing on at least one of the environmental image and the environmental sound to obtain an object recognition result corresponding to the object recognition task.

[0137] Exemplarily, the function implementation module 130 includes a second function implementation sub-module.

[0138] The second implementation submodule is used to display and / or broadcast risk warning information corresponding to the object recognition result according to the object recognition result.

[0139] Exemplarily, the information determination module 120 includes a task selection submodule and an information determination submodule.

[0140] The task selection submodule is used to determine a target task corresponding to the task selection operation from the preset tasks in response to the detected task selection operation.

[0141] The information determination submodule is used to determine task information related to the target task according to the environmental information.

[0142] The function implementation module 130 includes a third implementation submodule.

[0143] The third function implementation submodule is used to control the near-eye display device to implement the function corresponding to the target task according to the task information related to the target task.

[0144] Exemplarily, the acquisition module 110 includes an image acquisition submodule and a sound acquisition submodule.

[0145] The image acquisition submodule is used to control the near-eye display device to capture the environment in which it is located according to a preset shooting frequency to obtain the environment image.

[0146] The sound acquisition submodule is used to control the near-eye display device to collect the sound of the environment according to a preset sampling rate to obtain the environmental sound.

[0147] The control device also includes an information deletion submodule.

[0148] The information deletion submodule is used to delete at least one of the environmental image and the environmental sound.

[0149] It should be noted that technicians in the relevant field can clearly understand that for the convenience and simplicity of description, the specific working processes of the above-described devices and modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0150] The method of the present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0151] Exemplarily, the above-mentioned method and apparatus may be implemented in the form of a computer program, which may be run on a near-eye display device or a server to control the near-eye display device. Exemplarily, the near-eye display device may include AR glasses, MR glasses, AR helmets, MR helmets, etc., which are not limited here. The server may be a separate server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, and big data and artificial intelligence platforms.

[0152] See also Figure 4 , Figure 4 It is a schematic block diagram of the structure of a near-eye display device provided in an embodiment of the present application.

[0153] like Figure 4 As shown, the near-eye display device includes a memory and a processor. The memory and the processor may be connected via a system bus, and the memory may include a storage medium and an internal memory.

[0154] The storage medium may store an operating system and a computer program. When the computer program is executed, the processor may execute any method for controlling a near-eye display device.

[0155] The processor is used to provide computing and control capabilities to support the operation of the entire near-eye display device.

[0156] The internal memory provides an environment for the operation of the computer program in the storage medium. When the computer program is executed by the processor, the processor can execute any control method of the near-eye display device.

[0157] Those skilled in the art will understand that Figure 4 The structure shown in the figure is merely a block diagram of a portion of the structure related to the scheme of the present application, and does not constitute a limitation on the near-eye display device to which the scheme of the present application is applied. The specific near-eye display device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0158] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0159] In one embodiment, the processor is used to execute a computer program and implement the following steps when executing the computer program:

[0160] Acquire environmental information of an environment in which the near-eye display device is located, wherein the environmental information includes at least one of an environmental image and an environmental sound;

[0161] Determining task information related to a preset task of the near-eye display device according to the environmental information;

[0162] According to the task information, the near-eye display device is controlled to implement the function corresponding to the preset task.

[0163] It should be noted that technicians in the relevant field can clearly understand that for the convenience and conciseness of description, the specific working process of controlling the near-eye display device described above can refer to the corresponding process in the aforementioned near-eye display device control method embodiment, and will not be repeated here.

[0164] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. The method implemented when the computer program is executed by a processor can refer to the various embodiments of the control method of the near-eye display device of the present application.

[0165] The computer-readable storage medium may be an internal storage unit of the near-eye display device described in the aforementioned embodiment, such as a hard disk or memory of the near-eye display device. The computer-readable storage medium may also be an external storage device of the near-eye display device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the near-eye display device.

[0166] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0167] It should also be understood that the term "and / or" used in the specification of this application and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0168] The serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific implementation mode of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for controlling a near-eye display device, characterized in that: include: Acquire environmental information of an environment in which the near-eye display device is located, wherein the environmental information includes at least one of an environmental image and an environmental sound; Determining task information related to a preset task of the near-eye display device according to the environmental information; According to the task information, the near-eye display device is controlled to implement the function corresponding to the preset task.

2. The control method according to claim 1, characterized in that: The preset tasks include object search tasks; The control method further comprises: Determine a target object corresponding to the object search task; The determining, according to the environmental information, task information related to a preset task of the near-eye display device includes: Detecting the position of the target object in the environmental image, and / or detecting the sound feature of the target object in the environmental sound and obtaining the position of the sound feature; The position of the target object in the environment where the near-eye display device is located is determined according to the position of the target object in the environment image and / or the position of the sound feature.

3. The control method according to claim 2, characterized in that: The controlling the near-eye display device to implement the function corresponding to the preset task according to the task information includes: Navigation information for the target object is displayed and / or broadcast according to the position of the target object in the environment where the near-eye display device is located.

4. The control method according to claim 1, characterized in that: The preset tasks include object recognition tasks; The determining, according to the environmental information, task information related to a preset task of the near-eye display device includes: Perform object recognition processing on at least one of the environmental image and the environmental sound to obtain an object recognition result corresponding to the object recognition task.

5. The control method according to claim 4, characterized in that: The controlling the near-eye display device to implement the function corresponding to the preset task according to the task information includes: Based on the object recognition result, risk warning information corresponding to the object recognition result is displayed and / or broadcast.

6. The control method according to any one of claims 1 to 5, characterized in that: The determining, according to the environmental information, task information related to a preset task of the near-eye display device includes: In response to the detected task selection operation, determining a target task corresponding to the task selection operation from the preset tasks; Determining task information related to the target task according to the environmental information; The controlling the near-eye display device to implement the function corresponding to the preset task according to the task information includes: According to the task information related to the target task, the near-eye display device is controlled to implement the function corresponding to the target task.

7. The control method according to any one of claims 1 to 5, characterized in that: The obtaining of environmental information of the environment in which the near-eye display device is located includes at least one of the following: According to a preset shooting frequency, the near-eye display device is controlled to shoot the environment in which it is located to obtain the environment image; According to a preset sampling rate, the near-eye display device is controlled to collect the sound of the environment to obtain the environmental sound; After controlling the near-eye display device to implement the function corresponding to the preset task according to the task information, the control method further includes: Delete at least one of the environmental image and the environmental sound.

8. A control device for a near-eye display device, characterized in that: The control device comprises: An acquisition module, used to acquire environmental information of an environment in which the near-eye display device is located, wherein the environmental information includes at least one of an environmental image and an environmental sound; an information determination module, configured to determine task information related to a preset task of the near-eye display device according to the environmental information; A function implementation module is used to control the near-eye display device to implement the function corresponding to the preset task according to the task information.

9. A near-eye display device, characterized in that: The near-eye display device includes a memory and a processor; The memory is used to store computer programs; The processor is used to execute the computer program and implement the steps of the control method of the near-eye display device according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for controlling a near-eye display device according to any one of claims 1 to 7 are implemented.

Images