Head-mounted display device-based interest recognition method, head-mounted display device, and medium

Abstract

The application discloses a head-mounted display device-based interest recognition method, a head-mounted display device and a medium, relates to the technical field of virtual display, and discloses a head-mounted display device-based interest recognition method, which comprises the following steps: recognizing the emotional state of a wearer through a detection device of a head-mounted display device; generating an emotional arousal instruction when the emotional state is detected to be a preset excited state; acquiring the visual picture of the wearer when the emotional arousal instruction is detected, and taking the visual picture as sample data; and determining the interest tendency of the wearer according to the sample data. The application acquires the visual picture of a user in an emotional arousal state, so that the head-mounted display device can acquire the picture that causes the emotional fluctuation of the wearer, i.e., the picture that interests the user, thereby enabling the interest of the user to be recognized based on the visual picture, obtaining the interest tendency of the user, and enabling the head-mounted display device to recognize the interest of the user, thereby being more in line with the personalized needs of the user.

Description

Technical Field

[0001] This application relates to the field of virtual display technology, and in particular to an interest recognition method, head-mounted display device, and medium based on a head-mounted display device. Background Technology

[0002] The technologies used in head-mounted display devices mainly include AR (Augmented Reality), VR (Virtual Reality), and MR (Mixed Reality). Based on these technologies, users can experience real or virtual visual scenes through various sensors, cameras, or optical components in the device when wearing a head-mounted display device.

[0003] However, in conventional applications, head-mounted display devices are mainly used to passively present users with real or virtual scenes that they choose to see, and the content is presented based on the user's own operation. They cannot proactively recommend content that users are interested in, nor can they match users with other users with similar interests to meet users' social needs.

[0004] In summary, how to fully utilize head-mounted display devices to intelligently identify and understand users' interests, thereby achieving personalized content recommendations and enhancing the device's customized service efficiency, has become a pressing technical problem that needs to be solved in this field.

[0005] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0006] The main objective of this application is to provide an interest recognition method, head-mounted display device, and medium based on a head-mounted display device, aiming to solve the technical problem of how to identify a user's interests based on a head-mounted display device.

[0007] To achieve the above objectives, this application proposes an interest recognition method based on a head-mounted display device, the method comprising:

[0008] The wearer's emotional state is identified through the detection equipment on the head-mounted display device;

[0009] When the emotional state is detected to be a preset excited state, an emotion activation command is generated;

[0010] When the emotional arousal command is detected, the wearer's visual image is acquired and used as sample data;

[0011] The wearer's interests are determined based on the sample data.

[0012] In one embodiment, the detection device includes: an in-body camera; the step of identifying the wearer's emotional state through the detection device of the head-mounted display includes:

[0013] The wearer's facial expressions are captured by the internal camera;

[0014] The facial expressions are used to obtain facial features, and the wearer's emotional state is determined based on these facial features.

[0015] In one embodiment, the step of determining the wearer's interest preferences based on the sample data further includes:

[0016] When the head-mounted display device is detected to be in a wearing state, environmental images are captured by the external camera of the head-mounted display device at preset time intervals;

[0017] The environmental images were used as sample data.

[0018] In one embodiment, the step of acquiring the wearer's visual image includes:

[0019] Determine whether there is full-screen image content on the display screen of the head-mounted display device;

[0020] If the image content exists, then a screenshot of the image content is taken to obtain the visual image;

[0021] If the image content does not exist, the visual image is acquired based on the external camera of the head-mounted display device.

[0022] In one embodiment, the step of acquiring the visual image based on the external camera of the head-mounted display device includes:

[0023] The position of the wearer's eyes is obtained through the sensors of the head-mounted display device;

[0024] A target coordinate system is established based on the eye position and the camera position of the external camera of the head-mounted display device to determine the relative positional relationship between the eye position and the camera position;

[0025] The wearer's gaze direction and focal length are obtained through the in-body camera of the head-mounted display device;

[0026] Based on the relative positional relationship, the gaze direction is converted into the first image direction of the external camera, and the gaze focal length is converted into the first image focal length of the external camera;

[0027] The wearer's visual image is obtained based on the relative positional relationship, the first image direction, and the first image focal length.

[0028] In one embodiment, the step of acquiring the wearer's visual image based on the relative positional relationship, the first image direction, and the first image focal length includes:

[0029] When there is no display screen on the head-mounted display device, external images from different directions and at different focal lengths are captured by the external camera;

[0030] Determine whether there exists a first external image among the external images that has the same focal length as the first image and the same orientation as the first image;

[0031] If the first external image exists, then the first external image is used as the visual image;

[0032] If the first external image does not exist, then a second external image is determined among the external images. The angle difference between the second image direction of the second external image and the first image direction is less than a preset angle threshold, and the focal length difference between the second image focal length of the second external image and the first image focal length is less than a preset focal length threshold.

[0033] When there are multiple second external images, each second image is cropped and spliced ​​together according to the direction of the first image to obtain the middle image;

[0034] The intermediate image is processed based on the focal length of the first image and the focal lengths of each of the second images to obtain the visual image, so that the focal length of the visual image is the same as the focal length of the first image.

[0035] In one embodiment, the step of acquiring the wearer's visual image based on the relative positional relationship, the first image direction, and the first image focal length includes:

[0036] When a display screen of the head-mounted display device is displaying an image, the shooting direction and shooting focal length of the external camera are adjusted according to the first image direction and the first image focal length.

[0037] The external image is obtained by taking pictures with the external camera according to the shooting direction and the shooting focal length;

[0038] The visual image is obtained by combining the external image with the displayed image based on the relative positional relationship.

[0039] In one embodiment, the step of determining the wearer's interest tendency based on the sample data includes:

[0040] Each sample image in the sample data is preprocessed to obtain intermediate sample images with the same image format;

[0041] Based on a preset image processing model, label prediction is performed on each intermediate sample image to obtain the feature label of each intermediate sample image;

[0042] The frequency of occurrence of each of the aforementioned feature tags is calculated, and the target feature tags with frequencies exceeding a preset frequency are taken as the wearer's interest preferences.

[0043] In addition, to achieve the above objectives, this application also proposes a head-mounted display device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the interest recognition method based on the head-mounted display device as described above.

[0044] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the interest recognition method based on a head-mounted display device as described above.

[0045] This application provides an interest recognition method based on a head-mounted display device. This application detects the wearer's emotional state through the head-mounted display device. When the wearer's emotional state meets the preset excitement state, an emotion arousal command is generated. In response to the emotion arousal command, the head-mounted display device acquires the wearer's visual image at this moment and uses the visual image as sample data to predict the wearer's interest tendency.

[0046] In summary, this application acquires the visual images of a user when they are excited, enabling the head-mounted display device to obtain images that evoke emotional fluctuations in the wearer, i.e., images that interest the user. Based on these visual images, the device can identify the user's interests and determine their interest tendencies, thus allowing the head-mounted display device to recognize the user's interests and better meet the user's personalized needs. Attached Figure Description

[0047] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0048] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1This is a flowchart illustrating an embodiment of the interest recognition method based on a head-mounted display device provided in this application;

[0050] Figure 2 This is a flowchart illustrating an embodiment of the interest recognition method based on a head-mounted display device provided in this application;

[0051] Figure 3 This is a schematic diagram of the overall process of an embodiment of the interest recognition method based on a head-mounted display device provided in this application;

[0052] Figure 4 This is a flowchart illustrating an embodiment of the interest recognition method based on a head-mounted display device provided in this application;

[0053] Figure 5 This is a schematic diagram of the hardware operating environment involved in the interest recognition method based on a head-mounted display device in the embodiments of this application.

[0054] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0055] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0056] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0057] The main solution of this application embodiment is: to identify the wearer's emotional state through the detection device of the head-mounted display device; to generate an emotion stimulation command when the emotional state is detected to be a preset excitement state; to acquire the wearer's visual image when the emotion stimulation command is detected, and to use the visual image as sample data; and to determine the wearer's interest tendency based on the sample data.

[0058] In this embodiment, for ease of description, the following description uses a head-mounted display device as the execution subject.

[0059] The technologies used in head-mounted display devices mainly include AR (Augmented Reality), VR (Virtual Reality), and MR (Mixed Reality). Based on these technologies, users can experience real or virtual visual scenes through various sensors, cameras, or optical components in the device when wearing a head-mounted display device.

[0060] However, in conventional applications, head-mounted display devices are mainly used to passively present users with real or virtual scenes that they choose to see, and the content is presented based on the user's own operation. They cannot proactively recommend content that users are interested in, nor can they match users with other users with similar interests to meet users' social needs.

[0061] In summary, how to fully utilize head-mounted display devices to intelligently identify and understand users' interests, thereby achieving personalized content recommendations and enhancing the device's customized service efficiency, has become a pressing technical problem that needs to be solved in this field.

[0062] To address the aforementioned issues, this application provides an interest recognition method based on a head-mounted display device. This application detects the wearer's emotional state using the head-mounted display device. When the wearer's emotional state meets a preset arousal state, an emotional arousal command is generated. In response to the emotional arousal command, the head-mounted display device acquires the wearer's visual image at that moment and uses the visual image as sample data to predict the wearer's interest tendencies.

[0063] In summary, this application acquires the visual images of a user in an emotionally aroused state, enabling the head-mounted display device to obtain images that evoke emotional fluctuations in the wearer, i.e., images that interest the user. Based on these visual images, the device can identify the user's interests and determine their interest tendencies, thus allowing the head-mounted display device to recognize the user's interests and better meet the user's personalized needs.

[0064] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a head-mounted display device. The following description uses a head-mounted display device as an example to illustrate this embodiment and the subsequent embodiments.

[0065] Based on this, embodiments of this application provide an interest recognition method based on a head-mounted display device, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the interest recognition method based on a head-mounted display device according to this application.

[0066] In this embodiment, the interest recognition method based on a head-mounted display device includes steps S10 to S40:

[0067] Step S10: Identify the wearer's emotional state using the detection device on the head-mounted display device;

[0068] It should be noted that, in this embodiment, the head-mounted display device is a wearable display technology device designed to project visual content directly in front of the user's eyes, creating an immersive visual experience. This type of device is widely used in VR, AR, and MR fields, providing users with unique interaction methods and content display platforms by simulating or enhancing the real world as seen by the user.

[0069] It should be noted that in this embodiment, all privacy data used was obtained with the user's knowledge and authorization.

[0070] In this embodiment, when a user is wearing a head-mounted display device and an authorized device collects the user's interests and preferences, the head-mounted display device monitors the wearer's emotional state in real time through the detection device on the device.

[0071] In one feasible implementation, the detection device may include: an in-body camera; step S10 may include steps S11 to S12:

[0072] Step S11: Acquire the wearer's facial expressions using the internal camera;

[0073] It should be noted that, in this embodiment, the internal camera is a small, high-sensitivity camera installed in the head-mounted display device, commonly referred to as the internal camera, used to capture real-time images of the wearer's face. The internal camera is specially designed to clearly record dynamic changes in key facial areas such as the user's eyes, eyebrows, and mouth under different lighting conditions without interfering with the user's normal activities or line of sight.

[0074] In this embodiment, since human emotions are usually reflected in facial expressions, and facial expressions are the most accurate reflection of emotions, facial expressions are used as the basis for judging emotional state in this embodiment. When the head-mounted display device is correctly worn and activated, the internal camera will capture the wearer's facial expressions in real time, and the wearer's emotional state can be reflected through facial expressions.

[0075] Step S12: Obtain facial expression features from facial expressions and determine the wearer's emotional state based on the facial expression features.

[0076] In this embodiment, after acquiring an image of a facial expression, the wearer's emotional state can be determined by analyzing it using existing image processing algorithms and machine learning models. This process first identifies and extracts key facial feature points, such as the position of the corners of the eyes, the shape of the corners of the mouth, and the angle of the eyebrows. These feature points combine to form specific expression patterns. Subsequently, these patterns are compared with a pre-established expression-emotion correspondence model in a database to determine the wearer's current emotional state, such as happiness, sadness, surprise, anger, fear, or disgust.

[0077] As an example, when a wearer sees an interesting image while wearing the head-mounted display, the image triggers emotional changes in the wearer, which are reflected in their facial expressions. The internal camera can keenly capture this series of subtle dynamics: raised eyebrows, widened eyes, a slight upturn of the corners of the mouth, and tense cheek muscles, presenting a mixed expression of excitement and anticipation. The device's image processing software immediately processes the captured image, using feature point detection technology to identify key facial landmarks, such as the coordinates of the corners of the eyes, the arch of the eyebrows, the sides of the nose, and the corners of the mouth. By analyzing the relative positional changes and muscle tension between these points, the system identifies features such as "raised eyebrows" and "widened eyes," which are typical hallmarks of an excited expression. These features are then input into a pre-trained machine learning model, which has been trained on millions of labeled facial expression images and can accurately identify seven basic emotions (happiness, sadness, anger, surprise, fear, disgust, and neutral) as well as other complex emotions. By matching the currently captured features with excitement expression templates in the model's database, the device determines the user's current emotional state as "excitement."

[0078] In this embodiment, facial expressions can be used to identify the wearer's emotional state, which can accurately distinguish the wearer's specific emotions.

[0079] In another feasible implementation, the wearer's emotional state can be detected by detecting the wearer's physiological signals, such as heart rate and skin conductance.

[0080] In this embodiment, detecting the wearer's emotional state by detecting the wearer's physiological signals can respond quickly and the detection speed is fast.

[0081] It is understandable that the first implementation of step S10 provided above can more accurately distinguish different emotions than the second implementation, thereby improving the detection accuracy and requiring no additional hardware equipment, thus reducing costs. On the other hand, since the second implementation does not require complex feature analysis and other processing compared to the first implementation, it can quickly judge the wearer's emotions, thereby improving the response speed of the judgment.

[0082] The above are only two feasible implementations of step S10 provided in this embodiment. This embodiment does not specifically limit the specific implementation of step S10.

[0083] Step S20: When the detected emotional state is a preset excited state, an emotional arousal instruction is generated;

[0084] It should be noted that, in this embodiment, the preset excitement state is a preset value set by the technician to reflect the user's excitement state. That is, the preset excitement state can indicate that the wearer is in an interested state at this time, that is, the wearer is interested in the picture currently seen, which is beneficial for subsequent interest analysis.

[0085] In this embodiment, once the device detects that the wearer's emotions have reached a preset "excitement" state, it will automatically generate a specific instruction or signal. This instruction can be a simple code message used to trigger a series of subsequent operations.

[0086] Step S30: When an emotional arousal command is detected, the wearer's visual image is acquired and used as sample data;

[0087] It should be noted that, in this embodiment, the visual image refers to the image actually seen by the wearer, which may be a real-world image, a real-world image superimposed with a virtual image, or a virtual reality image.

[0088] In this embodiment, when the emotion arousal command is activated, the camera of the head-mounted display device begins to capture the visual scene in front of the user, and this visual data is used as sample data for subsequent operations.

[0089] Step S40: Determine the wearer's interest preferences based on the sample data.

[0090] It should be noted that in this embodiment, interest can be expressed in many ways, such as tags that the user is interested in, or the proportion of various tags that the user is interested in, etc., and no limitation is made here.

[0091] In this embodiment, there are many ways to determine the wearer's interest tags based on sample data. For example, existing image recognition technology can be used to analyze the collected visual sample data, identify recurring themes, colors, shapes or other visual features, and through pattern recognition and correlation analysis, the system can summarize the visual elements or scene categories that the user is most interested in when in an excited state.

[0092] In another feasible implementation, steps A10 to A20 may be included before step S40:

[0093] Step A10: When it is detected that the head-mounted display device is being worn, the environmental images are captured by the external camera of the head-mounted display device at preset time intervals;

[0094] Step A20: Use the environmental imagery as sample data.

[0095] In this embodiment, with the user's authorization, when the user is wearing the head-mounted display device, the external camera can take pictures at preset time intervals (e.g., every five minutes) to collect images of the user's surrounding environment and use the environmental images as sample data to assist in the analysis of the user's interests.

[0096] Using environmental images to assist in user interest recognition can further segment user characteristics and improve the accuracy of interest recognition.

[0097] In one feasible implementation, step S40 may include steps S41 to S43:

[0098] Step S41: Preprocess each sample image in the sample data to obtain intermediate sample images with the same image format;

[0099] In this embodiment, before performing interest prediction based on sample data, the collected sample images need to undergo a series of standardization processes to ensure that all images have a uniform format and quality in subsequent analysis, facilitating subsequent processing. Preprocessing methods can include many types. For example, images of different resolutions can be uniformly scaled and cropped to a standard size to facilitate algorithm processing. Then, color correction can be performed on the images to reduce the impact of differences in lighting or environment. Next, filtering techniques can be applied to reduce noise in the images and improve clarity. Then, the brightness and contrast of the images can be adjusted to meet the requirements of the model processing. Finally, the image file format can be unified, such as converting to JPEG (Joint Photographic Experts Group, an image file format) or PNG (Portable Network Graphics, an image file format), to facilitate unified processing.

[0100] Step S42: Based on the preset image processing model, perform label prediction on each intermediate sample image to obtain the feature label of each intermediate sample image;

[0101] It should be noted that, in this embodiment, the preset image processing model can be an existing model used to predict features in an image, and is not limited here.

[0102] In this embodiment, a deep learning or machine learning image recognition model, such as a convolutional neural network (CNN), is used to analyze the preprocessed image. The model automatically identifies and assigns corresponding feature labels to the image based on its content. These labels may include the category of objects in the image (such as animals, buildings, people), the scene (such as beach, city, forest), or the style (such as abstract, classical, modern), etc.

[0103] Step S43: Calculate the frequency of occurrence of each feature tag, and take the target feature tags that exceed the preset frequency among the occurrence frequencies as the wearer's interest tendency.

[0104] In this embodiment, the feature labels of all sample images can be statistically analyzed to calculate the number of times each label appears, and a frequency threshold can be set. Only feature labels that appear more frequently than the threshold are considered to be the wearer's significant interest.

[0105] Using the above method for interest prediction can eliminate accidental or unrepresentative labels, ensuring the accuracy and reliability of interest tendencies. Furthermore, with the addition of new samples, interest tendencies can be dynamically adjusted to reflect changes in user interests, enhancing the system's adaptability and user experience.

[0106] Feature clustering can also be used to analyze the images of the sample data, and no further limitations are made here.

[0107] This application acquires the visual images of a user when they are in an emotionally aroused state, enabling the head-mounted display device to obtain images that evoke emotional fluctuations in the wearer, i.e., images that interest the user. Based on these visual images, the device can identify the user's interests and determine their interest tendencies, thus allowing the head-mounted display device to recognize the user's interests and better meet the user's personalized needs.

[0108] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 In step S30, the step of acquiring the wearer's visual image may include steps S31 to S33:

[0109] Step S31: Determine whether there is full-screen image content on the display screen of the head-mounted display device;

[0110] In this embodiment, when acquiring the wearer's visual image, it is necessary to first determine whether the wearer is currently viewing full-screen image content, such as panoramic video or games, in order to determine whether there is a real scene in the wearer's visual image.

[0111] Step S32: If image content exists, take a screenshot of the image content to obtain a visual image;

[0112] In this embodiment, if the user is watching a panoramic video or playing a game, they can directly take a screenshot of the image content played on the head-mounted display device to obtain the screen that interests them.

[0113] Step S33: If there is no image content, then the visual image is acquired based on the external camera of the head-mounted display device.

[0114] In this embodiment, if the user can still see the scene in the real world, the user's visual image may be the display of the environment or a combination of virtual and real images. Therefore, it is necessary to obtain the user's visual image based on the external camera of the head-mounted display device.

[0115] In one feasible implementation, step S33, the step of acquiring visual images based on the external camera of the head-mounted display device, may include steps S331 to S335:

[0116] Step S331: Obtain the position of the wearer's eyeballs using the sensors of the head-mounted display device;

[0117] In this embodiment, once the head-mounted display device is correctly worn and activated, the built-in sensor network (typically including infrared cameras, proximity sensors, gyroscopes, accelerometers, etc.) begins to operate. The sensors can identify the position of the wearer's eyeballs; for example, an infrared camera in conjunction with a specific infrared illumination element. These sensors accurately track the wearer's eye movements, including the position and orientation of the eyeballs, and even subtle changes in the pupil, by emitting invisible infrared light and capturing its reflection.

[0118] Step S332: Establish a target coordinate system based on the eye position and the camera position of the external camera of the head-mounted display device to determine the relative positional relationship between the eye position and the camera position;

[0119] In this embodiment, once the precise position of the eyeball is obtained, the device further integrates this information with the known physical position of the external camera on the head-mounted display device. This involves using the concept of a three-dimensional spatial coordinate system, where the camera position serves as a reference point, and combining it with the eyeball position data to calculate the spatial relationship between the two. During the calculation, it is not limited to using the external camera position as the reference point; as long as the relative positional relationship between the external camera and the eyeball can be obtained, it is acceptable.

[0120] In this embodiment, by determining the relative positional relationship between the eyeball and the external camera, the external camera can perform better in simulating human vision and can also reproduce the image that the human eye wants to record more accurately.

[0121] Step S333: Obtain the wearer's gaze direction and gaze focal length through the in-body camera of the head-mounted display device;

[0122] In this embodiment, to obtain the gaze direction and focal length, it is necessary to first identify the wearer's eye information. An internal camera lens is aimed at the wearer's eyes to record dynamic information such as eye rotation, blinking, and pupil changes. To ensure data accuracy, the internal camera also needs to be equipped with appropriate infrared illumination to clearly capture eye details under various lighting conditions. Compared to static images, video data can better capture the continuity and subtle changes in eye movements, laying the foundation for accurate analysis. Then, using existing image processing and computer vision algorithms, key parameters are automatically extracted from continuous eye videos. Combined with the obtained eye direction and pupil size, eye movement models and geometric optics principles are used to estimate the user's gaze point position in three-dimensional space and its corresponding focal length. The gaze direction is calculated based on the eye rotation angle, while the focal length can be estimated by the change in pupil size, combined with existing correlation curves between eye muscles and eye focal length.

[0123] In this embodiment, analyzing the direction and focal length of the human eye's gaze can reveal the image the wearer wants to capture at the moment the shooting command is given, thereby improving the matching degree between the captured target image and the image desired by the user.

[0124] Step S334: Convert the gaze direction to the first view direction of the external camera according to the relative position relationship, and convert the gaze focal length to the first view focal length of the external camera;

[0125] In this embodiment, a coordinate system transformation is performed using the known relative position of the eyeball and the external camera, as well as the gaze direction and focal length obtained through eye tracking. After the transformation, the direction and focal length of the human eye mapped onto the external camera can be obtained, namely the "first image direction" and the "first image focal length".

[0126] Step S335: Obtain the wearer's visual image based on the relative positional relationship, the direction of the first image, and the focal length of the first image.

[0127] In this embodiment, after calculating the relative positional relationship, the direction of the first image, and the focal length of the first image, the wearer's visual image can be obtained based on the above data.

[0128] Specifically, in one feasible implementation, step S335 may include steps B10 to B60:

[0129] Step B10: When there is no display screen on the head-mounted display device, take pictures of the external scene from different directions and at different focal lengths using the external camera;

[0130] In this embodiment, when there is no displayed image on the head-mounted display device's screen, it means that the wearer is only seeing a real-world scene. At this time, the external camera on the head-mounted display device will capture a series of images of the external environment. These images cover a wide field of view and different focal length settings, ensuring comprehensive coverage from near to far. This acquisition strategy aims to build a rich visual database for subsequent matching and selection processes. The camera may employ rapid continuous shooting or multi-lens simultaneous shooting techniques to efficiently capture environmental information.

[0131] Step B20: Determine whether there is a first external image among the external images that has the same focal length as the first image and the same orientation as the first image.

[0132] In this embodiment, all collected external images are filtered to find images that perfectly match the direction and focal length of the first image.

[0133] Step B30: If a first external image exists, then the first external image is used as the visual image.

[0134] In this embodiment, once an external scene is found that perfectly matches the user's gaze direction and focal length, that scene is designated as the visual scene. This means that the scene best matches the user's current visual focus and can be used as the visual scene seen by the user.

[0135] Step B40: If there is no first external image, then determine the second external image among all external images. The angle difference between the second image direction and the first image direction of the second external image is less than a preset angle threshold, and the focal length difference between the second image focal length and the first image focal length of the second external image is less than a preset focal length threshold.

[0136] It should be noted that, in this embodiment, the angle threshold and focal length threshold can be set according to the actual situation, and are not limited here.

[0137] In this embodiment, if there is no perfectly matching first external image, then an alternative image that is close to the ideal conditions is searched. By comparing the differences in direction and focal length between all acquired external images and the first image, second external images that are not perfectly matching but are still close enough are selected.

[0138] Step B50: When there are multiple second external images, the second images are cropped and spliced ​​according to the direction of the first image to obtain the middle image;

[0139] In this embodiment, when multiple second external images meet the conditions, the system needs to integrate these images to approximate the viewpoint and focal length requirements of the first image as closely as possible. This typically involves cropping each second image according to the orientation of the first image, removing off-center or overlapping portions, and then stitching them together to form a coherent intermediate image. During this process, factors such as edge matching and perspective correction also need to be considered to ensure a natural and smooth stitching.

[0140] Step B60: Process the intermediate image according to the focal length of the first image and the focal lengths of each second image to obtain a visual image, so that the focal length of the visual image is the same as the focal length of the first image.

[0141] In this embodiment, in order to ultimately present the user with an image that matches their viewing focal length, the system needs to perform further focal length adjustment processing on the intermediate image. For example, methods such as digital zoom, resampling, or depth map adjustment can be used to change the depth perception and detail clarity of the image to match the user's expected focal length.

[0142] In this embodiment, the above-described method is used to obtain the visual images that the user is interested in in the real world. This method can accurately obtain the user's visual point, making the image acquisition more accurate and thus improving the accuracy of subsequent interest recognition.

[0143] For example, please refer to Figure 3 After a user puts on the head-mounted display device, the device begins to predict interests. First, it takes photos at fixed intervals, such as one every 5 minutes, to collect data on the wearer's frequently used environment. Simultaneously, a coordinate system is established to determine the positional relationship between the wearer's eyes and the external camera. Then, emotion recognition is used to determine the images or objects the wearer is interested in. Next, the internal camera is activated to acquire eye information. Eye-tracking technology captures this information, recording pupil size and the direction of eye gaze, along with the time. Simultaneously, the external camera takes another photo, and the time is recorded. The photos are then filtered, and the time of the captured eye information is aligned with the coordinate system data at the same moment to determine the direction and distance of the viewer's observation. This data is then converted into the focusing direction and focal length parameters for the photo, allowing the device to extract the objects the wearer is interested in. Once a certain amount of data is collected from the timed photos and the filtered photos, big data analysis can be used to extract the objects the wearer is interested in and the environment they frequently live in. The analysis results are continuously updated as more data is collected. Through big data analysis, we can connect with like-minded individuals to find friends or partners who resonate with us in the vast world.

[0144] In another feasible implementation, please refer to Figure 4 Step S335 may include steps C10 to C30:

[0145] Step C10: When there is a display screen on the head-mounted display device, adjust the shooting direction and shooting focal length of the external camera according to the first image direction and the first image focal length.

[0146] In this embodiment, when the display screen of the head-mounted display device shows a picture, but the picture is not displayed in full screen, it means that the visual picture seen by the user is a combination of the displayed picture and the external picture. Therefore, it is necessary to adjust the shooting direction and shooting focal length of the external camera according to the direction and focal length of the first picture.

[0147] Step C20: Take a picture of the external scene using the external camera according to the shooting direction and focal length;

[0148] In this embodiment, after adjusting the parameters of the external camera, the user can obtain the actual external view by taking a picture with the external camera.

[0149] Step C30: Based on the relative positional relationship, the external image and the displayed image are combined to obtain a visual image.

[0150] In this embodiment, the visual image that the user actually sees can be obtained by synthesizing the external image and the displayed image based on the relative positional relationship between the external camera and the human eye's vision.

[0151] As an example, when a user wears a head-mounted display device and uses AR technology, they see a virtual cartoon character dancing on real steps and find it interesting. After the device detects that the user is interested in this scene, it needs to acquire the real-world scene and the virtual scene displayed on the device at the same time. Then, it combines the real scene and the virtual scene to obtain the visual effect seen by the user.

[0152] In this embodiment, the image seen by the user is synthesized from the image seen by the user and the image seen by the virtual image, which is beneficial for the accurate identification of interests and improves the accuracy of interest identification.

[0153] The above two embodiments are merely two feasible implementations of step S335 provided in this embodiment. This embodiment does not specifically limit the specific implementation of step S335.

[0154] In addition, after obtaining the user's interests, the device can make targeted content recommendations based on the user's interests to improve the user experience, and match users based on their interests when they make friends, thereby increasing the common ground for making friends.

[0155] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the interest recognition method based on head-mounted display devices. Any simple modifications based on this technical concept are within the protection scope of this application.

[0156] This application provides a head-mounted display device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the interest recognition method based on the head-mounted display device in the first embodiment described above.

[0157] The following is for reference. Figure 5 It shows a schematic diagram of a structure suitable for implementing the head-mounted display device of the present application embodiments. Figure 5 The head-mounted display device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of this application.

[0158] like Figure 5 As shown, the head-mounted display device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the head-mounted display device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the head-mounted display device to communicate wirelessly or wiredly with other devices to exchange data. Although head-mounted display devices with various systems are shown in the figures, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0159] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0160] The head-mounted display device provided in this application employs the interest recognition method based on the head-mounted display device in the above embodiments, which can solve the technical problem of how to identify user interests based on a head-mounted display device. Compared with the prior art, the beneficial effects of the head-mounted display device provided in this application are the same as those of the interest recognition method based on the head-mounted display device provided in the above embodiments, and other technical features in this head-mounted display device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0161] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0162] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0163] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the interest recognition method based on a head-mounted display device in the above embodiments.

[0164] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0165] The aforementioned computer-readable storage medium may be included in the head-mounted display device; or it may exist independently and not assembled into the head-mounted display device.

[0166] The aforementioned computer-readable storage medium carries one or more programs that, when executed by the head-mounted display device, cause the head-mounted display device to:

[0167] The wearer's emotional state is identified through the detection equipment on the head-mounted display device;

[0168] When the emotional state is detected to be a preset excited state, an emotion activation command is generated;

[0169] When the emotional arousal command is detected, the wearer's visual image is acquired and used as sample data;

[0170] The wearer's interests are determined based on the sample data.

[0171] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0172] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0173] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0174] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described interest recognition method based on a head-mounted display device, thereby solving the technical problem of how to identify a user's interests based on a head-mounted display device. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the interest recognition method based on a head-mounted display device provided in the above embodiments, and will not be repeated here.

[0175] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the interest recognition method based on a head-mounted display device as described above.

[0176] The computer program product provided in this application can solve the technical problem of how to identify a user's interests based on a head-mounted display device. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the interest identification method based on a head-mounted display device provided in the above embodiments, and will not be repeated here.

[0177] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. An interest recognition method based on a head-mounted display device, characterized in that, The method includes: The wearer's emotional state is identified through the detection equipment on the head-mounted display device; When the emotional state is detected to be a preset excited state, an emotion triggering command is generated; When the emotional arousal command is detected, the wearer's visual image is acquired and used as sample data; The wearer's interests are determined based on the sample data; The step of acquiring the wearer's visual image includes: Determine whether there is full-screen image content on the display screen of the head-mounted display device; If the image content exists, then a screenshot of the image content is taken to obtain the visual image; If the image content does not exist, the visual image is acquired based on the external camera of the head-mounted display device; The step of acquiring the visual image based on the external camera of the head-mounted display device includes: The position of the wearer's eyes is obtained through the sensors of the head-mounted display device; A target coordinate system is established based on the eye position and the camera position of the external camera of the head-mounted display device to determine the relative positional relationship between the eye position and the camera position; The wearer's gaze direction and focal length are obtained through the in-body camera of the head-mounted display device; Based on the relative positional relationship, the gaze direction is converted into the first image direction of the external camera, and the gaze focal length is converted into the first image focal length of the external camera; The wearer's visual image is obtained based on the relative positional relationship, the first image direction, and the first image focal length. The step of obtaining the wearer's visual image based on the relative positional relationship, the first image direction, and the first image focal length includes: When there is no display screen on the head-mounted display device, external images from different directions and at different focal lengths are captured by the external camera; Determine whether there exists a first external image among the external images that has the same focal length as the first image and the same orientation as the first image; If the first external image exists, then the first external image is used as the visual image; If the first external image does not exist, then a second external image is determined among the external images. The angle difference between the second image direction of the second external image and the first image direction is less than a preset angle threshold, and the focal length difference between the second image focal length of the second external image and the first image focal length is less than a preset focal length threshold. When there are multiple second external images, each second image is cropped and spliced ​​together according to the direction of the first image to obtain the middle image; The intermediate image is processed based on the focal length of the first image and the focal lengths of each of the second images to obtain the visual image, so that the focal length of the visual image is the same as the focal length of the first image.

2. The method as described in claim 1, characterized in that, The detection device includes: an internal camera; the step of identifying the wearer's emotional state through the head-mounted display device includes: The wearer's facial expressions are captured by the internal camera; The facial expressions are used to obtain facial features, and the wearer's emotional state is determined based on these facial features.

3. The method as described in claim 1, characterized in that, Prior to the step of determining the wearer's interest based on the sample data, the method further includes: When the head-mounted display device is detected to be in a wearing state, environmental images are captured by the external camera of the head-mounted display device at preset time intervals; The environmental images were used as sample data.

4. The method as described in claim 1, characterized in that, The step of obtaining the wearer's visual image based on the relative positional relationship, the first image direction, and the first image focal length includes: When a display screen of the head-mounted display device is displaying an image, the shooting direction and shooting focal length of the external camera are adjusted according to the first image direction and the first image focal length. The external image is obtained by taking pictures with the external camera according to the shooting direction and the shooting focal length; The visual image is obtained by combining the external image with the displayed image based on the relative positional relationship.

5. The method as described in claim 1, characterized in that, The step of determining the wearer's interest tendency based on the sample data includes: Each sample image in the sample data is preprocessed to obtain intermediate sample images with the same image format; Based on a preset image processing model, label prediction is performed on each intermediate sample image to obtain the feature label of each intermediate sample image; The frequency of occurrence of each of the aforementioned feature tags is calculated, and the target feature tags with frequencies exceeding a preset frequency are taken as the wearer's interest preferences.

6. A head-mounted display device, characterized in that, The head-mounted display device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the interest recognition method based on the head-mounted display device as described in any one of claims 1 to 5.

7. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the interest recognition method based on a head-mounted display device as described in any one of claims 1 to 5.

Images