Object determination method and device, electronic equipment and storage medium

Abstract

The invention relates to an object determination method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring watching angle information of eyes of a user and an image acquired by a camera; mapping the watching angle information to a target area in the image according to a relative position relationship between the camera and the eyes of the user; and determining an object contained in the target area as a target object watched by the user. According to the technical scheme, the real object watched by the user can be accurately determined directly according to the watching angle information of the user, and the target object determination efficiency and accuracy are greatly improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of human-computer interaction, and particularly to an object determination method and apparatus, an electronic device, and a storage medium. Background Art

[0002] Human-computer interaction in the XR field refers to the interaction between a human and real or virtual objects in the environment. The computer needs to know the specific object that the user wants to interact with. The more specific the object is, the higher the interaction efficiency. In the related art, the interaction between a human and a real / virtual object is based on images captured by a camera. The camera transmits the captured images to the computer in the form of a whole (frame) so that the computer can display the captured images in front of the user. The user selects an object from the images displayed in front of him / her for interaction. Since there are usually many objects in the whole image, and the clear pointing angle of human eye vision is 10 degrees (that is, the human eye can clearly see the content within 10 degrees centered on the line of sight focus), it is difficult for the computer to clearly identify the specific object that the user wants to interact with, resulting in low interaction efficiency and reduced user interaction experience. Summary of the Invention

[0003] The present disclosure provides an object determination method and apparatus, an electronic device, and a storage medium to solve the deficiencies in the related art.

[0004] According to a first aspect of an embodiment of the present disclosure, an object determination method is proposed, including:

[0005] Obtaining the gaze angle information of the user's eyes and the images captured by the camera;

[0006] Mapping the gaze angle information to a target area in the image according to the relative position relationship between the camera and the user's eyes;

[0007] Determining the object included in the target area as the target object gazed at by the user.

[0008] Optionally, when the camera follows the movement of the user's head, the relative position relationship is the static relative position relationship between the camera and the user's eyes calibrated in advance; or, when the camera does not follow the movement of the user's head, the relative position relationship is the dynamic relative position relationship between the camera and the user's eyes calibrated temporarily.

[0009] Optionally, the calibration method of the relative position relationship includes at least one of the following: when both the camera and the user's eyes capture a QR code, obtaining first gaze angle information of the user's eyes with respect to the QR code and a calibration image containing the QR code collected by the camera, determining the distance and posture of the camera relative to the QR code according to the position of the QR code in the calibration image, and determining the relative position relationship according to the distance and posture and the first gaze angle information; or, when either the camera or the user's eyes have captured a calibration object, obtaining description information of the calibration object and feeding back the description information to the other party of the camera and the user's eyes, and when the other party captures the calibration object matching the description information, obtaining second gaze angle information of the user's eyes with respect to the calibration object and the distance and posture of the camera relative to the calibration object, so as to determine the relative position relationship according to the distance and posture and the second gaze angle information.

[0010] Optionally, determining the object included in the target area as the target object that the user is gazing at includes: identifying the object included in the target area and determining the target object according to the identification result; or, determining the area to which the target area belongs according to each object and its area in the recognized image, and determining the object included in the area as the target object.

[0011] Optionally, the method further includes: when it is determined that the user has an interaction requirement for the target object, if it is determined that the target object is in the image and not in the target area, obtaining the relative position relationship between the target area and the target object, and generating turning information of the user's head according to the relative position relationship, where the turning information is used to indicate that the user's head turns to the position where the target object is located.

[0012] Optionally, the method further includes: when the number of turns of the user's head reaches a threshold number of times and the target object is still not in the target area, triggering a calibration operation to recalibrate the relative position relationship between the camera and the user's eyes.

[0013] According to a second aspect of the embodiments of the present disclosure, an object determination system is provided, the system includes an eye movement module, a camera, and a processing module, where,

[0014] The eye movement module is configured to capture gaze angle information of the user's eyes and send the captured gaze angle information to the processing module;

[0015] The camera is configured to collect an image and send the collected image to the processing module;

[0016] The processing module is configured to map the gaze angle information to a target area in the image according to the relative position relationship between the camera and the user's eyes, and determine an object included in the target area as the target object gazed at by the user.

[0017] According to a third aspect of the embodiments of the present disclosure, an object determination device is provided, including:

[0018] An acquisition unit, configured to acquire the gaze angle information of the user's eyes and an image collected by the camera;

[0019] A mapping unit, configured to map the gaze angle information to a target area in the image according to the relative position relationship between the camera and the user's eyes;

[0020] A determination unit, configured to determine an object included in the target area as the target object gazed at by the user.

[0021] According to a fourth aspect of the embodiments of the present disclosure, an electronic device is provided, including:

[0022] A processor;

[0023] A memory for storing processor-executable instructions;

[0024] Wherein, the processor is configured to implement the method described in the embodiments of the first aspect above.

[0025] According to a fifth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of the method described in the embodiments of the first aspect above are implemented.

[0026] The technical solutions provided by the embodiments of the present disclosure may include the following beneficial effects:

[0027] As can be seen from the above embodiments, the present disclosure determines the gaze area (i.e., the target area) of the user in the image collected by the camera by acquiring the gaze angle information of the user's eyes and mapping the gaze angle information to the image collected by the camera according to the relative position relationship between the camera and the user's eyes. The object included in this gaze area is the target object gazed at by the user. This method of determining the object does not need to display the image collected by the camera in front of the user, and can directly and accurately determine the real object gazed at by the user according to the gaze angle information of the user, greatly improving the determination efficiency and accuracy of the target object, so that the user can directly perform human-computer interaction with the real object seen by himself / herself subsequently, effectively improving the interaction efficiency and the user's interaction experience.

[0028] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and do not limit the present disclosure. Brief Description of the Drawings

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present disclosure. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0030] Figure 1 is a schematic architecture diagram of an object determination system shown according to an embodiment of the present disclosure.

[0031] Figure 2 is a schematic diagram of a deployment of an eye movement module, a camera, and a processing module in the same electronic device shown according to an embodiment of the present disclosure.

[0032] Figure 3 is a schematic flowchart of an object determination method shown according to an embodiment of the present disclosure.

[0033] Figure 4 is a schematic diagram of mapping gaze angle information to a target area in an image shown according to an embodiment of the present disclosure.

[0034] Figure 5 is a schematic diagram of an identification result shown according to an embodiment of the present disclosure.

[0035] Figure 6 is a schematic block diagram of an object determination device shown according to an embodiment of the present disclosure.

[0036] Figure 7 is a schematic block diagram of an electronic device shown according to an embodiment of the present disclosure. Detailed Embodiments

[0037] The following will clearly and completely describe the technical solutions in the embodiments of the present disclosure in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only some embodiments of the present disclosure, rather than all embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present disclosure.

[0038] The terms used in the embodiments of the present disclosure are for the purpose of describing specific embodiments only and are not intended to limit the embodiments of the present disclosure. The singular forms "a" and "the" used in the embodiments of the present disclosure and the appended claims are also intended to include the plural forms unless the context clearly dictates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0039] It should be understood that although the terms first, second, third, etc. may be used in the embodiments of the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the embodiments of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "when" or "while" or "in response to determining".

[0040] For the purpose of simplicity and easy understanding, when representing the size relationship herein, the terms used are "greater than" or "less than", "higher than" or "lower than". However, for those skilled in the art, it can be understood that the term "greater than" also encompasses the meaning of "greater than or equal to", and "less than" also encompasses the meaning of "less than or equal to"; the term "higher than" encompasses the meaning of "higher than or equal to", and "lower than" also encompasses the meaning of "lower than or equal to".

[0041] Next, one or more embodiments of the present disclosure will be described in detail.

[0042] Figure 1 is a schematic architecture diagram of an object determination system shown according to an embodiment of the present disclosure. As Figure 1 shown, the object determination system may include an eye movement module 11, a camera 12, and a processing module 13. Among them, the eye movement module 11 is used to capture the gaze angle information of the user's eyes and send the gaze angle information to the processing module 13. The camera 12 is used to collect images and send the collected images to the processing module 13. The processing module 13 is used to map the gaze angle information to the target area in the received image according to the relative position relationship between the camera 12 and the user's eyes, and determine the object included in the target area as the target object gazed at by the user. The eye movement module 11 and the processing module 13 may be directly or indirectly connected by means of wired communication or wireless communication; similarly, the camera 12 and the processing module 13 may be directly or indirectly connected by means of wired communication or wireless communication, and the present disclosure does not make special limitations.

[0043] It should be noted that the eye movement module 11, the camera 12, and the processing module 13 are logically independent of each other and can be physically deployed in the same electronic device or separately deployed in multiple electronic devices. The electronic device refers to a device with data processing capabilities, which may include, but is not limited to, devices such as smartphones, desktop computers, tablet computers, laptop computers, e-book readers, smart watches, and smart bracelets. Among them, the electronic device to which the processing module belongs needs to have a certain computing power.

[0044] Figure 2 FIG. 4 is a schematic diagram showing the deployment of an eye movement module, a camera, and a processing module in the same electronic device as shown in an exemplary embodiment. As Figure 2 shown, the electronic device is smart glasses, and the eye movement module, the camera, and the processing module are all deployed in the smart glasses (the processing module is not shown in the figure). During operation, the smart glasses can run an object determination system to achieve the purpose of accurately positioning an object and complete the object determination solution of the present disclosure.

[0045] In addition, the eye movement module can also be deployed on the user's glasses, and the camera and the processing module can be deployed on the user's mobile phone. In this case, the eye movement module, the camera, and the processing module are respectively deployed in different electronic devices, and through the cooperation between the two electronic devices of the glasses and the mobile phone, the object determination solution of the present disclosure can also be completed.

[0046] Figure 3 FIG. 5 is a schematic flow chart of an object determination method shown according to an embodiment of the present disclosure. As Figure 3 shown, the method may include the following steps:

[0047] S301: Obtain the gaze angle information of the user's eyes and the image collected by the camera.

[0048] The gaze angle information of the user's eyes characterizes the gaze direction of the user's eyes, and the gaze angle information of the user's eyes can be monitored by an eye tracking technology. In this embodiment, an eye movement module is used to capture the gaze angle information of the user's eyes. The eye movement module monitors the movement of the eyes by using technologies such as infrared rays, cameras, or electrodes. These technologies can accurately measure the position and movement of the user's eyeballs and convert them into digital signals, so as to track and record the movement of the user's eyes at different gaze points. It should be noted that by using the eye movement module to capture the gaze angle information of the user's eyes, the eye movement module can be deployed in an electronic device carried by the user (such as Figure 2 the smart glasses shown), in which case, it is equivalent to the eye movement module being on the user's body. In addition, the eye movement module can also be deployed in an electronic device outside the user's body. For example, when the user is facing a computer, the eye movement module can be deployed on the computer to capture the gaze direction of the user's eyes in front of the computer.

[0049] S302: Map the gaze angle information to the target area in the image according to the relative position relationship between the camera and the user's eyes.

[0050] The relative position relationship between the camera and the user's eyes refers to the spatial position or orientation relationship between the camera and the user's eyes. This relative position relationship includes a rotational relationship and / or a translational relationship, that is, through certain rotational operations and / or translational operations, the user's eyes can be moved to the position where the camera is located, or the camera can be moved to the position where the user's eyes are located.

[0051] Combined with Figure 2 the smart glasses shown, Figure 4 is a schematic diagram showing how to map the gaze angle information to the target area in the image as shown in an exemplary embodiment. As Figure 4 shown, both the camera and the eye movement module are deployed on the smart glasses. When the user wears the smart glasses, the relative position relationship between the camera and the user's eyes is usually a fixed position relationship. For the sake of easy display, Figure 4 only the camera on the smart glasses is shown in [the figure], and the smart glasses and the eye movement module are not shown. Figure 4 The dotted line ① in [the figure] represents the gaze angle (i.e., the gaze direction) of the user's eyes. According to the relative position relationship between the camera and the user's eyes, this gaze angle is mapped to the image captured by the camera, and a target area (the area within the dotted box) can be obtained. This target area corresponds to the user's gaze area in the real space.

[0052] In one embodiment, the relative position relationship between the camera and the user's eyes can be divided into a static relative position relationship and a dynamic relative position relationship. The static relative position relationship can be understood as that this relative position relationship is fixed and does not change. The dynamic relative position relationship can be understood as that this relative position relationship is in dynamic change.

[0053] When the camera moves with the user's head, the relative position relationship between the camera and the user's eyes is a static relative position relationship. For example, Figure 4 the camera deployed on the smart glasses in [the figure] moves with the user's head, so the relative position relationship between the camera and the user's eyes is fixed. In this case, before implementing the object determination method of the present disclosure, the static relative position relationship between the camera and the user's eyes can be calibrated in advance, so that the calibrated relative position relationship can be directly obtained for mapping when determining the object, which helps to improve the efficiency of object determination.

[0054] In the case where the camera does not follow the movement of the user's head, the relative position relationship between the camera and the user's eyes is a dynamic relative position relationship. For example, when the camera is deployed on the user's arm, the camera will not follow the movement of the user's head, so the relative position relationship between the camera and the user's eyes will change. Another example is when the camera is deployed in the space outside the user's body (such as the ceiling of a meeting room), the camera will also not follow the movement of the user's head in the meeting room. In the above situations, at the current moment when implementing the object determination method of the present disclosure, it is necessary to temporarily calibrate the dynamic relative position relationship between the camera and the user's eyes, so as to ensure the accuracy of the calibrated dynamic relative position relationship, and then perform mapping based on the accurate dynamic relative position relationship.

[0055] In one embodiment, the relative position relationship between the camera and the user's eyes can be calibrated by the camera and the user's eyes capturing the same object. The specific calibration methods can include the following two methods:

[0056] ① The same object is a two-dimensional code of a specific size, and this two-dimensional code can obtain relative distance and attitude information, such as an ARUCO code. A two-dimensional code of a specific size can be displayed on the user's mobile phone. Then, when it is determined that both the camera and the user's eyes have captured the two-dimensional code, the first gaze angle information of the user's eyes with respect to the two-dimensional code and the calibration image containing the two-dimensional code collected by the camera are obtained. The first gaze angle information can be obtained through an eye movement module. Then, the calibration image is parsed, and the distance and attitude of the camera relative to the two-dimensional code are determined according to the position of the two-dimensional code in the calibration image and the size of the two-dimensional code. The relative position relationship between the camera and the user's eyes is calculated based on the first gaze angle information and the distance and attitude of the camera relative to the two-dimensional code. The specific calculation method can refer to the content of determining the relative position relationship in related technologies, which will not be elaborated here.

[0057] ② The same object mentioned above can be any calibration object other than the QR code. When either the camera or the user's eye has captured the calibration object, obtain the description information of the calibration object and send the description information to the other party between the camera and the user's eye, so that the other party can capture the calibration object that matches the description information. For example, for the image captured by the camera, identify the objects included in the image, then select a calibration object from the image, and feedback the description information of the calibration object to the user (voice feedback, screen display feedback or other feedback methods), so that the user can rotate the eyes to capture the calibration object according to the feedback description information. Another example is to obtain the description information of a certain calibration object within the user's field of view and send the description information to the camera, so that the camera can capture the calibration object that matches the description information. When both the camera and the user's eye have captured the calibration object, obtain the second gaze angle information of the user's eye with respect to the calibration object and the distance and attitude of the camera relative to the calibration object, and determine the relative position relationship between the camera and the user's eye based on the distance and attitude and the second gaze angle information.

[0058] S303: Determine the object included in the target area as the target object gazed at by the user.

[0059] In an embodiment, after determining the target area, the objects included in the target area can be identified, and then the target object gazed at by the user can be determined according to the recognition result. Combining Figure 4 , it can be recognized that a TV set is included in the target area, then the TV set is the target object gazed at by the user. This way of determining the target object only needs to identify the objects included in the target area without identifying other objects in the image, which helps to improve the determination efficiency and accuracy of the target object.

[0060] In addition, after determining the target area, the area to which the target area belongs can be determined according to the recognition results of each object in the image and the area where each object is located, and then the objects included in the area can be determined as the target objects. The recognition results of each object in the image and the area where each object is located can be pre-recognized or temporarily recognized when implementing the object determination method of the present disclosure. Combining Figure 4 , first identify each object included in the image collected by the camera and the area where it is located. The division rule of the area where each object is located can be set according to requirements, and the present disclosure does not limit the division rule of the area. Figure 5 It is a schematic diagram of a recognition result shown in an exemplary embodiment. As Figure 5As shown in the figure, the image captured by the camera includes three objects: a mobile phone, a television, and a fan. Among them, the television is within area 1, the fan is within area 2, and the mobile phone is within area 3. Area 4 is a blank area that does not contain any objects. Mapping the user's gaze angle information onto the image results in a target area (such as the dashed box in Figure 5 ). The area to which the target area belongs is area 1. Therefore, the television contained in area 1 can be used as the target object that the user is gazing at. This method does not require performing an identification operation after determining the target area, further improving the efficiency of determining the target object.

[0061] After identifying the target object, the user can interact with the target object through an electronic device including the aforementioned processing module. For example, the user can inquire about relevant information of the target object (including information such as the status, performance, and appearance of the target object), issue control instructions to the target object, etc.

[0062] In the above embodiment, the method of determining the object does not need to display the image captured by the camera in front of the user, and can directly and accurately determine the real object that the user is gazing at based on the user's gaze angle information, greatly improving the efficiency and accuracy of determining the target object. As a result, the user can directly perform human-computer interaction with the real object they see later, effectively improving the interaction efficiency and the user's interaction experience.

[0063] In one embodiment, when it is determined that the user has an interaction requirement for the target object, if the target object is within the image captured by the camera and the target object is not within the target area, a prompt message for turning the head can be fed back to the user, so that after the user turns the head according to the prompt message, they can turn to the position where the target object is located. Combining Figure 5 , assume that the user wants to turn on the fan, that is, the user wants to interact with the fan. According to the relative position relationship between the camera and the user's eyes, the gaze angle information of the user's eyes is mapped to the target area of the image captured by the camera, and it is obtained that the target area is within area 4, that is, the target area does not contain any objects. The fan is within the image and is near the target area. At this time, the relative position relationship between the target area (within area 4) and the target object (within area 2) can be obtained, and then the turning information of the user's head can be generated according to this relative position relationship and fed back to the user, thereby instructing the user to turn the head to the position where the target object is located. This method prompts the user to turn the head according to the relative position relationship between the target object and the target area when the target object that the user wants to interact with cannot be located, thereby helping the user to more accurately focus the line of sight on the target object in order to promote subsequent interaction operations.

[0064] In one embodiment, it is possible that the number of turns of the user's head reaches the number threshold and the target object is still not within the target area. In other words, after the number of turns of the user's head reaches the number threshold, the target object that the user wants to interact with is still not included in the mapped target area. The number threshold can be one or more times. In this case, the calibration operation can be triggered to recalibrate the relative position relationship between the camera and the user's eyes. Then, based on the newly calibrated relative position relationship, the target object is determined to avoid repeated and ineffective object determination operations caused by incorrect relative position relationships.

[0065] Corresponding to the foregoing embodiments of the object determination method, the present disclosure also provides an embodiment of an object determination device.

[0066] Please refer to Figure 6 , Figure 6 which is a schematic block diagram of an object determination device provided by an exemplary embodiment. The device may include: an acquisition unit 602, a mapping unit 604, and a determination unit 606. Among them:

[0067] The acquisition unit 602 is configured to acquire the gaze angle information of the user's eyes and the image collected by the camera.

[0068] The mapping unit 604 is configured to map the gaze angle information to the target area in the image according to the relative position relationship between the camera and the user's eyes.

[0069] The determination unit 606 is configured to determine the object included in the target area as the target object gazed at by the user.

[0070] Optionally, when the camera follows the movement of the user's head, the relative position relationship is the static relative position relationship between the camera and the user's eyes calibrated in advance; or, when the camera does not follow the movement of the user's head, the relative position relationship is the dynamic relative position relationship between the camera and the user's eyes calibrated temporarily.

[0071] Optionally, the calibration method for the relative position relationship includes at least one of the following: when both the camera and the user's eyes capture a two-dimensional code, obtain the first gaze angle information of the user's eyes with respect to the two-dimensional code and the calibration image containing the two-dimensional code collected by the camera, determine the distance and pose of the camera relative to the two-dimensional code according to the position of the two-dimensional code in the calibration image, and determine the relative position relationship according to the distance and pose and the first gaze angle information; or, when either the camera or the user's eyes have captured a calibration object, obtain the description information of the calibration object and feedback the description information to the other party of the camera and the user's eyes, and when the other party captures the calibration object that matches the description information, obtain the second gaze angle information of the user's eyes with respect to the calibration object and the distance and pose of the camera relative to the calibration object, so as to determine the relative position relationship according to the distance and pose and the second gaze angle information.

[0072] Optionally, the determining unit 606 is specifically configured to: identify the objects included in the target area and determine the target object according to the identification result; or, determine the area to which the target area belongs according to the objects in the recognized image and the areas where they are located, and determine the objects included in this area as the target object.

[0073] Optionally, the device further includes:

[0074] A steering information generation unit 608, configured to, when it is determined that the user has an interaction requirement for the target object, if it is determined that the target object is in the image and not in the target area, obtain the relative position relationship between the target area and the target object, and generate steering information of the user's head according to the relative position relationship, where the steering information is used to indicate that the user's head turns to the position where the target object is located.

[0075] Optionally, the device further includes:

[0076] A triggering unit 610, configured to trigger a calibration operation to recalibrate the relative position relationship between the camera and the user's eyes when the number of turns of the user's head reaches a threshold number of times and the target object is still not in the target area.

[0077] Regarding the device in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments of the related method, and will not be elaborated here.

[0078] For the apparatus embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the descriptions of the method embodiments. The apparatus embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed to multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the objectives of the present disclosure. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0079] Figure 7 FIG. is a schematic block diagram of an electronic device according to an embodiment of the present disclosure. For example, the electronic device 700 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0080] Referring to Figure 7 , the electronic device 700 may include one or more of the following components: a processing component 702, a memory 704, a power component 706, a multimedia component 708, an audio component 710, an input / output (I / O) interface 712, a sensor component 714, and a communication component 716.

[0081] The processing component 702 generally controls the overall operation of the electronic device 700, such as operations associated with display, telephone call, data communication, camera operation, and recording operation. The processing component 702 may include one or more processors 720 to execute instructions to complete all or part of the steps of the above method. In addition, the processing component 702 may include one or more modules to facilitate the interaction between the processing component 702 and other components. For example, the processing component 702 may include a multimedia module to facilitate the interaction between the multimedia component 708 and the processing component 702.

[0082] The memory 704 is configured to store various types of data to support the operation of the electronic device 700. Examples of these data include instructions for any application or method operating on the electronic device 700, contact data, phone book data, messages, pictures, videos, etc. The memory 704 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disk.

[0083] The power supply component 706 provides power for various components of the electronic device 700. The power supply component 706 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 700.

[0084] The multimedia component 708 includes a screen that provides an output interface between the electronic device 700 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can not only sense the boundaries of the touch or swipe actions, but also detect the duration and pressure associated with the touch or swipe operation. In some embodiments, the multimedia component 708 includes a front camera and / or a rear camera. When the electronic device 700 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each of the front camera and the rear camera can be a fixed optical lens system or have a focal length and optical zoom capabilities.

[0085] The audio component 710 is configured to output and / or input audio signals. For example, the audio component 710 includes a microphone (MIC) that is configured to receive external audio signals when the electronic device 700 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 704 or transmitted via the communication component 716. In some embodiments, the audio component 710 further includes a speaker for outputting audio signals.

[0086] The I / O interface 712 provides an interface between the processing component 702 and a peripheral interface module, and the peripheral interface module can be a keyboard, a click wheel, buttons, etc. These buttons can include, but are not limited to: a home button, a volume button, a power-on button, and a lock button.

[0087] The sensor assembly 714 includes one or more sensors for providing an assessment of the status of various aspects of the electronic device 700. For example, the sensor assembly 714 can detect the on / off state of the electronic device 700, the relative positioning of components, such as the display and keypad of the electronic device 700. The sensor assembly 714 can also detect a change in the position of the electronic device 700 or a component of the electronic device 700, the presence or absence of user contact with the electronic device 700, the orientation or acceleration / deceleration of the electronic device 700, and a change in the temperature of the electronic device 700. The sensor assembly 714 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 714 can also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 714 can also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0088] The communication component 716 is configured to facilitate communication between the electronic device 700 and other devices in a wired or wireless manner. The electronic device 700 can access a wireless network based on communication standards, such as WiFi, 2G or 3G, 4G LTE, 7G NR, or a combination thereof. In an exemplary embodiment, the communication component 716 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 716 further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0089] In an exemplary embodiment, the electronic device 700 can be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for performing the methods described in any of the above embodiments.

[0090] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as the memory 704 including instructions that can be executed by the processor 720 of the electronic device 700 to complete the above methods. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0091] Other embodiments of the present disclosure will be readily apparent to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include known common general knowledge or conventional technical means in the technical field not disclosed in the present disclosure. The specification and examples are only to be considered as exemplary, and the true scope and spirit of the present disclosure are pointed out by the following claims.

[0092] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present disclosure is only limited by the appended claims.

Claims

1. An object determination method, characterized in that including: obtaining the gaze angle information of the user's eyes and the image captured by the camera; mapping the gaze angle information to a target area in the image according to the relative position relationship between the camera and the user's eyes; determining the object included in the target area as the target object gazed at by the user.

2. The method according to claim 1, wherein when the camera follows the movement of the user's head, the relative position relationship is the static relative position relationship between the pre-calibrated camera and the user's eyes; or when the camera does not follow the movement of the user's head, the relative position relationship is the dynamic relative position relationship between the temporarily calibrated camera and the user's eyes.

3. The method according to claim 1, characterized in that, The calibration method of the relative position relationship includes at least one of the following: when both the camera and the user's eyes capture a two-dimensional code, obtaining the first gaze angle information of the user's eyes for the two-dimensional code and the calibration image containing the two-dimensional code captured by the camera, determining the distance and pose of the camera relative to the two-dimensional code according to the position of the two-dimensional code in the calibration image, and determining the relative position relationship according to the distance and pose and the first gaze angle information; or when either the camera or the user's eyes have captured a calibration object, obtaining the description information of the calibration object and feeding back the description information to the other party of the camera and the user's eyes, and when the other party captures the calibration object matching the description information, obtaining the second gaze angle information of the user's eyes for the calibration object and the distance and pose of the camera relative to the calibration object, so as to determine the relative position relationship according to the distance and pose and the second gaze angle information.

4. The method according to claim 1, wherein The determining the object included in the target area as the target object gazed at by the user includes: identifying the object included in the target area and determining the target object according to the identification result; or determining the area to which the target area belongs according to the objects recognized in the image and the areas where they are located, and determining the object included in the area as the target object.

5. The method according to claim 1, characterized in that, The method further includes: when it is determined that the user has an interaction requirement for the target object, if it is determined that the target object is in the image and not in the target area, obtaining the relative position relationship between the target area and the target object, and generating steering information of the user's head according to the relative position relationship, where the steering information is used to indicate that the user's head turns to the position where the target object is located.

6. The method according to claim 5, wherein The method further includes: when the number of turns of the user's head reaches a threshold number of times and the target object is still not in the target area, triggering a calibration operation to re-calibrate the relative position relationship between the camera and the user's eyes.

7. An object determination system, characterized in that, The system includes an eye movement module, a camera, and a processing module, where The eye movement module is used to capture the gaze angle information of the user's eyes and send the captured gaze angle information to the processing module; The camera is used to collect images and send the collected images to the processing module; The processing module is used to map the gaze angle information to the target area in the image according to the relative position relationship between the camera and the user's eyes, and determine the object included in the target area as the target object gazed by the user.

8. An object determination device, characterized in that, Comprising: An acquisition unit, configured to acquire the gaze angle information of the user's eyes and the images collected by the camera; A mapping unit, configured to map the gaze angle information to the target area in the image according to the relative position relationship between the camera and the user's eyes; A determination unit, configured to determine the object included in the target area as the target object gazed by the user.

9. An electronic device, characterized in that, Comprising: A processor; A memory for storing instructions executable by the processor; Wherein, the processor is configured to implement the method according to any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the method according to any one of claims 1 to 6.

Images