Intention recognition method, device and equipment

Abstract

The embodiment of the invention discloses an intention recognition method, device and equipment. The method comprises the steps of synchronously obtaining an environment image of an environment where a user is located and hand motion data of the user; the environment image is recognized, and the category of at least one candidate object and bounding box information of the candidate object in the environment image are obtained; according to the hand motion data, determining a pointing area of a gesture of a user in the environment image and a gesture type of the gesture; determining a target object concerned by a user from the candidate objects according to the bounding box information corresponding to the pointing area and the candidate objects; and generating a user intention instruction according to the gesture category and the category of the target object.

Description

Technical Field

[0001] This disclosure relates to the field of intent recognition technology, and more specifically, to an intent recognition method, apparatus, and device. Background Technology

[0002] With the development of wearable devices such as smart glasses, their application in human-computer interaction is becoming increasingly widespread. Accurately understanding the user's interaction intent has become crucial for improving the user experience of smart glasses. Currently, common intent recognition technologies mainly rely on single-modal information processing.

[0003] One type of solution primarily relies on computer vision-based object recognition technology. This approach uses the smart glasses' camera to capture environmental images and leverages deep learning models to identify object categories within the scene. However, this method can only answer "what objects are in the environment," not determine which specific object the user is "focusing on" among multiple similar objects. Another type focuses on gesture recognition. This approach uses sensors to capture the user's hand movements and map them to control commands. However, this pure gesture recognition, detached from specific environmental semantics, typically only executes general, predefined commands and cannot associate the user's actions with specific objects in the environment. For example, when a user makes a "query" gesture, the system cannot automatically determine the object being queried; the user still needs to specify the target through other means (such as voice or gaze), leading to a cumbersome interaction process. Both of these technical solutions suffer from insufficient accuracy in intent recognition and redundant operation steps in existing smart glasses. Summary of the Invention

[0004] The purpose of this disclosure is to provide a new technical solution for intent recognition.

[0005] According to a first aspect of the present disclosure, an intent recognition method is provided, the method comprising: Simultaneously acquire environmental images of the user's surroundings and the user's hand movement data; Identify the environmental image to obtain the category of at least one candidate object and its bounding box information in the environmental image; Based on the hand movement data, determine the pointing area of ​​the user's gesture in the environmental image and the gesture category of the gesture; Based on the bounding box information corresponding to the pointing area and each of the candidate objects, the target object that the user is interested in is determined from each of the candidate objects; Based on the gesture category and the target object category, a user intent command is generated. Optionally, the step of simultaneously acquiring environmental images of the user's surroundings and the user's hand movement data includes: The environmental images are acquired, and a first timestamp is generated for each frame of the environmental image. Collect the hand movement data and generate a second timestamp for the hand movement data; For each frame of the environmental image, hand movement data with the second timestamp located within a set time window centered on the corresponding first timestamp is selected from the hand movement data to synchronize with the environmental image. Optionally, identifying the environmental image and obtaining the category of at least one candidate object and its bounding box information in the environmental image includes: The environmental image is input into a set target detection model to obtain the detection results of one or more objects in the environmental image; wherein, the detection results include the object category, confidence score and bounding box information; By comparing the confidence level with the confidence threshold, at least one candidate object is selected from one or more objects; Obtain the category and bounding box information corresponding to each candidate object.

[0006] Optionally, determining the pointing area of ​​the user's gesture in the environmental image and the gesture category of the gesture based on the hand movement data includes: Based on the hand movement data, determine whether the gesture is a valid pointing gesture; If the gesture is a valid pointing gesture, the gesture category to which the gesture belongs is identified based on the hand movement data; Based on the hand movement data and the user's eye-tracking data, the pointing area of ​​the gesture in the environmental image is determined. Optionally, determining the pointing area of ​​the gesture in the environmental image based on the hand movement data and the user's eye-tracking data includes: Based on the eye-tracking data, determine the first coordinate of the user's gaze point in the image coordinate system of the environmental image; The second coordinate of the user's index fingertip in the image coordinate system is determined based on the hand movement data. The pointing region is determined in the image coordinate system based on the first coordinate and the second coordinate. The line connecting the first coordinate and the second coordinate defines a pointing direction line, which is used to determine the position or direction of the pointing area. Optionally, determining the target object of interest from the candidate objects based on the bounding box information corresponding to the pointing region and each candidate object includes: Calculate the spatial overlap between the pointing region and the bounding box of each candidate object; The target object is determined from the at least one candidate object based on the spatial overlap of each of the aforementioned objects. Optionally, determining the target object from the at least one candidate object based on the spatial overlap includes: By comparing the spatial overlap with the overlap threshold, associated objects of the environment image are selected from the at least one candidate object; The frequency of occurrence of the same object selected as the associated object in multiple consecutive frames of the environmental images is counted. The object that appears most frequently is identified as the target object.

[0007] Optionally, generating the user intent instruction based on the gesture category and the target object category includes: Based on the gesture category and the target object category, query the set intent mapping rule base to obtain the user intent command corresponding to the gesture category and the target object category; The intent mapping rule base stores the mapping relationship between gesture categories, target object categories, and user intent commands.

[0008] According to a second aspect of the present disclosure, an intent recognition device is provided, the device comprising: The acquisition module is used to simultaneously acquire environmental images of the user's surroundings and the user's hand movement data; The recognition module is used to recognize the environmental image and obtain the category of at least one candidate object and its bounding box information in the environmental image. The first determining module is used to determine the pointing area of ​​the user's gesture in the environmental image and the gesture category of the gesture based on the hand movement data; The second determining module is used to determine the target object that the user is interested in from the candidate objects based on the bounding box information corresponding to the pointing area and each candidate object respectively. The generation module is used to generate user intent commands based on the gesture category and the target object category.

[0009] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a memory for storing executable computer instructions; and a processor for executing the method described in accordance with the first aspect above, under the control of the executable computer instructions.

[0010] One beneficial effect of this disclosure is that it can simultaneously analyze environmental images and hand motion data to obtain the category of at least one candidate object and its bounding box information in the environmental image, the pointing area of ​​the gesture in the environmental image and the gesture category, and associate and match the pointing area of ​​the gesture with the bounding box information of the candidate object. This enables the accurate location of the target object that the user is actually interested in from multiple candidate objects, realizing the transformation from "identifying objects" to "identifying objects that the user is interested in", greatly reducing the false judgment rate. Furthermore, it can generate user intent commands based on the gesture category and the category of the target object, so that the gesture commands can be accurately associated with the target object, realizing the intent recognition of "performing a specific operation on the target object", simplifying the interaction process, reducing operation steps, and making the interaction more intuitive and smooth.

[0011] Other features and advantages of this specification will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0012] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of this specification and, together with their description, serve to explain the principles of this specification.

[0013] Figure 1 This is a schematic diagram of the hardware configuration of the electronic device provided in the embodiments of this disclosure; Figure 2 This is a flowchart illustrating the intent recognition method provided in the embodiments of this disclosure; Figure 3 This is a flowchart illustrating an example of an intent recognition method disclosed herein; Figure 4 This is a block diagram of the intent recognition device provided in the embodiments of this disclosure; Figure 5 This is a block diagram of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0014] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the embodiments of the present disclosure.

[0015] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0016] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0017] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0018] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0019] <Hardware Configuration> Figure 1 This is a block diagram of the hardware configuration of an electronic device 1000 according to an embodiment of the present disclosure.

[0020] Electronic device 1000 can be a head-mounted display device, such as smart glasses. Electronic device 1000 can also be a handheld mobile device, such as a smartphone, tablet, or portable inspection device. Electronic device 1000 can also be an in-vehicle smart device or other computing device with image acquisition and motion sensing capabilities.

[0021] In one embodiment, such as Figure 1 As shown, the electronic device 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a speaker 1700, a microphone 1800, etc.

[0022] The processor 1100 may include, but is not limited to, a central processing unit (CPU) or a microprocessor (MCU). The memory 1200 includes, for example, ROM (Read-Only Memory), RAM (Random Access Memory), and non-volatile memory such as a hard disk. The interface device 1300 includes, for example, various bus interfaces, such as serial bus interfaces (including USB interfaces) and parallel bus interfaces. The communication device 1400 is capable of wired or wireless communication. The display device 1500 is, for example, a liquid crystal display (LCD), an LED display, or a touch screen. The input device 1600 includes, for example, a touchscreen, a keyboard, or a gamepad. The electronic device 1000 can output audio information via a speaker 1700 and acquire audio information via a microphone 1800.

[0023] Those skilled in the art should understand that, although in Figure 1 The present specification shows a number of devices of electronic device 1000. However, the electronic device 1000 of the embodiments of this specification may only involve some of the devices, or may also include other devices, which is not limited here.

[0024] In this embodiment, the memory 1200 of the electronic device 1000 is used to store instructions that control the processor 1100 to operate in order to implement or support the implementation of the intent recognition method according to any embodiment. Those skilled in the art can design instructions based on the schemes disclosed in this specification. How the instructions control the processor to operate is well known in the art and will not be described in detail here.

[0025] In the above description, those skilled in the art can design instructions based on the solutions provided in this disclosure. How the instructions control the processor to operate is well known in the art, and therefore will not be described in detail here.

[0026] Figure 1 The electronic devices shown are for illustrative purposes only and are not intended to limit this disclosure, its application, or its use.

[0027] <Method Implementation> Figure 2 An intent recognition method according to an embodiment of this disclosure is shown. This intent recognition method can be performed by an electronic device, which may be... Figure 1 The electronic device shown is 1000. (For example...) Figure 2 As shown, the intent recognition method of this embodiment may include the following steps S2100 to S2500: Step S2100: Simultaneously acquire environmental images of the user's surroundings and the user's hand movement data.

[0028] The environmental image can be an image captured by the camera of an electronic device that reflects the user's real physical environment. This camera can be a red-green-blue (RGB) camera, which can capture images of the user's real-world environment at a set sampling frame rate, such as 30fps.

[0029] Among them, hand motion data can be data reflecting the spatial motion state of the user's hand, which can be collected collaboratively by the inertial measurement unit and gesture sensor of the electronic device at a set sampling frame rate, such as 120fps. Specifically, hand motion data can be data used to characterize the user's hand posture, position, motion trajectory and key point state in three-dimensional space.

[0030] In one example, to address the time synchronization issue between environmental images and hand movement data, a timestamp alignment method can be used to add millisecond-level timestamps to each frame of data. Based on the timestamp of the environmental image, hand movement data within a set time window is selected. Specifically, step S2100, which synchronizes the acquisition of the user's environmental image and hand movement data, can further include: acquiring the environmental image and generating a first timestamp for each frame of the environmental image; acquiring hand movement data and generating a second timestamp for the hand movement data; and for each frame of the environmental image, selecting hand movement data whose second timestamp falls within a set time window centered on the corresponding first timestamp, to synchronize with the environmental image.

[0031] The time window can be set in advance based on actual experience and scenarios; for example, the time window can be set to 50ms.

[0032] The first timestamp mentioned above can be the exposure center time of each frame of the environmental image. The second timestamp mentioned above can be the time when the inertial measurement unit collects inertial data or the time when the gesture sensor collects gesture data.

[0033] For example, the timestamp T of each frame of the environment image can be used. image Based on this, timestamps in [T] were selected from the hand motion data. image -50ms, T image Hand movement data within the +50ms interval are calibrated and aligned to ensure that vision and gesture correspond to the same interaction moment in time.

[0034] Optionally, image preprocessing can be performed on the environmental image, such as distortion correction, white balance adjustment, or illumination normalization, to improve recognition robustness. Kalman filtering can be applied to the hand motion data to filter out high-frequency jitter noise.

[0035] After executing the above step S2100 to synchronously acquire the environmental image of the user's environment and the user's hand movement data, proceed to: Step S2200: Identify the environmental image and obtain the category of at least one candidate object and its bounding box information in the environmental image.

[0036] Candidate objects are those in the environmental image whose confidence level exceeds a confidence threshold. The confidence threshold can be a pre-set value based on practical experience and the scene; for example, it could be 0.7. These candidate objects can be any objects within the current field of view that the user might be interested in, initially identified through visual algorithms.

[0037] In one example, step S2200, which identifies the environmental image and obtains the category of at least one candidate object and its bounding box information in the environmental image, may further include: inputting the environmental image into a set target detection model to obtain the detection results of one or more objects in the environmental image, the detection results of which may include the object's category, confidence level, and bounding box information; comparing the confidence level and a confidence threshold to filter out at least one candidate object from one or more objects; and obtaining the category and bounding box information corresponding to each candidate object.

[0038] The object detection model is used to identify objects in an environmental image and output the detection result of the object. The input of the object detection model is the environmental image, and the output is the detection result of the object in the environmental image. The object detection model can be a YOLOv8-nano model. For example, the object detection model can be a YOLOv8-nano model that has been pre-pruned and quantized to adapt to the embedded computing power of electronic devices such as smart glasses.

[0039] Here, the detection results can include the object's category, confidence score, and bounding box information. The object's category indicates what the object is, such as a book, a cup, or a mobile phone. The confidence score can be a value between 0 and 1, representing how confident the object detection model is in its own judgment. The bounding box information can be the coordinates of a bounding box, such as a rectangle, representing the object's position and extent in the environmental image.

[0040] In this example, a sliding time window buffer pool can be set up, for example, a buffer pool with a duration of 1 second. The detection results of objects identified from the environmental image are stored in the sliding time window buffer pool, while old data older than 1 second is removed, so as to provide the most recent historical object information for subsequent correlation analysis.

[0041] After performing the above step S2200 to identify the environmental image and obtain the category of at least one candidate object and its bounding box information in the environmental image, proceed to: Step S2300: Based on the hand movement data, determine the pointing area of ​​the user's gesture in the environmental image and the gesture category of the gesture.

[0042] In this embodiment, the pointing area of ​​the user's gesture in the above-mentioned environmental image and the gesture category can be determined based on hand movement data.

[0043] In one example, step S2300, which determines the pointing area of ​​the user's gesture in the environmental image and the gesture category based on the hand movement data, may further include the following steps S2310 to S2330: Step S2310: Based on the hand movement data, determine whether the gesture is a valid pointing gesture.

[0044] In step S2310, a valid pointing gesture feature can be pre-set. This hand key point state data and the pointing gesture feature can include an extended index finger and a bent or clenched thumb, middle finger, ring finger, and little finger. In practice, the hand key point state data and the pointing gesture feature in the hand movement data can be compared. If the hand key point state data matches the pointing gesture feature, the gesture can be preliminarily determined to be a candidate valid pointing gesture. Further, based on the motion trajectory data in the hand movement data, such as acceleration or velocity, it can be determined whether the hand is stable. If the hand is stable, the candidate valid pointing gesture can be determined to be a valid pointing gesture.

[0045] Of course, it is also possible to combine hand movement data and eye-tracking data collected by an eye tracker in an electronic device to determine whether the gesture is a valid pointing gesture.

[0046] Step S2320: If the gesture is a valid pointing gesture, identify the gesture category based on the hand movement data.

[0047] In step S2320, if the gesture is a valid pointing gesture, the temporal sequence of hand key points and the temporal sequence of motion trajectory from the hand motion data can be input into a pre-trained gesture classification model. The gesture classification model can then output the gesture category to which the gesture belongs. This gesture recognition model can be a long short-term memory network or a temporal convolutional network.

[0048] Step S2330: Determine the pointing area of ​​the gesture in the environmental image based on hand movement data and user eye tracking data.

[0049] In step S2330, determining the pointing area of ​​the gesture in the environmental image based on hand movement data and eye tracking data can be achieved in the following way: determining the first coordinate of the user's gaze point in the image coordinate system of the environmental image based on eye tracking data; determining the second coordinate of the user's index fingertip in the image coordinate system based on hand movement data; and determining the pointing area in the image coordinate system based on the first and second coordinates.

[0050] The line connecting the first and second coordinates defines the direction line, which can be used to represent the user's intended direction.

[0051] For example, an elliptical region can be generated as the pointing region, centered on the midpoint M between the first coordinate (viewpoint) and the second coordinate (fingertip point). The major axis of the ellipse is parallel to the pointing direction line, and the minor axis is perpendicular to the major axis.

[0052] For example, in the image coordinate system, a circular region can be directly defined as the pointing region by using the second coordinate as the center and the set pixel length as the radius. In this case, the first coordinate (the line-of-sight point) does not directly participate in the calculation of the region's shape and position. That is to say, the pointing region can also be directly determined based on the second coordinate.

[0053] Through the above steps S2310 to S2330, hand movement data and eye tracking data can be fused to determine the pointing area, thereby improving the accuracy of pointing area determination.

[0054] After performing the above step S2300, which determines the pointing area of ​​the user's gesture in the environmental image and the gesture category based on the hand movement data, the process proceeds to: Step S2400: Based on the bounding box information corresponding to the pointing area and each of the candidate objects, determine the target object that the user is interested in from each of the candidate objects.

[0055] In this embodiment, the pointing area can be compared with the bounding box information corresponding to each candidate object to determine the target object that the user is interested in from among the candidate objects.

[0056] In one example, step S2400, which determines the target object of interest from the candidate objects based on the bounding box information corresponding to the pointing region and each candidate object, may further include the following steps S2410 to S2420: Step S2410: Calculate the spatial overlap between the pointed region and the bounding box of each candidate object.

[0057] Among them, spatial overlap can be the intersection over union (IoU). For example, if the pointing region is A and the bounding box of one of the candidate objects is B, the IoU value can be the ratio of the intersection area to the union area of ​​the two, that is, IoU=Area(A∪B)Area(A∩B).

[0058] In step S2410, for each candidate object, the spatial overlap between its bounding box and the pointed region can be calculated. For example, for each candidate object, the IoU value between its bounding box and the pointed region can be calculated.

[0059] Step S2420: Determine the target object from at least one candidate object based on the degree of spatial overlap.

[0060] Specifically, determining the target object from at least one candidate object based on spatial overlap can be achieved as follows: compare spatial overlap with an overlap threshold to filter out associated objects in the environment image from at least one candidate object; count the frequency of occurrence of the same object that has been filtered as associated object in multiple consecutive frames of environment images; and determine the object with the highest frequency as the target object.

[0061] The overlap threshold can be a value preset based on actual experience and scenario; for example, the overlap threshold can be 0.3.

[0062] In step S2420, candidate objects with IoU values ​​greater than or equal to the overlap threshold can be marked as associated objects of the environment image. The frequency of occurrence of the same object marked as associated object in multiple consecutive frames of environment images, such as the most recent 5 consecutive frames of environment images, is counted, and the object with the highest frequency of occurrence is determined as the target object.

[0063] Based on the above steps S2410 to S2420, it can effectively filter noise caused by misidentification of single-frame environmental images or instantaneous interference.

[0064] After performing step S2400 above, and determining the target object of interest from the candidate objects based on the bounding box information corresponding to the pointing area and each candidate object, the process proceeds to: Step S2500: Generate user intent instructions based on the gesture category and the target object category.

[0065] In this embodiment, user intent commands can be generated based on the gesture category and the target object category. This not only realizes the transition from "identifying objects" to "identifying objects that the user is interested in", but also realizes intent recognition of "performing specific operations on the target object".

[0066] In one example, step S2500, which generates a user intent instruction based on the gesture category and the target object category, can be implemented as follows: based on the gesture category and the target object category, query the set intent mapping rule base to obtain the user intent instruction corresponding to the gesture category and the target object category.

[0067] The intent mapping rule base stores the mapping relationship between gesture categories, target object categories, and user intent commands. This intent mapping rule base can be defined by the user to flexibly adapt to different application scenarios.

[0068] Through the embodiments of this disclosure, it is possible to simultaneously analyze environmental images and hand motion data to obtain the category of at least one candidate object and its bounding box information in the environmental image, the pointing area of ​​the gesture in the environmental image and the gesture category, and associate and match the pointing area of ​​the gesture with the bounding box information of the candidate object. This enables the accurate location of the target object that the user is actually interested in from multiple candidate objects, realizing the transformation from "identifying objects" to "identifying objects that the user is interested in", greatly reducing the false judgment rate. Furthermore, it can generate user intent commands based on the gesture category and the category of the target object, so that the gesture commands can be accurately associated with the target object, realizing the intent recognition of "performing a specific operation on the target object", simplifying the interaction process, reducing operation steps, and making the interaction more intuitive and smooth.

[0069] <Example> The following example demonstrates an intent recognition method, in which reference... Figure 3 The electronic device may include a perception data synchronization module, an object recognition and gesture recognition module (specifically including an object recognition module and a gesture recognition module), an intent association analysis module, and an intent output module. In this example, the intent recognition method may include the following steps: In step S310, the sensing data synchronization module of the electronic device acquires environmental images and generates a first timestamp for each frame of environmental images, acquires hand motion data and generates a second timestamp for each set of hand motion data, and selects hand motion data from the hand motion data whose second timestamp is located within a set time window centered on the corresponding first timestamp for each frame of environmental images, so as to synchronize with the environmental images.

[0070] In step S320, the object recognition module of the electronic device inputs the environmental image into a set target detection model to obtain the detection results of one or more objects in the environmental image. The detection results include the object's category, confidence level, and bounding box information. The module also compares the confidence level with a confidence threshold to filter out at least one candidate object from the one or more objects, and obtains the category and bounding box information corresponding to each candidate object.

[0071] In step S330, the gesture recognition module of the electronic device determines whether the gesture is a valid pointing gesture based on the hand movement data. If the gesture is a valid pointing gesture, it identifies the gesture category based on the hand movement data and determines the pointing area of ​​the gesture in the environmental image based on the hand movement data and the user's eye tracking data.

[0072] In step S340, the intent association analysis module of the electronic device calculates the spatial overlap between the pointing region and the bounding box of each candidate object, and filters out the associated objects in the environment image from at least one candidate object based on the spatial overlap.

[0073] In step S350, the intent association analysis module of the electronic device also counts the frequency of occurrence of the same object that has been filtered as an associated object in multiple consecutive frames of environmental images, and identifies the object with the highest frequency of occurrence as the target object.

[0074] In step S360, the intent association analysis module of the electronic device also queries the preset intent mapping rule library according to the gesture category and the target object category to obtain the operation instructions corresponding to the gesture category and the target object category as user intent instructions.

[0075] In step S370, the intent output module of the electronic device sends the user's intent command and records each intent recognition result in the local log.

[0076] This example demonstrates several key aspects. First, the deep fusion of object recognition and gesture pointing effectively overcomes the limitations of single-modal recognition. It accurately locates the specific object a user is interested in within a multi-object scenario, significantly reducing the probability of misjudgment and ensuring the recognition results better match the user's actual needs. Second, it simplifies the user's interaction process. Users no longer need additional input methods like voice or buttons; they can express their intent about a specific object simply by pointing and using a simple gesture. This reduces interaction steps and operation time, resulting in a smoother and more natural human-computer interaction experience. Third, through multimodal data synchronization and alignment, and temporal correlation verification, the solution improves stability in complex scenarios such as changing lighting, slight hand tremors, and partial object occlusion, ensuring reliable intent recognition results.

[0077] <Device Embodiment> Figure 4 This is a schematic diagram illustrating the principle of an intent recognition method according to one embodiment, with reference to... Figure 4 As shown, the intent recognition device 400 may include an acquisition module 410, an recognition module 420, a first determination module 430, a second determination module 440, and a generation module 450.

[0078] The acquisition module 410 is used to simultaneously acquire environmental images of the user's surroundings and the user's hand movement data; The recognition module 420 is used to recognize the environmental image and obtain the category of at least one candidate object and its bounding box information in the environmental image. The first determining module 430 is used to determine the pointing area of ​​the user's gesture in the environmental image and the gesture category of the gesture based on the hand movement data. The second determining module 440 is used to determine the target object that the user is interested in from the candidate objects based on the bounding box information corresponding to the pointing area and each candidate object respectively. The generation module 450 is used to generate user intent commands based on the gesture category and the target object category.

[0079] In one embodiment, the acquisition module 410 is specifically used to acquire the environmental image and generate a first timestamp for each frame of the environmental image; acquire the hand movement data and generate a second timestamp for the hand movement data; for each frame of the environmental image, select hand movement data whose second timestamp is located within a set time window centered on the corresponding first timestamp from the hand movement data, so as to synchronize with the environmental image.

[0080] In one embodiment, the recognition module 420 is specifically used to input the environmental image into a set target detection model to obtain the detection results of one or more objects in the environmental image; wherein, the detection results include the object's category, confidence level, and bounding box information; compare the confidence level and a confidence threshold to filter at least one candidate object from the one or more objects; and obtain the category and bounding box information corresponding to each candidate object.

[0081] In one embodiment, the first determining module 430 is specifically configured to determine whether the gesture is a valid pointing gesture based on the hand movement data; if the gesture is a valid pointing gesture, identify the gesture category to which the gesture belongs based on the hand movement data; and determine the pointing area of ​​the gesture in the environmental image based on the hand movement data and the user's eye-tracking data. In one embodiment, the first determining module 430 is specifically used to determine the first coordinate of the user's gaze point in the image coordinate system of the environmental image based on the eye-tracking data; determine the second coordinate of the user's index fingertip in the image coordinate system based on the hand movement data; and determine the pointing area in the image coordinate system based on the first coordinate and the second coordinate; wherein the line connecting the first coordinate and the second coordinate defines a pointing direction line, and the pointing direction line is used to determine the position or direction of the pointing area.

[0082] In one embodiment, the second determining module 440 is specifically configured to calculate the spatial overlap between the pointing region and the bounding box of each of the candidate objects; and determine the target object from the at least one candidate object based on the spatial overlap.

[0083] In one embodiment, the second determining module 440 is specifically used to compare the spatial overlap and the overlap threshold, filter out related objects in the environmental image from the at least one candidate object; count the occurrence frequency of the same object that is filtered as the related object in multiple consecutive frames of the environmental image; and determine the object with the highest occurrence frequency as the target object.

[0084] In one embodiment, the generation module 450 is specifically configured to query a predefined intent mapping rule library based on the gesture category and the target object category to obtain user intent instructions corresponding to the gesture category and the target object category; wherein, the predefined intent mapping rule library stores the mapping relationship between gesture category, target object category and user intent instructions.

[0085] According to embodiments of this disclosure, it can simultaneously analyze environmental images and hand motion data to obtain the category of at least one candidate object and its bounding box information in the environmental image, the pointing area of ​​the gesture in the environmental image, and the gesture category. It then associates and matches the pointing area of ​​the gesture with the bounding box information of the candidate object, thereby accurately locating the target object that the user is actually interested in from multiple candidate objects. This achieves the transformation from "identifying objects" to "identifying objects that the user is interested in," greatly reducing the false judgment rate. Furthermore, its gesture commands can accurately associate with the target object, realizing the intention recognition of "performing a specific operation on the target object," simplifying the interaction process, reducing operation steps, and making the interaction more intuitive and smooth. <Equipment Example> Figure 5 This is a schematic diagram of the hardware structure of an electronic device according to one embodiment. For example... Figure 5 As shown, the electronic device 1000 includes a processor 1100 and a memory 1200.

[0086] The memory 1200 can be used to store executable computer instructions.

[0087] The processor 1100 can be used to execute an intent recognition method according to embodiments of the present disclosure, under the control of executable computer instructions.

[0088] The electronic device 1000 can be as follows: Figure 1 The electronic device 1000 shown may also be a device with other hardware structures, which are not limited here.

[0089] In another embodiment, the electronic device 1000 may include the above-mentioned intent recognition device 400. In one embodiment, each module of the intent recognition device 400 can be implemented by the processor 1100 running computer instructions stored in the memory 1200.

[0090] Computer-readable storage media This disclosure also provides a computer-readable storage medium storing computer instructions thereon, which, when executed by a processor, perform the intent recognition method provided in this disclosure.

[0091] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0092] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0093] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0094] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0095] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0096] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0097] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0098] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation in a combination of software and hardware are equivalent.

[0099] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of this disclosure is defined by the appended claims.

Claims

1. An intent recognition method, the method comprising: Simultaneously acquire environmental images of the user's surroundings and the user's hand movement data; Identify the environmental image to obtain the category of at least one candidate object and its bounding box information in the environmental image; Based on the hand movement data, determine the pointing area of ​​the user's gesture in the environmental image and the gesture category of the gesture; Based on the bounding box information corresponding to the pointing area and each of the candidate objects, the target object that the user is interested in is determined from each of the candidate objects; Based on the gesture category and the target object category, a user intent command is generated.

2. The method according to claim 1, wherein, The simultaneous acquisition of environmental images of the user's surroundings and the user's hand movement data includes: The environmental images are acquired, and a first timestamp is generated for each frame of the environmental image. Collect the hand movement data and generate a second timestamp for the hand movement data; For each frame of the environmental image, hand movement data with the second timestamp located within a set time window centered on the corresponding first timestamp is selected from the hand movement data to synchronize with the environmental image.

3. The method according to claim 1, wherein, The step of identifying the environmental image and obtaining the category of at least one candidate object and its bounding box information in the environmental image includes: The environmental image is input into a set target detection model to obtain the detection results of one or more objects in the environmental image; wherein, the detection results include the object category, confidence score and bounding box information; By comparing the confidence level with the confidence threshold, at least one candidate object is selected from one or more objects; Obtain the category and bounding box information corresponding to each candidate object.

4. The method according to claim 1, wherein, The step of determining the pointing area of ​​the user's gesture in the environmental image and the gesture category of the gesture based on the hand movement data includes: Based on the hand movement data, determine whether the gesture is a valid pointing gesture; If the gesture is a valid pointing gesture, the gesture category to which the gesture belongs is identified based on the hand movement data; Based on the hand movement data and the user's eye-tracking data, the pointing area of ​​the gesture in the environmental image is determined.

5. The method according to claim 4, wherein, Determining the pointing area of ​​the gesture in the environmental image based on the hand movement data and the user's eye-tracking data includes: Based on the eye-tracking data, determine the first coordinate of the user's gaze point in the image coordinate system of the environmental image; The second coordinate of the user's index fingertip in the image coordinate system is determined based on the hand movement data. The pointing region is determined in the image coordinate system based on the first coordinate and the second coordinate. The line connecting the first coordinate and the second coordinate defines a pointing direction line, which is used to determine the position or direction of the pointing area.

6. The method according to claim 1, wherein, The step of determining the target object of interest from the candidate objects based on the bounding box information corresponding to the pointing region and each candidate object includes: Calculate the spatial overlap between the pointing region and the bounding box of each candidate object; The target object is determined from the at least one candidate object based on the spatial overlap of each of the aforementioned objects.

7. The method according to claim 6, wherein, Determining the target object from the at least one candidate object based on the spatial overlap includes: By comparing the spatial overlap with the overlap threshold, associated objects in the environmental image are selected from the at least one candidate object; The frequency of occurrence of the same object selected as the associated object in multiple consecutive frames of the environmental images is counted. The object that appears most frequently is identified as the target object.

8. The method according to claim 1, wherein, The step of generating user intent commands based on the gesture category and the target object category includes: Based on the gesture category and the target object category, query the set intent mapping rule base to obtain the user intent command corresponding to the gesture category and the target object category; The intent mapping rule base stores the mapping relationship between gesture categories, target object categories, and user intent commands.

9. An intent recognition device, the device comprising: The acquisition module is used to simultaneously acquire environmental images of the user's surroundings and the user's hand movement data; The recognition module is used to recognize the environmental image and obtain the category of at least one candidate object and its bounding box information in the environmental image. The first determining module is used to determine the pointing area of ​​the user's gesture in the environmental image and the gesture category of the gesture based on the hand movement data; The second determining module is used to determine the target object that the user is interested in from the candidate objects based on the bounding box information corresponding to the pointing area and each candidate object respectively. The generation module is used to generate user intent commands based on the gesture category and the target object category.

10. An electronic device, comprising: Memory is used to store executable computer instructions; A processor configured to perform the method according to any one of claims 1-8, under the control of the executable computer instructions.

Images