Vehicle-mounted pointing interaction method and vehicle

Abstract

The application discloses a vehicle-mounted pointing interaction method and a vehicle, and relates to the technical field of human-computer interaction. Cabin perception data of a pointing action of an occupant in a vehicle is acquired; a pointing ray of the occupant is generated according to the cabin perception data, and the pointing ray is mapped to a world coordinate system; three-dimensional spatial position information of a candidate target is acquired; a target pointed at by the occupant is determined from the candidate target by matching the pointing ray in the world coordinate system with the three-dimensional spatial position information of the candidate target, and description information of the target is output to the occupant. The application can automatically identify an external target of the vehicle according to a natural pointing action of the occupant and output related information, thereby improving the convenience and efficiency of interaction with the external target of the vehicle.

Description

Technical Field

[0001] This application relates to the technical field of human-computer interaction, and in particular to an in-vehicle pointing interaction method and vehicle. Background Technology

[0002] In the field of in-vehicle human-machine interaction, common interaction methods include voice interaction, touch screen interaction, and gesture interaction. Among them, voice interaction has limited recognition rate in noisy in-vehicle environments, and users need to organize their language to describe their goals, resulting in low interaction efficiency; touch screen interaction requires users to shift their eyes and hands to the central control screen, which may affect driving safety; gesture interaction has been used in some models for function control, such as volume adjustment and answering phone calls.

[0003] In related technologies, the acquisition of external environmental information mainly relies on the active search function of navigation maps. Users need to enter their destination or keywords in the map application to obtain surrounding information, and it is not yet possible to actively recommend relevant content based on the user's natural behavior (such as pointing out the window). Some models are equipped with augmented reality head-up display (AR-HUD) navigation functions, which can overlay navigation information onto the road ahead. However, this function usually only serves the preset navigation route and does not have the ability to identify and recommend temporarily interested targets.

[0004] External visual perception technology has been used in the field of autonomous driving for road object detection and traffic sign recognition, but its application mainly serves to assist driving decisions rather than human-machine interaction within the cabin. Meanwhile, in-cabin gesture recognition technology is also primarily limited to functional control scenarios and has not yet been integrated with external environmental perception and map data.

[0005] In summary, in the relevant technologies, the three perception domains of in-cabin gesture perception, external visual perception, and map point of interest data work independently and lack a fusion mechanism, making it difficult to achieve a natural interactive experience where the system automatically identifies and recommends relevant information when a user points to a target outside the window.

[0006] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is related technology. Summary of the Invention

[0007] This application provides an in-vehicle pointing interaction method and vehicle, which can automatically identify external targets and output descriptive information based on the pointing actions of occupants, thereby solving the technical problem in related technologies that cannot achieve external target interaction based on pointing behavior.

[0008] This application provides an in-vehicle pointing interaction method, which includes: Acquire cabin perception data of occupants' pointing gestures; The occupant's pointing ray is generated based on the cabin perception data, and the pointing ray is mapped to the world coordinate system; Obtain the three-dimensional spatial location information of the candidate target; By matching the pointing ray in the world coordinate system with the three-dimensional spatial position information of the candidate targets, the target pointed to by the occupant is determined from the candidate targets, and the description information of the target is output to the occupant.

[0009] Compared with related technologies, the in-vehicle pointing interaction method provided in this application acquires cabin perception data of the pointing actions of occupants, generates and maps pointing rays to the world coordinate system, and then matches them with the three-dimensional spatial position information of candidate targets to finally determine the target pointed to by the occupant and output descriptive information. Since occupants do not need to organize their speech for voice search or manually input on the touchscreen, the recognition and recommendation of external targets can be triggered simply through natural pointing actions, thereby reducing the operational burden on occupants and improving the convenience and efficiency of external target interaction. Simultaneously, by mapping the pointing ray to the world coordinate system and matching it with the three-dimensional spatial position information of candidate targets, the actual object pointed to by the occupant can be accurately located from multiple possible targets, thereby reducing the deviation in target recognition and improving the accuracy of pointing recognition.

[0010] Optionally, the step of acquiring cabin perception data of occupant pointing actions includes: Images of the occupants' hands are captured by an in-cabin camera, and multiple key points of the hand bones, including at least the shoulder, elbow, wrist, and fingertip, are extracted from them. The step of generating the occupant's pointing ray based on the cabin sensing data includes: Using the line connecting the wrist to the tip of the index finger as the initial pointing direction, and combining the extension direction from the shoulder to the elbow, the initial pointing direction is weighted and corrected to obtain the pointing ray of the occupant in the cockpit coordinate system.

[0011] Because the pointing ray generation process not only extracts the line connecting the wrist to the tip of the index finger as the initial pointing direction, but also further extracts the extension direction from the shoulder to the elbow, and applies a weighted correction to the initial pointing direction, the pointing ray calculation incorporates the spatial relationships of multiple key skeletal points in the upper limb. Compared to schemes that rely solely on hand pointing (such as using only the finger direction), this reduces pointing deviations caused by the user only rotating the wrist without changing the overall arm direction, or errors introduced by inconsistencies between the shoulder / elbow reference direction and the finger direction, thus improving the accuracy and stability of the pointing ray in the cockpit coordinate system. Furthermore, the introduction of shoulder and elbow key points provides a more rigid reference benchmark for the pointing ray, resulting in greater consistency in pointing rays generated by the same user under different pointing postures, which is beneficial for improving the accuracy of subsequent multi-level coordinate mapping.

[0012] Optionally, the step of mapping the pointing ray to the world coordinate system includes: Based on the occupant's seating position and the calibration parameters of the in-cabin camera, the pointing ray in the cockpit coordinate system is converted into a pointing ray in the vehicle coordinate system; Based on the vehicle's current pose, current heading angle, and the installation pose of the external sensors, the pointing rays in the vehicle coordinate system are converted into pointing rays in the world coordinate system.

[0013] Because the pointing ray is first converted from the cockpit coordinate system to the vehicle coordinate system based on the occupant's seat position and the calibration parameters of the in-cabin camera when mapping to the world coordinate system, and then converted from the vehicle coordinate system to the world coordinate system based on the vehicle's current pose, current heading angle, and the installation pose of the external sensors, the occupant's finger direction can be accurately mapped to the world coordinate system through two-level coordinate transformation. This reduces pointing errors caused by parallax between the occupant's eye position and finger position, changes in vehicle motion attitude, and installation deviations of different sensors, improving the accuracy of pointing recognition and providing a more reliable ray input for subsequent target matching.

[0014] Optionally, the step of obtaining the three-dimensional spatial location information of the candidate target includes: Visual perception data is collected by external sensors, and the three-dimensional bounding box of the candidate target is calculated based on the visual perception data. Obtain the 3D bounding boxes of the points of interest corresponding to the candidate targets from the map data; The three-dimensional bounding box and the three-dimensional enclosing box are used as the three-dimensional spatial location information of the candidate target.

[0015] Visual perception data is collected by external sensors, and the 3D bounding boxes of candidate targets are calculated. Simultaneously, the 3D bounding boxes of the points of interest corresponding to the candidate targets are obtained from map data. Both are used together as the 3D spatial location information of the candidate targets. In this way, subsequent ray-matching can utilize both the real-time detection results of visual perception and the prior location information of map data, thereby providing a unified spatial location benchmark for multi-source data fusion. This helps to improve the comprehensiveness and accuracy of candidate target localization and reduce the problems of missed detections or false detections that may be caused by a single information source (visual or map-only).

[0016] Optionally, the step of matching the pointing ray in the world coordinate system with the three-dimensional spatial position information of the candidate target includes: The first intersection determination is performed between the pointing ray in the world coordinate system and the three-dimensional bounding box to obtain a list of visual candidate targets. A second intersection determination is performed between the pointing ray in the world coordinate system and the three-dimensional bounding box to obtain a list of candidate targets on the map. Candidate targets that appear simultaneously in both the visual candidate target list and the map candidate target list will be used as the targets pointed to by the occupants. For candidate targets that only appear in the visual candidate target list or the map candidate target list, the target pointed to by the occupant is obtained by filtering.

[0017] By employing a matching mechanism that combines visual perception and map data verification, a candidate target simultaneously detected by external vision and recorded by a map POI is directly identified as the target pointed to by the occupants. This improves the reliability of target identification and reduces misjudgments caused by false detections from a single data source (such as visual recognition errors or map data bias). Furthermore, for candidate targets in a single list, subsequent filtering steps are introduced instead of direct discarding, allowing for correct target identification even when visual or map data is incomplete. This enhances the solution's fault tolerance and scenario adaptability.

[0018] Optionally, the step of filtering to obtain the target pointed to by the occupant includes: The candidate targets are sorted from closest to furthest from the occupant, and the candidate target closest to the occupant is taken as the target the occupant is pointing to. Alternatively, a salience score can be calculated based on the attribute characteristics of the candidate target, and the salience score can be weighted and fused with the distance ranking weight of the candidate target. The candidate target with the highest weighted fusion score can be used as the target that the occupant is pointing to. The attribute characteristics include the physical size of the candidate target, the popularity score, and the angular deviation between the candidate target and the occupant's line of sight. The popularity score is determined based on the historical access frequency of map points of interest or user evaluation scores. Alternatively, when the difference in significance scores between the top two candidate targets in the weighted fusion score ranking is less than a preset threshold, an interactive confirmation mechanism with the occupant is triggered, and the candidate target selected by the occupant is taken as the target pointed to by the occupant. Alternatively, the occupant's gesture features, vehicle pose, and candidate target set can be input into a deep learning model, and the candidate targets output by the deep learning model can be used as the targets pointed to by the occupant.

[0019] By sorting candidate targets by their distance from occupants from closest to furthest, and using the closest candidate target as the target, the system can quickly provide the most intuitive recommendation results in most common scenarios, reducing the computational overhead required for target confirmation and improving response efficiency.

[0020] By calculating a salience score based on the physical size of the candidate target, its popularity score, and the angular deviation from the occupant's line of sight, and then weighting and fusing this score with the distance ranking weight, the candidate target with the highest weighted fusion score is selected as the target. This approach comprehensively considers the target's spatial distance, visual salience, and user attention tendency, reducing the probability of misjudgment that may result from relying solely on distance ranking and improving the accuracy of target recognition in multi-candidate scenarios.

[0021] When the difference in significance scores between the top two candidate targets in the weighted fusion score ranking is less than a preset threshold, an interactive confirmation mechanism with the occupants is triggered, and the candidate target selected by the occupants is used as the target. This allows for the introduction of a user confirmation step when the confidence of the automatic decision is insufficient, reducing interaction failures caused by automatic recommendation errors and improving users' trust and satisfaction with the recommendation results.

[0022] By inputting occupant gesture features, vehicle pose, and candidate target set into a deep learning model, and having the model directly output pointing to the target, it is possible to learn complex pointing intention patterns from a large amount of data without the need for manual design of explicit sorting or weighting rules, thereby improving adaptability and recognition accuracy in complex scenarios (such as dense target groups and dynamic environments).

[0023] Optionally, the method further includes: The system guides the occupants to point in multiple known calibration directions. Based on the deviation between the occupants' actual pointing direction and the calibration direction, the system calculates the occupants' pointing offset vector. The pointing offset vector is used to correct the occupants' pointing ray and is adaptively updated based on the occupants' selection of candidate targets.

[0024] By guiding occupants to point in multiple known calibration directions and calculating their pointing offset vector based on the deviation between the actual pointing direction and the calibration direction, the system can quantify and compensate for systematic pointing errors caused by factors such as body structure and pointing habits of different occupants. This reduces pointing ray deviation caused by individual differences and improves the initial accuracy of pointing recognition. Furthermore, since the pointing offset vector is used to correct the occupant's pointing ray, and the offset vector is adaptively updated based on the occupant's selection of candidate targets in subsequent use, the system can continuously learn the occupant's pointing behavior characteristics during use. Personalized correction parameters are gradually optimized with increasing interaction frequency, thereby reducing residual bias after calibration and improving adaptability and stability for long-term use by the same occupant.

[0025] Optionally, the method further includes: Based on the vehicle's motion status, predict potential targets that will soon enter the vehicle's forward field of view, and preload the map data corresponding to the potential targets into the local cache; wherein, the pre-loading range is dynamically adjusted according to the vehicle speed. When the occupant performs a pointing action, the target corresponding to the pointing ray is matched first in the local cache.

[0026] By predicting potential targets about to enter the vehicle's forward field of view based on the vehicle's motion and pre-loading the corresponding map data into the local cache, data preparation can be completed before the occupant performs a pointing action. This reduces the waiting time for retrieving data from a remote source after pointing occurs, thereby reducing interaction latency. Because the pre-loading range is dynamically adjusted according to vehicle speed, excessive redundant data is avoided at low speeds, reducing local storage usage. At high speeds, the pre-loading range is expanded to cover rapidly approaching targets, achieving a better balance between cache hit rate and resource consumption, thus improving the adaptive capability of in-vehicle pointing interaction.

[0027] Optionally, the method further includes: When the external sensors cannot identify the category of the candidate target, the three-dimensional spatial position information of the candidate target is determined based on the outline position of the candidate target; When the external sensors fail, a fan-shaped search area is determined based on the direction of the pointing ray, and the three-dimensional spatial location information of the candidate target is determined from the map data within the fan-shaped search area. When both external sensors and map data are unavailable, directional summary information is output based on the directional ray.

[0028] By determining the three-dimensional spatial location information of candidate targets based on their outline positions, even when the visual perception module can detect the existence of a target but cannot output a specific category label, it can still use the geometric position of the target to match with the POI on the map. This expands the available scenarios for the pointing interaction function and reduces the situation where visual recognition is interrupted due to environmental factors such as lighting and weather, resulting in a complete inability to respond.

[0029] By determining a sector-shaped search area based on the direction of the pointing ray, and identifying the three-dimensional spatial location information of candidate targets within that area from map data, the system can still provide target recognition services by relying on in-cabin gestures and map data even when external visual data is completely unavailable. Although the recognition accuracy is reduced, the basic usability of the interactive function is maintained, thereby improving robustness in sensor failures or extreme environments.

[0030] By outputting directional summary information based on the pointing ray, even in extreme cases where information sources are extremely limited, it can still provide users with a general prompt about the type of area or place corresponding to the pointing direction, reducing the occurrence of complete silence or error exit, thereby improving the continuity and friendliness of the user experience.

[0031] This application also provides an in-vehicle pointing interaction system (hereinafter referred to as "the system"), which is used to implement the above-described in-vehicle pointing interaction method.

[0032] This application also proposes an in-vehicle pointing interaction device, the in-vehicle pointing interaction device comprising: The first module is used to acquire cabin perception data of the pointing actions of the occupants inside the vehicle. The second module is used to generate the occupant's pointing ray based on the cabin perception data, and to map the pointing ray to the world coordinate system; The third module is used to obtain the three-dimensional spatial location information of candidate targets, wherein the candidate targets are external targets. The steps of obtaining the three-dimensional spatial location information of candidate targets include: collecting visual perception data through external sensors and calculating the three-dimensional bounding box of the candidate target based on the visual perception data; obtaining the three-dimensional bounding box of the interest point corresponding to the candidate target from map data; and using the three-dimensional bounding box and the three-dimensional bounding box as the three-dimensional spatial location information of the candidate target. The fourth module is used to determine the target that the occupant is pointing to from the candidate targets by matching the pointing ray in the world coordinate system with the three-dimensional spatial position information of the candidate targets, and output the description information of the target to the occupant.

[0033] This application also proposes a vehicle comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the in-vehicle pointing interaction method described above.

[0034] This application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the vehicle-mounted pointing interaction method described above.

[0035] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the in-vehicle pointing interaction method described above. Attached Figure Description

[0036] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0037] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 An optional system schematic diagram provided for an embodiment of this application; Figure 2 An optional flowchart is provided for an embodiment of this application; Figure 3 This is a schematic diagram illustrating an optional target screening method provided in an embodiment of this application. Figure 4 An optional guided calibration schematic diagram provided for an embodiment of this application; Figure 5 A schematic diagram of an optional data caching method provided in this application embodiment; Figure 6 A schematic diagram illustrating an optional hierarchical output provided for an embodiment of this application; Figure 7 This is a schematic diagram of an optional scenario provided for an embodiment of this application; Figure 8 A schematic diagram of an optional device module structure provided for an embodiment of this application; Figure 9 This is a schematic diagram of an optional hardware operating environment structure provided for an embodiment of this application.

[0039] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0040] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0041] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0042] This application provides an in-vehicle pointing interaction method, which can be applied to a vehicle cockpit system with an in-cabin camera, an external sensor (such as a camera, millimeter-wave radar or lidar), a high-precision positioning system and map data. This vehicle cockpit system is an in-vehicle pointing interaction system.

[0043] Reference Figure 1 , Figure 1This is a schematic diagram of an optional system according to an embodiment of the in-vehicle pointing interaction method of this application. The system includes: an in-cabin gesture recognition module, an external visual recognition module, a pointing ray calculation module, a map POI (Point of Interest) matching module, a target confirmation module, an information recommendation and presentation module, and optional real-time assurance module and calibration module.

[0044] Regarding the in-cabin gesture recognition module, it collects images of the occupants' hands through an in-cabin camera, extracts key skeletal points of the shoulder, elbow, wrist, and index fingertip, and outputs pointing posture data in the cockpit coordinate system.

[0045] Regarding the directional ray calculation module, it takes the line connecting the wrist to the tip of the index finger as the initial direction, and performs weighted correction based on the direction from the shoulder to the elbow to generate a directional ray in the cockpit coordinate system; then, based on the occupant's seat position and camera calibration parameters, it transforms the ray to the vehicle coordinate system, and finally maps the ray to the world coordinate system based on the vehicle's pose and heading angle.

[0046] Regarding the external vision recognition module, it uses external sensors (cameras / radar) to detect environmental targets and outputs the three-dimensional bounding box (center point, size, orientation) of each target.

[0047] Regarding the map POI matching module, it obtains the 3D bounding box and semantic information (name, category, popularity, etc.) of the point of interest from the map data based on the vehicle location and the direction of the pointing ray.

[0048] Regarding the target confirmation module, it performs intersection checks between the pointing ray in the world coordinate system and the 3D bounding box and 3D bounding frame, respectively, to generate a visual candidate list and a map candidate list. Targets that appear in both lists are directly confirmed as pointing targets; targets that appear in only one list are filtered to obtain the final target through methods such as distance sorting, saliency-weighted fusion, or interactive confirmation.

[0049] Regarding the information recommendation and presentation module, it outputs target description information (name, introduction, etc.) through HUD (Head-Up Display), central control screen, or voice broadcast based on vehicle speed and passenger identity.

[0050] Regarding the real-time performance guarantee module, it dynamically preloads map data of the area ahead into the local cache based on vehicle speed; when a passenger points to a location, it prioritizes matching the data in the cache to reduce response latency.

[0051] Regarding the calibration module, it guides the occupants to point in a known calibration direction, calculates a personalized pointing offset vector and corrects the ray, and adaptively updates the offset based on the user's selection.

[0052] The above modules work together to achieve vehicle-external target recognition and information recommendation based on natural pointing actions.

[0053] Reference Figure 2 , Figure 2 This is an optional flowchart provided for an embodiment of this application.

[0054] In this embodiment, the in-vehicle pointing interaction method includes steps S10 to S40: Step S10: Acquire cabin perception data of the pointing actions of the occupants inside the vehicle; Optionally, step S10 can be implemented using image acquisition devices within the vehicle cabin. For example, one or more in-cabin cameras can be installed inside the vehicle cabin, including but not limited to driver monitoring cameras and in-vehicle wide-angle cameras. These cameras are arranged to capture images of the upper body and hand areas of occupants (including the driver, front passenger, and rear passengers). When executing the pointing interaction process, the system first acquires image frames containing the occupant's hands and upper limbs from the real-time video stream captured by the in-cabin cameras.

[0055] Optionally, the in-cabin perception data may include images of the occupant's hands, which contain skeletal keypoint information for reconstructing pointing posture. In actual processing, the system preprocesses the acquired raw images, such as denoising and contrast enhancement, and then inputs the preprocessed images into a pre-trained hand posture estimation model. This model can use a deep learning network (such as a convolutional neural network or a graph convolutional network) to output two-dimensional coordinates (image coordinate system) of multiple hand and upper limb skeletal keypoints.

[0056] To more accurately represent pointing gestures, the extracted skeletal keypoints include at least four categories: shoulder keypoints (to determine the base position of the upper limb), elbow keypoints (to represent the direction of forearm extension), wrist keypoints (to determine the starting point of the hand), and index fingertip keypoints (to indicate the farthest point of the pointing gesture). In some examples, thumbtip keypoints can also be added to distinguish pointing gestures from other gestures such as hand gestures. The extraction method for these skeletal keypoints is not limited to deep learning models; traditional image processing methods (such as contour detection and corner detection) can also be used, as long as the keypoints can be reliably located from the image.

[0057] Step S20: Generate the occupant's pointing ray based on the cabin perception data, and map the pointing ray to the world coordinate system; After acquiring in-cabin perception data of occupant pointing movements, the system can generate a pointing ray in the cockpit coordinate system based on this data. Optionally, the in-cabin perception data may include images of the occupant's hands captured by an in-cabin camera. The system extracts multiple key points of the hand skeleton from the hand images, such as key points of at least the shoulder, elbow, wrist, and index fingertip. Based on this, the system uses the line connecting the wrist to the index fingertip as the initial pointing direction. However, relying solely on the wrist-finger line may be affected by slight wrist rotations, resulting in an unstable pointing direction. Therefore, optionally, the system can also correct the initial pointing direction by combining the extension direction from the shoulder to the elbow. For example, a weighting coefficient can be set to linearly weight and fuse the shoulder-elbow direction with the wrist-finger direction, ultimately obtaining a three-dimensional ray in the cockpit coordinate system with the corrected pointing direction. In other examples, without weighted correction, the wrist-finger line can be used directly as the pointing direction to reduce computational complexity, suitable for platforms with limited computing power.

[0058] After obtaining the directional ray in the cockpit coordinate system, it needs to be transformed to the vehicle coordinate system. To do this, the system can establish a transformation matrix from the image / cockpit coordinate system to the vehicle coordinate system based on the occupant's seating position (e.g., driver's seat, front passenger seat, rear left or right side) and the calibration parameters of the in-cabin camera (including intrinsic and extrinsic parameters, i.e., the camera's focal length, principal point coordinates, and its installation position and angle relative to the vehicle coordinate system). In actual calculations, the system multiplies the starting point and direction vector of the directional ray in the cockpit coordinate system by this transformation matrix to obtain the directional ray in the vehicle coordinate system. This ray is still a three-dimensional ray, with its starting point corresponding to the position of the occupant's eyes in the vehicle coordinate system, and its direction representing the pointing direction relative to the vehicle body. Specifically, before transforming the cockpit coordinate system to the vehicle coordinate system, the three-dimensional coordinates of the occupant's eyes in the cockpit coordinate system are first determined based on the occupant's seating position and / or the facial image captured by the in-cabin camera. Then, using the eye position as the starting point of the ray and the corrected pointing direction as the direction of the ray, the directional ray in the cockpit coordinate system is generated. Next, based on the occupant's seating position and the calibration parameters of the in-cabin camera, the directional ray originating from the occupant's eye in the cockpit coordinate system is converted into a directional ray in the vehicle coordinate system. At this point, the starting point of the ray is the position of the occupant's eye in the vehicle coordinate system, and its direction represents the pointing direction relative to the vehicle body.

[0059] After obtaining the pointing ray in the vehicle coordinate system, it is mapped to the world coordinate system for location matching with map POIs or visually detected targets. The system can collect at least one of the following types of data: the vehicle's current pose (e.g., position and attitude angles fused from the inertial measurement unit (IMU), wheel speedometer, and high-precision positioning system), the vehicle's current heading angle (i.e., the angle between the vehicle's frontal orientation and true north), and the installation pose (calibration parameters relative to the vehicle coordinate system) of external sensors (such as forward-facing cameras and LiDAR). Using this data, the system can construct a transformation relationship from the vehicle coordinate system to the world coordinate system. For example, the starting point of the pointing ray in the vehicle coordinate system can be translated to the actual geographic coordinates of the occupant's eyes in the world coordinate system, and the direction vector can be rotated to the absolute direction in the world coordinate system, thereby obtaining a three-dimensional ray extending in the world coordinate system from the occupant's eye position along the pointing direction.

[0060] Optionally, the cockpit coordinate system uses the optical center of the in-cabin camera as its origin, with the X-axis pointing positive to the right along the vehicle's transverse direction, the Y-axis pointing positive forward along the vehicle's longitudinal direction, and the Z-axis pointing positive vertically upward, used to describe the spatial position of the occupant's body parts relative to the camera; the vehicle coordinate system uses the projection point of the vehicle's rear axle center on the ground as its origin, with the X-axis pointing positive forward along the vehicle's longitudinal direction, the Y-axis pointing positive to the left along the vehicle's transverse direction, and the Z-axis pointing positive vertically upward, used to unify the data from the vehicle's sensors; the world coordinate system adopts the universal transverse Mercator projection coordinate system, with its origin being the global zero point of the high-precision map, the X-axis pointing due east, the Y-axis pointing due north, and the Z-axis pointing to the zenith, used to describe the absolute geographic spatial position.

[0061] Optionally, the transformation from the cockpit coordinate system to the vehicle coordinate system can be calculated based on the calibration parameters of the in-cabin camera and the occupant's seat position offset; the transformation from the vehicle coordinate system to the world coordinate system can be calculated based on the vehicle's current pose and heading angle.

[0062] Physical quantities such as the occupant's eye position, the origin and direction of the pointing ray, etc., have different coordinate expressions in different coordinate systems, but these expressions are interconnected through strict coordinate transformation formulas. Optionally, using the vehicle coordinate system as the reference coordinate system, the coordinates of the eyes in the vehicle coordinate system are first estimated based on the occupant's seating position and human body model; then, using the calibration parameters of the in-cabin camera, the eye coordinates are transformed down to the cockpit coordinate system to assist in gesture recognition; simultaneously, using the vehicle's real-time pose, the eye coordinates are transformed up to the world coordinate system for spatial matching with map POIs. This ensures that the same physical point is always described in different coordinate systems, guaranteeing the consistency of the pointing ray calculation.

[0063] In one feasible implementation, the step of acquiring cabin perception data of occupant pointing actions includes: Images of the occupants' hands are captured by an in-cabin camera, and multiple key points of the hand bones, including at least the shoulder, elbow, wrist, and fingertip, are extracted from them. The step of generating the occupant's pointing ray based on the cabin sensing data includes: Using the line connecting the wrist to the tip of the index finger as the initial pointing direction, and combining the extension direction from the shoulder to the elbow, the initial pointing direction is weighted and corrected to obtain the pointing ray of the occupant in the cockpit coordinate system.

[0064] The system captures images of the occupant's hands using in-cabin cameras and extracts multiple key points of the hand skeleton from these images. The in-cabin cameras can include driver monitoring cameras, in-vehicle wide-angle cameras, or specially positioned infrared depth cameras; their specific installation locations can be adjusted according to the vehicle model and cabin layout. For example, the extracted key points of the hand skeleton include at least four points: the shoulder, elbow, wrist, and the tip of the index finger. By identifying and tracking the positions of these key points in the image coordinate system, the system can obtain the spatial relative relationships of the various joints of the occupant's upper limbs.

[0065] After obtaining the key points of the hand bones mentioned above, the pointing ray is further calculated. For example, the line connecting the wrist to the tip of the index finger is used as the initial pointing direction. This is because in a person's natural pointing motion, the index finger's direction is the most obvious, and the line connecting the wrist and the index finger, as the base of the index finger, can reflect the finger's pointing direction well. However, relying solely on the direction from the wrist to the tip of the index finger can sometimes be affected by the wrist rotation angle, leading to a deviation between the pointing direction and the occupant's true intention. To reduce this deviation, the extension direction from the shoulder to the elbow is introduced to weight and correct the initial pointing direction. The line connecting the shoulder to the elbow reflects the overall orientation of the upper arm. When the occupant makes a pointing motion, the orientation of the upper arm is usually more stable and has a more global pointing characteristic than the local rotation of the wrist. By weightedly fusing the wrist-index finger direction and the shoulder-elbow direction, a more robust pointing ray can be obtained.

[0066] After the above corrections, the system obtains the pointing ray in the cockpit coordinate system. This ray originates from the occupant's eye position. The coordinates of the occupant's eye position in the cockpit coordinate system are obtained by inverse coordinate transformation using the calibration parameters of the in-cabin camera, based on the eye coordinates previously estimated in the vehicle coordinate system (which can be estimated from seat position and human body model). This ray extends outward along the corrected pointing direction. The cockpit coordinate system is a coordinate system defined relative to the vehicle's internal space. It typically uses the vehicle center or a fixed reference point as the origin, with the direction of the vehicle's front as the positive X-axis, the direction to the driver's left as the positive Y-axis, and the vertically upward direction as the positive Z-axis. In the cockpit coordinate system, the pointing ray describes the positional relationship of the occupant's pointing direction relative to the vehicle itself, providing a foundation for subsequent mapping to the vehicle coordinate system and even the world coordinate system.

[0067] In one feasible implementation, the step of mapping the pointing ray to the world coordinate system includes: Based on the occupant's seating position and the calibration parameters of the in-cabin camera, the pointing ray in the cockpit coordinate system is converted into a pointing ray in the vehicle coordinate system; Based on the vehicle's current pose, current heading angle, and the installation pose of the external sensors, the pointing rays in the vehicle coordinate system are converted into pointing rays in the world coordinate system.

[0068] The system first converts the pointing ray in the cockpit coordinate system to a pointing ray in the vehicle coordinate system based on the occupant's seating position and the calibration parameters of the in-cabin camera. Since the occupants in different seats (driver's seat, front passenger seat, rear left, rear right) have different positions within the vehicle, and the camera installation positions and internal parameters vary between different vehicle models, known calibration information is needed for this conversion. In one example, the in-cabin camera is calibrated before leaving the factory, and its intrinsic parameters (focal length, principal point coordinates, distortion coefficients) and extrinsic parameters (rotation matrix and translation vector of the camera relative to the vehicle coordinate system) are pre-stored in the system. When an occupant performs a pointing action, the system first obtains the occupant's seating position information (obtainable through seat sensors or manual user settings) and calls the spatial offset of the corresponding seating area. Then, the system performs a matrix transformation on the pointing ray in the cockpit coordinate system (represented by the ray's origin and direction vector) with the camera's extrinsic parameters and the seat offset to obtain the expression of the ray in the vehicle coordinate system.

[0069] After obtaining the pointing ray in the vehicle coordinate system, the system further converts the pointing ray into a pointing ray in the world coordinate system based on the vehicle's current pose, current heading angle, and the installation pose of the external sensors. The world coordinate system is an absolute coordinate system fixed to the Earth (such as a latitude, longitude, and elevation coordinate system) used to describe geographical location and target position in the global environment. Since the vehicle's position and attitude constantly change during driving, the system needs to acquire the vehicle's pose information in real time. In one example, the system obtains the vehicle's current position (longitude, latitude, altitude) and attitude (roll angle, pitch angle, yaw angle) through an onboard positioning module (such as a Global Navigation Satellite System, Inertial Measurement Unit, wheel speedometer, etc.).

[0070] Step S30: Obtain the three-dimensional spatial location information of the candidate target; Optionally, the system can acquire visual perception data through external sensors and calculate the 3D bounding boxes of candidate targets based on the visual perception data. These external sensors may include forward-facing cameras, side-facing cameras, or surround-view cameras. Taking a vehicle driving scenario as an example, when targets such as buildings, traffic signs, other vehicles, or pedestrians appear in front of the vehicle, the external visual recognition module performs target detection and 3D reconstruction on the images acquired by the cameras, outputting the 3D bounding box of each detected target in the vehicle coordinate system or world coordinate system. This 3D bounding box is typically defined by parameters such as the center point coordinates, length, width, height, and orientation angle, and can accurately describe the target's spatial occupancy. In this way, the system can obtain the 3D position information of dynamic or static targets in the environment in real time.

[0071] Optionally, the system can obtain the 3D bounding boxes of points of interest (POIs) corresponding to candidate targets from map data, which can be high-definition maps or standard navigation maps. Taking high-definition maps as an example, the map data pre-stores the location coordinates and 3D contour information of a large number of POIs (such as shopping malls, hotels, gas stations, attractions, etc.), with each POI associated with a 3D bounding box. Based on the vehicle's current location and direction of travel, the system queries the map data for POIs located near the directional ray and extracts their 3D bounding boxes. This method is particularly suitable for targets with fixed geographical locations, such as buildings and landmarks.

[0072] Optionally, the system can employ both methods simultaneously, using both the 3D bounding box obtained through visual perception calculations and the 3D bounding box acquired from map data as the 3D spatial location information of the candidate target. For example, when an occupant points to a building outside the vehicle, the external vision system may detect the building's 3D bounding box, while the map data also stores the 3D bounding box of the building's POI. Sending both data together to the subsequent matching module improves the reliability and confidence of target recognition.

[0073] In one feasible implementation, the step of obtaining the three-dimensional spatial location information of the candidate target includes: Visual perception data is collected by external sensors, and the three-dimensional bounding box of the candidate target is calculated based on the visual perception data. Obtain the 3D bounding boxes of the points of interest corresponding to the candidate targets from the map data; The three-dimensional bounding box and the three-dimensional enclosing box are used as the three-dimensional spatial location information of the candidate target.

[0074] Optionally, external sensors may include forward-looking cameras, side-looking cameras, or surround-view cameras. Taking the vehicle's forward field of view as an example, the external vision recognition module acquires image sequences output by the cameras in real time and uses deep learning object detection algorithms to identify various targets in the images, such as buildings, traffic signs, other vehicles, and pedestrians. Subsequently, combining the intrinsic and extrinsic parameters (i.e., calibration parameters) of the sensors, through monocular depth estimation or binocular stereo matching, or with the assistance of LiDAR point cloud (if the vehicle is equipped), the 3D bounding box of each target in the vehicle coordinate system or world coordinate system is calculated. The 3D bounding box is usually represented by parameters such as the target's center point coordinates (x, y, z), 3D dimensions (length, width, height), and orientation angle (yaw angle), which can accurately describe the target's occupied area in physical space and can be directly used for subsequent intersection determination with pointing rays.

[0075] Optionally, the system can obtain the 3D bounding boxes of Points of Interest (POIs) corresponding to candidate targets from high-definition maps or standard navigation maps. High-definition maps typically pre-store the geographic coordinates, name, category, and 3D contour information (such as building height, roof shape, number of floors, etc.) of POIs, with each POI associated with a 3D bounding box. Based on the vehicle's current location and the direction of the directional ray, the system queries the map data for POIs within a certain range near the ray and extracts their 3D bounding boxes. For standard navigation maps, if precise 3D contour information is lacking, the system can use the POI's 2D location coordinates combined with a default height value (e.g., setting a typical height based on the building category) to estimate an approximate 3D bounding box. This method is particularly suitable for static targets with fixed geographic locations, such as buildings, landmarks, and shops.

[0076] Optionally, the system uses both the 3D bounding boxes obtained from visual perception and the 3D bounding boxes obtained from map data as the 3D spatial location information of candidate targets. These two sources of information are complementary: visual perception can detect dynamic targets (such as temporarily parked vehicles and pedestrians) and temporary objects not labeled on the map, while map data provides rich semantic labels and stable prior location information, and remains effective even under visually limited conditions (such as backlighting or nighttime). Simultaneously retaining both and sending them to subsequent matching modules (e.g., intersecting with directional rays) can significantly improve the reliability and robustness of target recognition.

[0077] Step S40: By matching the pointing ray in the world coordinate system with the three-dimensional spatial position information of the candidate targets, the target pointed to by the crew is determined from the candidate targets, and the description information of the target is output to the crew.

[0078] The matching process can employ a ray-bounding box intersection test. For each candidate target, the system determines whether the pointing ray intersects with the spatial region (such as a 3D bounding box) defined by the target's 3D spatial location information. If the pointing ray intersects with the spatial region of a target, and this intersection point is in the positive extension direction of the ray and is closest to the occupant, then that target is considered a preliminary candidate for matching. Since multiple candidate targets may intersect with the ray, the system needs to further filter these candidate targets to select the final target.

[0079] Furthermore, the system can simultaneously utilize 3D spatial location information from both visual perception and map data sources for dual matching, and then make fusion decisions based on the matching results. For example, the system first determines the intersection of the pointing ray in the world coordinate system with the 3D bounding box output by the external visual recognition module, obtaining all visually detected targets that intersect with the ray, forming a list of visual candidate targets. Each visual candidate target includes 3D location information, as well as target category labels (such as "building," "vehicle," "pedestrian," etc.) and detection confidence. Simultaneously, the system determines the intersection of the same pointing ray with the 3D bounding box of a point of interest (POI) in the high-precision map, obtaining all map POIs that intersect with the ray, forming a list of map candidate targets. Each map candidate target includes POI name, category, popularity score, and detailed description information. Next, the system performs fusion matching on the two lists: for the same target that appears in both the visual candidate target list and the map candidate target list (e.g., a visually detected building whose 3D bounding box position basically coincides with the 3D bounding box of a POI in the map), the system classifies this target as a high-confidence candidate target, which can be directly output as the target for occupant pointing. For candidate targets that only appear in one of the lists (e.g., a temporarily parked vehicle is detected visually, but there is no such POI in the map data; or a POI exists in the map, but it is not detected visually, for example, due to occlusion or insufficient light), the system classifies such targets as low-confidence candidate targets, which need to enter the ambiguity resolution process for further screening.

[0080] When only a single source of 3D spatial location information is available (e.g., lack of high-precision map data or visual sensor failure), the system can rely solely on that source for matching. For example, when only visual 3D bounding boxes are available, the system directly performs intersection tests between the pointing ray and each bounding box, and selects the intersection target closest to the occupant as the pointing target; when only map POI 3D bounding boxes are available, the system similarly performs ray-bounding box intersection tests and recommends targets based on distance or POI popularity.

[0081] The target's descriptive information can be obtained from a cloud-based knowledge base or local cache, including but not limited to the target's name, category, brief introduction, detailed description, images, and user reviews. The output method can be flexibly selected based on the vehicle's driving status, occupant preferences, and system configuration to balance the effectiveness of information delivery with driving safety.

[0082] Optionally, when the vehicle is in motion (e.g., at speeds exceeding a certain threshold), the system employs a non-intrusive presentation method to reduce driver distraction. Specifically, the system can overlay the name and brief information of the target onto the windshield at the corresponding location using a head-up display (HUD), while simultaneously providing a voice introduction. For instance, when a passenger points to a historical building outside the vehicle, the HUD displays its name and a brief description in the corresponding field of vision, while the vehicle's voice system announces, "The building ahead is a historical building, built in XX year." In this approach, the system avoids displaying the central control interface or large information cards, thus reducing obstruction of the driver's view and distraction.

[0083] Optionally, when the vehicle is parked or traveling at low speeds (e.g., below a certain threshold), the system can present information in richer formats. For example, it can display a detailed information card of the target on the central control screen, which may include multiple images, a complete text description, user ratings, and interactive elements (such as "More Details" or "Navigate To"). Simultaneously, the system can still provide a summary introduction via voice prompts. This approach provides occupants with more comprehensive information, meeting their needs for in-depth understanding, and has a smaller impact on driving safety due to the low speed or stationary state of the vehicle.

[0084] Optionally, the system can adaptively adjust the output method based on the occupant's identity (driver or front passenger). For example, when the pointing action is performed by the driver and the vehicle is in motion, the system prioritizes a lightweight presentation method using HUD + voice; when the pointing action is performed by the front passenger, even if the vehicle is in motion, information cards can be displayed on the central control screen or the front passenger entertainment screen, because the front passenger's operation has a relatively small impact on the driver's driving safety.

[0085] Optionally, multimodal output collaboration is also supported. For example, the HUD displays only the target name, while a brief information is displayed in a small window in the corner of the central control screen, and a full introduction is given via voice. Passengers can interactively control the information presentation through voice commands (such as "tell me more details" or "change to another one") or gestures (such as waving left or right to switch recommended targets).

[0086] In one feasible implementation, the step of matching the pointing ray in the world coordinate system with the three-dimensional spatial position information of the candidate target includes: The first intersection determination is performed between the pointing ray in the world coordinate system and the three-dimensional bounding box to obtain a list of visual candidate targets. A second intersection determination is performed between the pointing ray in the world coordinate system and the three-dimensional bounding box to obtain a list of candidate targets on the map. Candidate targets that appear simultaneously in both the visual candidate target list and the map candidate target list will be used as the targets pointed to by the occupants. For candidate targets that only appear in the visual candidate target list or the map candidate target list, the target pointed to by the occupant is obtained by filtering.

[0087] The system first performs an initial intersection determination between the pointing ray in the world coordinate system and the 3D bounding box obtained through external visual perception. The pointing ray is defined by its origin (e.g., the user's eye position in the world coordinate system) and its direction vector. For each 3D bounding box output by the external vision module (typically described by its eight vertices or center point plus length, width, height, and orientation angle), the system calculates whether the pointing ray intersects with the bounding box. Intersection determination can employ classic geometric algorithms using ray-aligned bounding boxes or oriented bounding boxes, for example, by solving the parametric equations of the ray and finding the intersection points with the six faces of the bounding box, and checking whether the intersection points fall within the valid range. Targets corresponding to 3D bounding boxes that intersect with the pointing ray are added to a visual candidate target list. Each entry in this list contains at least the target's identifier, 3D location, category label (e.g., "building," "vehicle," "pedestrian," etc.), and distance parameters to the intersection point. This allows for the rapid filtering of targets geometrically located near the occupant's pointing direction and detected by external sensors.

[0088] The system simultaneously performs a second intersection determination between the pointing ray in the world coordinate system and the 3D bounding boxes of Points of Interest (POIs) obtained from the map data. Similar to the first intersection determination, the system queries the map data for POIs located within a certain range in front of the vehicle, with each POI associated with a 3D bounding box (e.g., determined by the foundation outline and height range of a building). The system calculates the intersection of the pointing ray with the 3D bounding box of each POI and adds all intersecting POIs to the map's candidate target list. Entries in this list include at least the POI's name, category, popularity rating (e.g., based on historical visit frequency or user reviews), 3D bounding box, and intersection distance. Thus, utilizing prior information from the map data, the system identifies fixed targets with clear geographic semantics in the pointing direction, even if the target is temporarily not detected by visual sensors (e.g., partially obscured by trees).

[0089] In practical applications, after obtaining the visual candidate target list and the map candidate target list, the system needs to fuse and match the two lists to improve the confidence of target recognition. Optionally, the system first identifies candidate targets that appear in both lists simultaneously. For example, a building may be detected by external vision and form a 3D bounding box, and a corresponding POI 3D bounding box may also exist in the map data, with their spatial locations largely overlapping. The system directly identifies this target as the target pointed to by the occupants, without further filtering. Because it integrates information from both visual perception and map data, the recognition of this type of target has a high confidence level.

[0090] For candidate targets that appear only in one of the lists (i.e., only in the visual candidate target list or only in the map candidate target list), the system does not output them directly. Instead, it marks them as low-confidence candidates and proceeds to further filtering processes (such as distance ranking, saliency-weighted fusion, interactive confirmation, or deep learning models). For example, a target appearing only in the visual candidate target list might be a temporarily appearing object (such as construction cones or temporarily parked vehicles) or a newly constructed building not yet included in the map data. The system still needs to filter it, but can provide auxiliary information such as "the target is not marked on the map" when outputting the result. For targets appearing only in the map candidate target list, the target might be occluded by obstacles and not detected by the visual sensor. The system will rely on map information for filtering and provide explanations such as "based on location estimation."

[0091] Through the aforementioned dual-list fusion matching mechanism, the system can fully utilize the complementarity of multi-source data, ensuring rapid response to high-confidence targets while not overlooking low-confidence targets from a single source, thus balancing the accuracy and recall of identification.

[0092] It should be understood that the first intersection determination and the second intersection determination can be executed in parallel or sequentially. In scenarios with limited computing resources, the system can also execute the first intersection determination first, and only trigger the second intersection determination when the visual candidate target list is empty or the number of candidates is too small, in order to save power consumption. This application does not limit the execution order of the two determination steps.

[0093] Optionally, the determination of whether a point appears in both lists simultaneously requires a spatial location consistency check. For example, the system can set a spatial distance threshold (e.g., 1 meter). Only when the distance between the center of the visual 3D bounding box and the center of the map POI's 3D bounding box is less than this threshold, and their category semantics are compatible (e.g., the visual recognition of "building" matches the map POI's "shopping mall"), are they considered to point to the same target. This verification method can reduce erroneous fusion caused by positioning errors.

[0094] In one feasible implementation, refer to Figure 3 The step of filtering to obtain the target pointed to by the occupant includes: The candidate targets are sorted from closest to furthest from the occupant, and the candidate target closest to the occupant is taken as the target the occupant is pointing to. Alternatively, a salience score can be calculated based on the attribute characteristics of the candidate target, and the salience score can be weighted and fused with the distance ranking weight of the candidate target. The candidate target with the highest weighted fusion score can be used as the target that the occupant is pointing to. The attribute characteristics include the physical size of the candidate target, the popularity score, and the angular deviation between the candidate target and the occupant's line of sight. The popularity score is determined based on the historical access frequency of map points of interest or user evaluation scores. Alternatively, when the difference in significance scores between the top two candidate targets in the weighted fusion score ranking is less than a preset threshold, an interactive confirmation mechanism with the occupant is triggered, and the candidate target selected by the occupant is taken as the target pointed to by the occupant. Alternatively, the occupant's gesture features, vehicle pose, and candidate target set can be input into a deep learning model, and the candidate targets output by the deep learning model can be used as the targets pointed to by the occupant.

[0095] In some embodiments, the system can sort candidate targets by distance from the occupant to the nearest, and select the closest candidate target as the target the occupant is pointing at. Specifically, for each candidate target that appears only in a single list, the system obtains the target's 3D spatial position (e.g., the center point of the 3D bounding box or the near-Earth corner of the 3D bounding box) and calculates its Euclidean distance to the occupant (typically with the occupant's shoulder position or the vehicle's center of gravity as a reference point). Then, all candidate targets are sorted in ascending order of distance, and the closest target is selected as the final output. This method is computationally simple, has a fast response time, and is suitable for scenarios where the distance between candidate targets differs significantly, such as when the occupant points to a lone tree outside the window, which is closest to the vehicle and has no other nearby interfering targets.

[0096] In other embodiments, the system can calculate a saliency score based on the attribute features of candidate targets, and then weight and fuse the saliency score with the distance ranking weight of the candidate targets, selecting the candidate target with the highest weighted fusion score as the target pointed to by the occupant. The attribute features may include the physical size of the candidate target, its popularity score, and the angular deviation between the candidate target and the occupant's line of sight. Specifically, the system first normalizes each candidate target: the distance value is normalized to a distance score (the closer the distance, the higher the score); the physical size is normalized to a size score (the larger the volume, the higher the score); the popularity score (e.g., based on historical visit frequency or user rating scores of map POIs) is normalized to a popularity score; and the angular deviation between the candidate target and the occupant's line of sight is normalized to an angle score (the smaller the deviation, the higher the score). Then, weight coefficients are assigned to the scores of each dimension (e.g., distance weight 0.4, size weight 0.2, popularity weight 0.2, angle weight 0.2), and a weighted sum is calculated to obtain the saliency score. Simultaneously, distance ranking itself is treated as a separate weighting factor (e.g., the first-ranked target is assigned a weight of 1.0, the second-ranked target 0.8, and so on, decreasing sequentially). Finally, the salience score is multiplied or added to the distance ranking weights to obtain the final score for each candidate target, and the highest-scoring target is selected as the target. This method can comprehensively reflect multiple characteristics of the target. For example, when pointing to a large landmark building in the distance (such as a Ferris wheel) and a small newsstand nearby, although the newsstand is closer, the large building may have higher salience and visibility. After weighted fusion, the system may select the large building as the recommendation result, which is more in line with the actual intention of the occupant. The specific values ​​of each weight coefficient can be determined by those skilled in the art through offline calibration or online learning; this application does not impose any limitations on this.

[0097] In other embodiments, when the difference in significance scores between the top two candidate targets in the aforementioned weighted fusion score ranking is less than a preset threshold, the system can trigger an interactive confirmation mechanism with the occupant and use the candidate target selected by the occupant as the target. The preset threshold can be set according to actual needs, for example, 0.1 (normalized score range 0~1). When the difference in scores between the first and second place is less than 0.1, the confidence of the automatic decision is considered insufficient, and user confirmation is required. The interactive confirmation mechanism can be implemented in various ways. As an example, the system can display a candidate list on the central control screen or HUD (e.g., displaying "Do you mean A or B?"), and the user can select one of the targets by swiping left or right or touching. As another example, the system can use voice prompts "Do you mean A or B?", and the user can confirm by replying "A" or "B" with voice. As yet another example, the system can detect the slight direction of the user's head rotation: when the user maintains the pointing posture, if the head turns to one side, the target corresponding to that side is confirmed as the target. By introducing interactive confirmation, the system can respect the user's explicit intent when automatic decision-making is ambiguous, thereby reducing recommendation errors and improving user experience.

[0098] In other embodiments, the system can input occupant gesture features, vehicle pose, and a set of candidate targets into a deep learning model, and use the candidate targets output by the model as the targets the occupant points to. The deep learning model can be an intent recognition network based on a graph neural network or Transformer architecture. The model's input includes: a sequence of key points on the occupant's hand skeleton (e.g., the 3D coordinates of the wrist and index fingertip), the occupant's head pose (pitch, yaw, and roll angles), the vehicle's current position and heading angle, and the feature vector of each target in the candidate target set (including position coordinates, size, category, popularity, etc.). The model is pre-trained to learn a non-linear mapping from multimodal inputs to the probability distribution of the pointing target. The output layer is the confidence score of each target in the candidate target set, and the system selects the target with the highest confidence score as the final output. This approach is suitable for scenarios where a large amount of real-world pointing interaction data has been collected. The model can implicitly learn complex pointing habits and environmental factors without requiring explicit weighting rules or thresholds, and has strong generalization capabilities. Regarding the specific network structure and training methods of deep learning models, those skilled in the art can select and implement them based on conventional practices, and this application will not elaborate further.

[0099] It should be understood that the above four screening methods are not mutually exclusive, and the system can dynamically select or combine them according to the actual scenario. For example, on vehicle-mounted platforms with limited computing power, the distance-first method can be used first; in areas with abundant POI attribute data, the saliency-weighted fusion method can be used; when the confidence level of the automatic decision is low, it can automatically fall back to the interactive confirmation method; when there is strong computing power and training data support in the cloud, the deep learning model method can be enabled. This application does not limit this.

[0100] In one feasible implementation, refer to Figure 4 The method further includes: The system guides the occupants to point in multiple known calibration directions. Based on the deviation between the occupants' actual pointing direction and the calibration direction, the system calculates the occupants' pointing offset vector. The pointing offset vector is used to correct the occupants' pointing ray and is adaptively updated based on the occupants' selection of candidate targets.

[0101] As an optional implementation, the system can guide the occupant to point to multiple known calibration directions upon first use or when the calibration is actively triggered by the occupant. The selection of calibration directions must meet two conditions: first, the precise orientation of the calibration target in the vehicle coordinate system or world coordinate system is known; second, the occupant can naturally point to these targets from inside the vehicle. For example, the system can display a calibration guidance interface on the central control screen or HUD, instructing the occupant to point sequentially to three known calibration targets outside or inside the vehicle, such as the left rearview mirror, right rearview mirror, and interior rearview mirror. The direction vectors of these three calibration targets in the vehicle coordinate system can be obtained in advance through vehicle design parameters or precisely calculated through camera calibration and vehicle geometry. When the occupant points to a calibration target as prompted, the system collects key points of the occupant's hand bones using the aforementioned method, calculates the actual pointing ray in the cockpit coordinate system, and then maps it to the vehicle coordinate system to obtain the actual direction vector. Since the true direction vector of the calibration target is known, the system can calculate the deviation between the actual direction and the true direction. This deviation may include angular errors (pitch angle, yaw angle) and offsets, etc. By statistically fitting the deviations of multiple calibration directions (e.g., taking the mean or using the least squares method), the system calculates a personalized pointing offset vector for the current crew member. This vector is used to correct subsequent pointing rays: after each pointing ray is generated, the system superimposes the offset vector onto the original ray to obtain the corrected ray, which is then used to match with candidate targets.

[0102] Crew members' pointing habits are not static and may undergo subtle changes over time, or the limited sample size during the initial calibration may result in an inaccurate offset vector. To continuously improve pointing accuracy, the system can also adaptively update the pointing offset vector based on the crew member's selection of candidate targets. Specifically, after the system outputs a list of candidate targets based on the corrected pointing ray and determines the crew member's final target selection through methods such as distance priority, significance weighting, or interactive confirmation (e.g., crew member voice confirmation, manual selection, or determination by the slight turning direction of the head), the system can infer that in this interaction, the ideal pointing ray should be the ray that precisely hits the center of the target or has the highest intersection test score. Therefore, the system can calculate the deviation between the original pointing ray actually generated and the ideal pointing ray, using this deviation as a new observation sample, and update the current pointing offset vector online using methods such as moving average or exponentially weighted moving average.

[0103] The aforementioned calibration and update process can be triggered in various scenarios. For example, if the system detects an occupant seated for the first time after the vehicle starts and has not yet performed the calibration, it will automatically prompt the occupant to perform a steering calibration; the occupant can also manually initiate the calibration through the central control settings interface. Adaptive updates, on the other hand, run continuously in the background by default, without requiring the occupant's awareness, thereby gradually improving personalized steering accuracy without increasing the interaction burden.

[0104] Furthermore, the selection of calibration targets is not limited to the three rearview mirrors mentioned above. Traffic signs or lampposts with fixed locations outside the vehicle, or points of interest (POIs) with known coordinates in the navigation map (such as gas stations or charging stations) can also be used as calibration targets, provided that the system can accurately determine the three-dimensional positions of these targets in the world coordinate system. In addition, for multi-occupant scenarios (such as the driver and front passenger seats), the system can maintain independent pointing offset vectors for each seat, distinguishing different occupants through facial recognition or seat sensors, and performing calibration and updates accordingly.

[0105] In one feasible implementation, refer to Figure 5 The method further includes: Based on the vehicle's motion status, predict potential targets that will soon enter the vehicle's forward field of view, and preload the map data corresponding to the potential targets into the local cache; wherein, the pre-loading range is dynamically adjusted according to the vehicle speed. When the occupant performs a pointing action, the target corresponding to the pointing ray is matched first in the local cache.

[0106] As an optional implementation, the system acquires the vehicle's current speed and heading angle in real time. Based on the vehicle's kinematic model, it predicts the area the vehicle will travel through in the future within a preset time period (e.g., 2 seconds). This area is typically defined as a fan-shaped or rectangular field of view extending along the heading direction from the vehicle's current position. For Points of Interest (POIs) within this area, the system preloads their relevant information from a map database (high-precision map or online map service), including but not limited to the POI's name, category, 3D bounding box, descriptive image, and popularity rating. The loaded data is stored in a local cache (e.g., a hash table or LRU cache in memory) for quick access later. For example, when the vehicle is traveling at 60 km / h on an urban road, the system predicts that the vehicle will travel approximately 33 meters in the next 2 seconds, so it loads POI data within a 100-meter range ahead. If the vehicle enters a highway and its speed increases to 120 km / h, the field of view changes more rapidly, and the system can correspondingly expand the preloading range to 300 meters to ensure cache hit rate. The dynamic adjustment of the preloading range can be determined by a linear or piecewise function between vehicle speed and loading distance, and this application does not impose specific limitations on this.

[0107] In some embodiments of this application, the pre-loading range is dynamically adjusted based on vehicle speed. For example, multiple speed ranges can be set, each corresponding to a loading radius: when the vehicle speed is less than 40 km / h, the loading radius is 200 meters; when the vehicle speed is between 40 km / h and 80 km / h, the loading radius is 350 meters; and when the vehicle speed is greater than 80 km / h, the loading radius is 500 meters. A continuous function can also be used, such as loading radius = base radius + vehicle speed × coefficient. This dynamic adjustment mechanism reduces loading too much unnecessary data at low speeds and helps cache sufficiently distant targets at high speeds, improving the hit rate of target matching. The system can also dynamically reduce or increase the loading range based on the current computational load and remaining memory capacity to balance performance and resources.

[0108] When an occupant performs a pointing action and generates a pointing ray, the system first searches for candidate targets in its local cache. Specifically, it performs an intersection test between the pointing ray and the 3D bounding boxes of all Points of Interest (POIs) in the cache (or performs a quick filter based on spatial indexing according to the ray direction). If a POI in the cache intersects with the ray, that POI is directly returned as a matching result, and the system proceeds to the subsequent target confirmation and information recommendation process. Because the cached data is stored in memory and requires no network requests or disk I / O, the latency of this matching process can be kept low, for example, within 300 milliseconds. When no candidate target is found in the cache, the system falls back to the complete three-way fusion matching path (i.e., simultaneously using external visual recognition, map POI lookup, and the aforementioned pointing ray matching). This fast matching path significantly reduces the response time of conventional pointing interactions and improves the smoothness of the user experience.

[0109] To ensure the timeliness and effectiveness of the cache, the system needs to dynamically update the preloaded content. Whenever the vehicle's position, heading, or speed changes significantly (e.g., traveling more than a certain distance or turning more than a certain angle), the system recalculates the prediction region, removes POIs no longer within the prediction region from the cache (or marks them as low priority), and loads data for newly entered POIs. The cache eviction policy can employ a distance- or time-based First-In-First-Out (FIFO) or Least Recently Used (LRU) algorithm. For example, a lifespan (e.g., 30 seconds) can be set for each POI; POIs that exceed their lifespan and are not matched can be evicted to free up memory space.

[0110] In some scenarios, such as when the vehicle has just started and the cache is empty, or when the occupant's pointing direction deviates significantly from the predicted driving direction (e.g., pointing to the side or rear), a cache miss may occur. In this case, the system performs a full path matching and adds information about the target being pointed to and its surrounding area to the cache as preparation for possible future pointing. Furthermore, if the system detects that the occupant frequently points in a direction that has not been pre-loaded (e.g., repeatedly pointing to the same viewpoint outside the right window), it can dynamically adjust the prediction model to include the area in that direction within the pre-loading range.

[0111] In one feasible implementation, refer to Figure 6 The method further includes: When the external sensors cannot identify the category of the candidate target, the three-dimensional spatial position information of the candidate target is determined based on the outline position of the candidate target; When the external sensors fail, a fan-shaped search area is determined based on the direction of the pointing ray, and the three-dimensional spatial location information of the candidate target is determined from the map data within the fan-shaped search area. When both external sensors and map data are unavailable, directional summary information is output based on the directional ray.

[0112] As an alternative implementation, when external sensors can detect the outline of a target but cannot identify its specific category (e.g., in backlight or rain / fog conditions, when the target detection algorithm can only output the object category and cannot distinguish between buildings, vehicles, or pedestrians), the system can still determine the three-dimensional spatial location information of the candidate target based on its outline position. Specifically, the external vision recognition module calculates the approximate three-dimensional bounding box of the target (including center coordinates, size, and orientation) based on edge information and depth estimation in the image. Although lacking semantic labels, this three-dimensional bounding box can still be used to determine intersection with pointing rays, and the intersection result is used as a candidate target. Subsequently, the system can combine this with POI locations in map data for matching: for example, associating the visual output "an object exists at a certain location" with POIs near the same location on the map, thereby indirectly inferring the target's identity. In this way, the system can maintain a certain target recognition accuracy even when visual recognition capabilities decline.

[0113] As an alternative implementation, when external sensors completely fail (e.g., a camera is completely blocked, its lens is damaged, or power is interrupted), the system no longer relies on visual data, but instead relies solely on in-cabin gesture recognition, vehicle pose, and map data to determine the pointing target. Specifically, the system determines a fan-shaped search area based on the direction of the pointing ray (mapped to the world coordinate system). The vertex of this fan-shaped search area is the user's eye position, the center line is the direction of the pointing ray, and the fan angle can be dynamically adjusted according to the confidence level of the pointing accuracy, for example, the default setting is ±15 degrees. The system retrieves all POIs located within this fan-shaped area from the map data and sorts them according to the distance between the candidate targets and the user, from closest to farthest, or in combination with the POI's popularity rating, selecting the top-ranked candidate targets as possible pointing results. Simultaneously, the system can add prompts when presenting the results, such as "Accuracy may be limited based on location estimation," to inform the user that the current recognition confidence level is low. This approach ensures that the interactive function is not completely interrupted even without visual input.

[0114] As another alternative implementation, when external sensors fail and map data is incomplete (e.g., the vehicle is driving in an area without map coverage, map data loading fails, or signal is lost), the system cannot obtain any precise 3D spatial location information. In this case, the system outputs directional summary information based on the pointing ray. For example, if the user points to the right front of the vehicle, the system can prompt "The direction to the right front is the XX area, mainly commercial / residential / natural landscape," or output "Your pointing direction is southeast, and there is currently no detailed information for this direction." Although this type of information lacks specific target names and details, it still provides the user with a directional overview of the environment, reducing the negative experience caused by the system being completely unresponsive. Simultaneously, the system can suggest that the user perform the pointing operation again after the signal is restored or the vehicle enters an area with map coverage.

[0115] It should be noted that the three degradation strategies described above can automatically switch based on sensor status and map availability during system operation, forming a smooth degradation chain from high-precision mode to low-precision mode and then to summary mode. The system can monitor the confidence output of external sensors and the loading status of map data in real time. Once an information source is detected to be unavailable, it automatically switches to the corresponding degradation processing flow. Those skilled in the art can flexibly set the degradation judgment threshold and switching timing according to the actual system configuration and performance requirements. These alternative implementations all fall within the protection scope of this application.

[0116] It should be understood that the degradation processing methods described in the above embodiments are not mutually exclusive, and the system can simultaneously enable some measures from multiple degradation strategies according to the actual situation. For example, when some external sensors are available (e.g., only certain categories can be detected) and the map data is complete, the relevant steps in Method 1 and Method 2 can be used in combination. This application does not limit this.

[0117] Reference Figure 7 , Figure 7This is an optional scenario illustration provided by an embodiment of this application. A fan-shaped search area with a center line and an angle of ±15° is set in front of the vehicle. This area contains three candidate targets: building A (shopping mall), building B (office building), and scenic spot C (ancient pagoda). The system obtains the occupant's pointing action through the in-cabin gesture recognition module, generates a pointing ray, and maps it to the world coordinate system. Subsequently, the target confirmation module performs intersection determination between the pointing ray and the 3D bounding box output by the external visual recognition module and the 3D bounding box output by the map POI matching module. After dual matching, building A appears in both the visual candidate list and the map candidate list and is confirmed as the target actually pointed to by the occupant. After confirmation, the information recommendation presentation module displays a recommendation interface on the HUD or central control screen, including: Target name: XX Shopping Mall; Distance: Approximately 200m; Description: This shopping mall was built in 2018…; Rating: ★★★★★; Accuracy level: High (three-way fusion). The three-way fusion refers to the simultaneous integration of in-cabin hand gestures, external visual perception, and map POI data, indicating a high level of confidence in the recognition results. If the candidate target comes from only a single path, the accuracy level will be reduced accordingly (e.g., "medium" or "low"). The angle of the fan-shaped search area can be dynamically adjusted according to the confidence level of the pointing recognition to balance recall and precision.

[0118] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the in-vehicle pointing interaction method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0119] This application also provides an in-vehicle pointing interaction device, please refer to... Figure 8 The in-vehicle pointing interaction device includes: The first module 10 is used to acquire cabin perception data of the pointing actions of the occupants inside the vehicle. The second module 20 is used to generate the occupant's pointing ray based on the cabin perception data and map the pointing ray to the world coordinate system; The third module 30 is used to obtain the three-dimensional spatial location information of the candidate target, wherein the candidate target is an external target; the step of obtaining the three-dimensional spatial location information of the candidate target includes: collecting visual perception data through external sensors and calculating the three-dimensional bounding box of the candidate target based on the visual perception data; obtaining the three-dimensional bounding box of the interest point corresponding to the candidate target from the map data; and using the three-dimensional bounding box and the three-dimensional bounding box as the three-dimensional spatial location information of the candidate target. The fourth module 40 is used to determine the target that the occupant is pointing to from the candidate targets by matching the pointing ray in the world coordinate system with the three-dimensional spatial position information of the candidate targets, and output the description information of the target to the occupant.

[0120] The in-vehicle pointing interaction device provided in this application, employing the in-vehicle pointing interaction method in the above embodiments, can solve the technical problem in related technologies that cannot achieve interaction with external targets based on pointing behavior. Compared with related technologies, the beneficial effects of the in-vehicle pointing interaction device provided in this application are the same as those of the in-vehicle pointing interaction method provided in the above embodiments, and other technical features in the in-vehicle pointing interaction device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0121] This application provides a vehicle, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to perform the in-vehicle pointing interaction method described in Embodiment 1 above.

[0122] The following is for reference. Figure 9 This illustration shows an optional structural diagram of a vehicle suitable for implementing embodiments of this application. The vehicle in embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 9 The vehicle shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of this application.

[0123] like Figure 9As shown, the vehicle may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for vehicle operation. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the vehicle to communicate wirelessly or wiredly with other devices to exchange data. Although the diagram shows vehicles with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems may be implemented alternatively.

[0124] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0125] The vehicle provided in this application, employing the in-vehicle pointing interaction method described in the above embodiments, can solve the technical problem in related technologies where it is impossible to achieve interaction with external targets based on pointing behavior. Compared with related technologies, the beneficial effects of the vehicle provided in this application are the same as those of the in-vehicle pointing interaction method provided in the above embodiments, and other technical features of the vehicle are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0126] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0127] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0128] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the vehicle-mounted pointing interaction method in the above embodiments.

[0129] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0130] The aforementioned computer-readable storage medium may be included in the vehicle or may exist independently and not installed in the vehicle.

[0131] The aforementioned computer-readable storage medium carries one or more programs that, when executed by a vehicle, cause the vehicle to: acquire cabin perception data of an occupant's pointing action; generate a pointing ray of the occupant based on the cabin perception data and map the pointing ray to a world coordinate system; acquire three-dimensional spatial location information of candidate targets, wherein the candidate targets are external targets; the step of acquiring the three-dimensional spatial location information of candidate targets includes: collecting visual perception data through external sensors and calculating a three-dimensional bounding box of the candidate target based on the visual perception data; acquiring a three-dimensional bounding box of the point of interest corresponding to the candidate target from map data; using the three-dimensional bounding box and the three-dimensional bounding box as the three-dimensional spatial location information of the candidate target; determining the target pointed to by the occupant from the candidate targets by matching the pointing ray in the world coordinate system with the three-dimensional spatial location information of the candidate target, and outputting the description information of the target to the occupant.

[0132] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0133] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0134] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0135] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described in-vehicle pointing interaction method, which can solve the technical problem in related technologies that cannot achieve interaction with external targets based on pointing behavior. Compared with related technologies, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the in-vehicle pointing interaction method provided in the above embodiments, and will not be repeated here.

[0136] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the in-vehicle pointing interaction method described above.

[0137] The computer program product provided in this application can solve the technical problem in related technologies that it is impossible to achieve external target interaction based on pointing behavior. Compared with related technologies, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the in-vehicle pointing interaction method provided in the above embodiments, and will not be repeated here.

[0138] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A vehicle-mounted pointing interaction method, characterized in that, The in-vehicle pointing interaction method includes: Acquire cabin perception data of occupants' pointing gestures; The occupant's pointing ray is generated based on the cabin perception data, and the pointing ray is mapped to the world coordinate system; The process of obtaining the three-dimensional spatial location information of a candidate target, wherein the candidate target is an external target; the steps of obtaining the three-dimensional spatial location information of the candidate target include: collecting visual perception data through external sensors, and calculating the three-dimensional bounding box of the candidate target based on the visual perception data; obtaining the three-dimensional bounding box of the point of interest corresponding to the candidate target from map data; and using the three-dimensional bounding box and the three-dimensional bounding box as the three-dimensional spatial location information of the candidate target. By matching the pointing ray in the world coordinate system with the three-dimensional spatial position information of the candidate targets, the target pointed to by the occupant is determined from the candidate targets, and the description information of the target is output to the occupant. The method further includes: Based on the vehicle's current speed and heading angle, the system predicts potential targets that will soon enter the vehicle's forward field of view using the vehicle's kinematic model, and preloads the map data corresponding to the potential targets into the local cache; wherein, the preloading range is dynamically adjusted according to the vehicle speed, current computing load, and remaining memory capacity. When the occupant performs a pointing action, the target corresponding to the pointing ray is matched first in the local cache.

2. The in-vehicle pointing interaction method as described in claim 1, characterized in that, The steps for acquiring cabin perception data of occupant pointing actions include: Images of the occupants' hands are captured by an in-cabin camera, and multiple key points of the hand bones, including at least the shoulder, elbow, wrist, and fingertip, are extracted from them. The step of generating the occupant's pointing ray based on the cabin sensing data includes: Using the line connecting the wrist to the tip of the index finger as the initial pointing direction, and combining the extension direction from the shoulder to the elbow, the initial pointing direction is weighted and corrected to obtain the pointing ray of the occupant in the cockpit coordinate system.

3. The in-vehicle pointing interaction method as described in claim 2, characterized in that, The step of mapping the pointing ray to the world coordinate system includes: Based on the occupant's seating position and the calibration parameters of the in-cabin camera, the pointing ray in the cockpit coordinate system is converted into a pointing ray in the vehicle coordinate system; Based on the vehicle's current pose, current heading angle, and the installation pose of the external sensors, the pointing rays in the vehicle coordinate system are converted into pointing rays in the world coordinate system.

4. The in-vehicle pointing interaction method as described in claim 1, characterized in that, The step of matching the pointing ray in the world coordinate system with the three-dimensional spatial position information of the candidate target includes: The first intersection determination is performed between the pointing ray in the world coordinate system and the three-dimensional bounding box to obtain a list of visual candidate targets. A second intersection determination is performed between the pointing ray in the world coordinate system and the three-dimensional bounding box to obtain a list of candidate targets on the map. Candidate targets that appear simultaneously in both the visual candidate target list and the map candidate target list will be used as the targets pointed to by the occupants. For candidate targets that only appear in the visual candidate target list or the map candidate target list, the target pointed to by the occupant is obtained by filtering.

5. The in-vehicle pointing interaction method as described in claim 4, characterized in that, The step of filtering to obtain the target pointed to by the occupant includes: The candidate targets are sorted from closest to furthest from the occupant, and the candidate target closest to the occupant is taken as the target the occupant is pointing to. Alternatively, a salience score can be calculated based on the attribute characteristics of the candidate target, and the salience score can be weighted and fused with the distance ranking weight of the candidate target. The candidate target with the highest weighted fusion score can be used as the target that the occupant is pointing to. The attribute characteristics include the physical size of the candidate target, the popularity score, and the angular deviation between the candidate target and the occupant's line of sight. The popularity score is determined based on the historical access frequency of map points of interest or user evaluation scores. Alternatively, when the difference in significance scores between the top two candidate targets in the weighted fusion score ranking is less than a preset threshold, an interactive confirmation mechanism with the occupant is triggered, and the candidate target selected by the occupant is taken as the target pointed to by the occupant. Alternatively, the occupant's gesture features, vehicle pose, and candidate target set can be input into a deep learning model, and the candidate targets output by the deep learning model can be used as the targets pointed to by the occupant.

6. The in-vehicle pointing interaction method as described in claim 1, characterized in that, The method further includes: The system guides the occupants to point in multiple known calibration directions. Based on the deviation between the occupants' actual pointing direction and the calibration direction, the system calculates the occupants' pointing offset vector. The pointing offset vector is used to correct the occupants' pointing ray and is adaptively updated based on the occupants' selection of candidate targets.

7. The in-vehicle pointing interaction method as described in claim 1, characterized in that, The method further includes: When the external sensors cannot identify the category of the candidate target, the three-dimensional spatial position information of the candidate target is determined based on the outline position of the candidate target; When the external sensors fail, a fan-shaped search area is determined based on the direction of the pointing ray, and the three-dimensional spatial location information of the candidate target is determined from the map data within the fan-shaped search area. When both external sensors and map data are unavailable, directional summary information is output based on the directional ray.

8. A vehicle, characterized in that, The vehicle includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the in-vehicle pointing interaction method as described in any one of claims 1 to 7.

Images