Robust spatial line of sight mapping and embodied interaction method, apparatus and system

Abstract

The present application relates to the technical field of intelligent cockpit human-computer interaction, and particularly relates to a robust space line-of-sight mapping and embodied interaction method, device and system. The method comprises: acquiring occupant data, including collecting the visual field image and the visual point coordinates of the driver by the head-mounted eye tracker; processing the occupant data, including: acquiring the screen area information in the physical environment, obtaining the screen area mask image and the origin coordinates of the screen area by using the improved YOLOv8 instance segmentation algorithm; using the edge contour detection suzuki algorithm to extract the edge contour of the mask image, and inputting the edge contour into the corner point detection model to obtain a corner point coordinate list; determining the screen origin according to the corner point coordinates, and performing projection transformation on the quadrilateral formed by the corner points; interactive action recognition and response, including: processing the visual point coordinate sequence to obtain the interactive action category and interacting with the user interface in the screen. The technical scheme is suitable for multiple types of screens, and can realize the mapping of the space line-of-sight on the screen and the interaction with the vehicle-mounted screen.

Description

Technical Field

[0001] This invention relates to the field of human-vehicle interaction technology, specifically to a robust spatial line-of-sight mapping and embodied interaction method, device, and system. Background Technology

[0002] In the field of vehicle human-machine interaction, traditional interaction methods such as touch screens and voice-controlled screens have certain limitations. Touch screens require passengers to take their hands off the steering wheel to operate, which not only distracts the driver but may also pose safety hazards in certain driving situations (such as at high speeds). While voice control frees up the hands to some extent, its performance is easily affected by noise inside and outside the vehicle, may conflict with the voices of other passengers, and its operation is relatively limited, only applicable to some preset operations.

[0003] With the development of technology, methods for interacting with in-vehicle screens using viewpoints have gradually emerged. However, these methods suffer from poor robustness when dealing with curved screens, rotating screens, and incomplete screens.

[0004] Specifically, the existing technology has the following problems:

[0005] Low accuracy: Voice interaction is easily affected by environmental noise, resulting in low recognition rates, and may be interfered with by other people's voices, leading to permission issues. Patent CN109493305A, a method and system for superimposing human eye gaze with a foreground image, obtains the occupant's gaze direction by directly capturing the face with a telemetry camera. However, it also has shortcomings in the accuracy of gaze positioning. Because the face occupies a small area in the camera-captured image, the resolution is low, and the eye area contains little information, resulting in inaccurate estimation of the gaze direction.

[0006] Poor real-time performance: Patents CN113561988A (a voice control method, vehicle, and readable storage medium based on eye tracking) and CN115547332A (a wake-up-free intent recall method, system, and vehicle based on eye attention) require step-by-step processing during interaction, such as first estimating the gaze direction and then detecting the voice for control. These two steps are not parallel, resulting in poor real-time performance. Furthermore, the method of using SLAM algorithms to build spatial models involves a significant computational burden, which also affects real-time performance.

[0007] Low degree of freedom: The patent with publication number CN110481419A, which describes a human-vehicle interaction method, system, vehicle and storage medium, and the patent with publication number CN113064481A, which describes an in-vehicle screen control method, device, computer equipment and storage medium, can only perform simple control operations, have low degree of control freedom, and cannot meet diverse interaction needs.

[0008] Low robustness: The corner detection model in the gesture control method, device, vehicle and storage medium of the vehicle-mounted sliding screen in the patent with publication number CN116820232A is easily affected by the screen content and the black border of the screen. It is also affected by the stability between cameras during the coordinate transformation process, resulting in low robustness of the method.

[0009] Highly invasive: Some methods add feature labels or display feature images around the display to facilitate the division of the display area, which affects the original display or user interface.

[0010] Lack of interactivity: The method and system for superimposing human eye gaze with foreground image in the patent with publication number CN109493305A only overlaps the gaze information with the display screen and cannot achieve real interactive actions.

[0011] Poor applicability: The screen area detection method and screen area detection system of the patent with publication number US2011274353A1 rely on the complete straight line of the screen edge to detect the screen position. It is difficult to implement on curved screens or partially obscured screens and cannot be well applied to various screen types. At the same time, it does not solve the problem of missing corner points. Summary of the Invention

[0012] The purpose of this invention is to propose a robust spatial gaze mapping and embodied interaction method, device and system that can effectively utilize gaze to capture multiple screens, and then locate the cursor on the screen by combining viewpoint and field of view images, thereby directly interacting with the user interface in the vehicle screen.

[0013] To achieve the above objectives, in a first aspect, the present invention provides a robust spatial line-of-sight mapping and embodied interaction method, comprising:

[0014] Acquire occupant data, including images of the driver's field of vision and viewpoint coordinates collected by a head-mounted eye tracker;

[0015] Processing occupant data, including:

[0016] To obtain screen region information, an improved YOLOv8 instance segmentation algorithm is used to obtain the screen region mask image and the origin coordinates of the screen region.

[0017] The edge contours of the mask image are extracted using the Suzuki edge contour detection algorithm and then fed into the corner detection model to obtain a list of corner coordinates.

[0018] The screen origin is determined based on the corner coordinates, and the quadrilateral formed by the corners is transformed by projection to obtain a rectangle and a projection matrix.

[0019] Interactive action recognition and response, including:

[0020] Process the viewpoint coordinate sequence to obtain the interactive action category; respond according to the interactive action category and the content displayed on the in-vehicle screen;

[0021] The beneficial effects of the basic scheme:

[0022] The improved YOLOv8 model in this technical solution offers advantages in real-time performance and accuracy compared to traditional object detection algorithms, enabling more accurate identification of screen boundaries (especially in partially occluded scenes in complex environments). The Suzuki algorithm, combined with corner detection, extracts edge contours and locates corners, then uses projection transformation to map any quadrilateral screen into a standard rectangle. This solves the problem of interaction coordinate offset caused by installation angles or curved surface designs of in-vehicle screens, thus improving interaction accuracy.

[0023] For various display screens, this invention uses image segmentation methods and additionally estimates the screen origin, achieving the same effect as a normal screen on rotating and curved screens. This invention uses a head-mounted eye tracker to obtain the gaze direction; the closer the eye tracker is to the eyes, the larger the eye area in the obtained image, resulting in higher accuracy. This invention develops a corner detection algorithm that effectively addresses the problem of missing corners. This invention adjusts the interaction for different occupants, enhancing the user experience.

[0024] This technical solution reduces reliance on physical buttons or touchscreens, aligning with the trend of "seamless interaction" and reducing the risk of driver distraction. With this solution, drivers can quickly access vehicle status information through their gaze. Furthermore, it can provide alternative interaction methods for drivers with physical disabilities or optimize the complexity of interaction for elderly users.

[0025] As a feasible preferred solution, the improved YOLOv8 instance segmentation model adds a point output header to the original YOLOv8 algorithm. The point output header contains two CBS modules and a convolutional layer, and the output is batchx*reg_max*2*w*h.

[0026] As a feasible preferred solution, the corner detection model includes three CBRSA modules, one maxpooling module, and two fully connected layers. The CBRSA module consists of a CONV2D layer, a BN layer, and a ReLU activation function connected in sequence, and then divided into two branches. Branch 1 includes a 1*C*1*1 convolutional layer and avgpooling in parallel, followed by a 1*2*3*3 convolutional layer. Then, branches 1 and 2 are multiplied to obtain the output.

[0027] As a feasible and preferred solution, the screen origin is determined by calculating the distance. The distance between the screen origin coordinates obtained from the point output head of the segmentation model and the coordinates of the four corner points is calculated, and the corner point with the smallest distance is determined as the screen origin.

[0028] As a feasible and preferred solution, a rectangle is obtained by projecting the quadrilateral formed by the four corner points, and the projection matrix is ​​calculated. The formula for the projection transformation is:

[0029]

[0030] in, These are the original coordinates. These are the transformed coordinates. It is a projection matrix.

[0031] As a feasible and preferred approach, the viewpoint coordinate sequence is processed to obtain the interaction action category, and then a threshold-based method or machine learning algorithm is used to classify and determine the interaction action category.

[0032] As a feasible and preferred option, it also includes adjusting the interaction sensitivity, including:

[0033] Process the interaction log file, extract the data from the first 64 frames of the interaction action as input samples to obtain the operation sensitivity; adjust the sliding distance and sliding speed according to the operation sensitivity.

[0034] As a feasible and preferred approach, the sensitivity calculation formula is as follows:

[0035]

[0036] in, The timestamp of the starting frame of the interactive action. For the timestamp of the interactive action, Maximum allowable response time;

[0037] Adjust sensitivity based on user swipe speed distribution: if the average user swipe speed is higher than the group average, then reduce the sensitivity. To reduce sensitivity; conversely, to increase sensitivity. To improve sensitivity.

[0038] Secondly, embodiments of this disclosure also provide a robust spatial line-of-sight mapping and embodied interaction system, which utilizes the aforementioned robust spatial line-of-sight mapping and embodied interaction method.

[0039] Thirdly, this disclosure also provides a robust spatial line-of-sight mapping and embodied interaction device, which includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the program, it implements the aforementioned robust spatial line-of-sight mapping and embodied interaction method. Attached Figure Description

[0040] Figure 1The diagram shows a schematic of the device deployment for this method.

[0041] Figure 2 The following is a simplified flowchart of the method.

[0042] Figure 3 The following is a detailed flowchart of the method.

[0043] Figure 4 The image shown is the point output header of the improved model of this method.

[0044] Figure 5 The diagram shows the corner detection model of this method.

[0045] Figure 6 The diagram shown is a perspective transformation diagram of this method.

[0046] Figure 7 The diagram shown is a schematic representation of the architecture of the electronic device in the embodiment.

[0047] Reference numerals: Electronic device 500, processor 501, communication interface 502, memory 503, bus 504. Detailed Implementation

[0048] To make the technical solution and advantages of this application clearer, the technical solution of the present invention will be further described in detail below with reference to the accompanying drawings. It is understood that the specific embodiments described herein are only some embodiments of the present invention, and are only used to explain this application, not to limit it. It should be noted that the technical features or combinations of technical features described in the following embodiments should not be considered isolated; they can be combined with each other to achieve better technical effects. The same reference numerals appearing in the accompanying drawings of the following embodiments represent the same features or components, and can be applied to different embodiments.

[0049] Furthermore, unless otherwise defined, the technical or scientific terms used in this invention description shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains.

[0050] The present invention will now be described in further detail with reference to the accompanying drawings:

[0051] Example 1

[0052] Reference Figure 1 The hardware underlying this technical solution includes a head-mounted eye tracker.

[0053] Head-mounted eye trackers can capture the driver's gaze direction. They feature high-precision cameras and sensors to obtain clear visual images and accurate viewpoint coordinates. During installation, calibration is required to match the eye tracker's coordinate system with the driver's actual gaze direction.

[0054] Reference Figure 2 and Figure 3 A robust method for spatial line-of-sight mapping and embodied interaction includes:

[0055] Step S100: Acquire occupant data, including using a head-mounted eye tracker to capture the driver's visual field image and viewpoint coordinates at a set frame rate (e.g., 60fps). The visual field image resolution is 640 * 640, and the viewpoint coordinates include the driver's viewpoint position coordinates (x, y) within the visual field image. During the acquisition process, the eye tracker transmits data to the onboard computer in real time.

[0056] Data preprocessing includes image denoising and image normalization.

[0057] Step S200, processing occupant data, including:

[0058] Step S201, obtain screen area information, including:

[0059] An improved YOLOv8 instance segmentation algorithm is used to obtain the screen region mask image and the origin coordinates of the screen region, including:

[0060] The preprocessed view image is input into the improved YOLOv8 instance segmentation model. The improved YOLOv8 instance segmentation model adds a point output header to the original YOLOv8 algorithm; this point output header is responsible for predicting the origin coordinates of the screen region. (Refer to...) Figure 4 The point output header contains two CBS modules and one convolutional layer, and the output is batch*reg_ma*2*w*h.

[0061] The model analyzes and processes the input image, outputting a screen region mask image and the origin coordinates of the screen region. The mask image is a binary image, where the screen region is white (pixel value 255) and the remaining regions are black (pixel value 0). Determining the origin is to handle scenarios where the origin is not fixed at the top left corner when the screen is rotated.

[0062] The edge contours of the mask image are extracted using the Suzuki edge contour detection algorithm, including:

[0063] The Suzuki algorithm is used to extract the edge contours of the acquired screen area mask image. The Suzuki algorithm can effectively extract the edge contours of objects, providing a foundation for subsequent corner detection.

[0064] The extracted edge contour image is used as input and fed into the corner detection model. The input to the corner detection model is a black-and-white contour image and a position weight map; the output is a list of corner coordinates. (See reference...) Figure 5 The corner detection model consists of three CBRSA modules, one maxpooling module, and two fully connected layers. The CBRSA module is a CONV2D layer, a BN layer, and a ReLU activation function connected sequentially. It is then divided into two branches. Branch 1 includes a 1*C*1*1 convolutional layer and avgpooling in parallel, followed by a 1*2*3*3 convolutional layer. The output is obtained by multiplying branches 1 and 2.

[0065] This method is helpful for corner detection on curved screens because curved screens cannot obtain four straight lines, making it impossible to use the intersection of lines to obtain corner points. Furthermore, by using data from missing corner points for training, it can also detect occluded corner points and predict their locations.

[0066] Step S202, determine the screen origin, including:

[0067] The screen origin is determined by calculating the distance. The coordinates of the screen origin obtained from the point output header of the segmentation model are used to calculate the distance with the coordinates of the four corner points. The corner point with the smallest distance is determined as the screen origin.

[0068] Step S203, refer to Figure 6 The quadrilateral formed by the four corner points is projected to obtain a rectangle, and the projection matrix is ​​calculated. The formula for the projection transformation is:

[0069]

[0070] in, These are the original coordinates. These are the transformed coordinates. It is the projection matrix. The projection matrix is ​​used to map the view coordinates onto this quadrilateral coordinate system, which is the screen area.

[0071] Step S300, interactive action recognition and response, includes:

[0072] Step S301: Process the viewpoint coordinate sequence to obtain the interactive action category, including:

[0073] The acquired viewpoint coordinate sequence is analyzed and processed to determine the type of the driver's interaction action. For example, a rapid movement of the viewpoint coordinates within a short period may indicate a swipe operation; a viewpoint coordinate remaining stationary in a certain area for an extended period may indicate a selection operation. Specific classification methods can employ threshold-based approaches or machine learning algorithms.

[0074] Save the coordinates and interaction action categories as an interaction log file. Each line in the log file includes the gaze coordinates and interaction action category for one frame, represented as "x, y, cls", where x and y are the gaze coordinates and cls is the interaction category number. If no interaction is performed, the interaction category number is 0.

[0075] Step S302, responding according to the type of interactive action and the content displayed on the vehicle screen, including:

[0076] If the interaction action is categorized as up, down, left, or right swipe, observe the screen and combine the swipe speed and distance to perform the swipe operation. For example, when a right swipe is detected, the content on the screen scrolls to the right accordingly, based on the swipe speed and distance.

[0077] If the interaction action category is selected, the action of clicking the button in the area where the viewpoint is located on the viewing screen will be executed. The system will determine the click location based on the viewpoint coordinates and trigger the corresponding function.

[0078] If the interaction action category is determined to be unselected, no action will be performed.

[0079] Step S400, adjust the interaction sensitivity, including:

[0080] Step S401, process the interaction log file to obtain the operation sensitivity, including:

[0081] The recorded interaction log files are processed to analyze the driver's interaction habits. The first 64 frames of data from the interaction actions are extracted as input samples, with a shape of 64 * 2 (2 representing x and y). Labels are one-hot encoded, where values ​​within a category represent sensitivity, and the rest are 0.

[0082] The sensitivity of the interaction is obtained by training a neural network model. The sensitivity calculation formula is as follows:

[0083]

[0084] in, The timestamp of the starting frame of the interactive action. For the timestamp of the interactive action, This is the maximum allowed response time (default value is 64 frames).

[0085] A neural network model is trained using labeled log files to obtain the sensitivity of interactions. The trained neural network model is then used to process labeled and unlabeled log file data to obtain sensitivity values ​​for all categories. The sensitivity is then normalized to a range of 0 to 1. The closer the sensitivity value is to 1, the more sensitive the representation.

[0086] Step S402: Adjust the sliding distance and sliding speed based on sensitivity. Adjust the sliding distance and sliding speed according to the calculated sensitivity. For example, when the sensitivity is 0.9, it indicates that the user is accustomed to sensitive operation, and the sliding distance and speed can be appropriately increased; when the sensitivity is 0.1, the sliding distance and speed can be appropriately decreased. Adjust the sensitivity according to the user's sliding speed distribution: if the average user sliding speed is higher than the group average, then decrease the sensitivity. To reduce sensitivity; conversely, to increase sensitivity. To improve sensitivity.

[0087] By analyzing historical operation data (such as swipe distance and click frequency), sensitivity parameters are dynamically adjusted to adapt to different user habits (such as elderly people needing a larger swipe area) or scenario requirements (such as reducing sensitivity to reduce misoperation when driving at high speed).

[0088] This disclosure provides a robust spatial line-of-sight mapping and embodied interaction system, which utilizes the aforementioned robust spatial line-of-sight mapping and embodied interaction method.

[0089] Those skilled in the art will understand that implementing all or part of the processes in a robust spatial line-of-sight mapping and embodied interaction method can be accomplished by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium. When executed, the program can include the processes of various embodiments of the robust spatial line-of-sight mapping and embodied interaction method. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0090] This application also provides a robust spatial line-of-sight mapping and embodied interaction device (electronic device 500), which utilizes the aforementioned robust spatial line-of-sight mapping and embodied interaction method. The device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the aforementioned robust spatial line-of-sight mapping and embodied interaction method. In this application embodiment, the processor is the control center of the computer method and can be a physical machine processor or a virtual machine processor.

[0091] Reference Figure 7 The electronic device 500 includes at least one processor 501, at least one communication interface 502, at least one memory 503, and at least one bus 504. The bus 504 is used for communication between these components, the communication interface 502 is used for signaling or data communication with other node devices, and the memory 503 stores machine-readable instructions executable by the processor 501. When the electronic device 500 is running, the processor 501 communicates with the memory 503 via the bus 504. When the machine-readable instructions are invoked by the processor 501, they execute the steps of the robust spatial line-of-sight mapping and embodied interaction method described above.

[0092] The above content is merely an embodiment of the present invention. Commonly known structures and characteristics of the solutions are not described in detail here. Those skilled in the art are aware of all common technical knowledge in the field prior to the application date or priority date, are aware of all existing technologies in that field, and have the ability to apply conventional experimental methods prior to that date. Those skilled in the art can improve and implement this solution based on the guidance provided in this application and their own capabilities. Some typical known structures or methods should not be obstacles for those skilled in the art to implement this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the structure of the present invention. These should also be considered within the scope of protection of the present invention, and will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.

Claims

1. A robust method for spatial line-of-sight mapping and embodied interaction, characterized in that, include: Acquire occupant data, including images of the driver's field of vision and viewpoint coordinates collected by a head-mounted eye tracker; Processing occupant data, including: The screen region information is obtained, and the preprocessed field-of-view image is input into the improved YOLOv8 instance segmentation model. The improved YOLOv8 instance segmentation algorithm is used to obtain the screen region mask image and the origin coordinates of the screen region. The improved YOLOv8 instance segmentation model adds a point output head to the original YOLOv8 algorithm. The point output head contains two CBS modules and one convolutional layer. The screen origin coordinates are predicted through the point output head. The edge contours of the mask image are extracted using the Suzuki edge contour detection algorithm and then fed into the corner detection model to obtain a list of corner coordinates. The screen origin is determined based on the corner coordinates, and the quadrilateral formed by the corners is transformed by projection to obtain a rectangle and a projection matrix. Determining the screen origin based on corner coordinates includes: The screen origin is determined by calculating the distance. The coordinates of the screen origin obtained from the point output head of the segmentation model are used to calculate the distance with the coordinates of the four corner points. The corner point with the smallest distance is determined as the screen origin. Interactive action recognition and response, including: The viewpoint coordinate sequence is processed to obtain the interactive action category; a response is made based on the interactive action category and the content displayed on the in-vehicle screen.

2. The robust spatial gaze mapping and embodied interaction method according to claim 1, characterized in that, The corner detection model comprises three CBRSA modules, one maxpooling module, and two fully connected layers. Each CBRSA module consists of a CONV2D layer, a BN layer, and a ReLU activation function connected sequentially, then divided into two branches. Branch 1 includes 1... C 1 One convolutional layer and avgpooling are run in parallel, followed by one... 2 3 The system consists of three convolutional layers, and then branches 1 and 2 are multiplied to obtain the output. Branch 2 is an identity mapping of the output features of the ReLU activation function.

3. The robust spatial gaze mapping and embodied interaction method according to claim 1, characterized in that, A rectangle is obtained by projecting a quadrilateral formed by its four corner points, and the projection matrix is ​​calculated. The formula for the projection transformation is: in, These are the original coordinates. These are the transformed coordinates. It is a projection matrix.

4. A robust spatial gaze mapping and embodied interaction method according to claim 1, characterized in that, The viewpoint coordinate sequence is processed to obtain the interactive action category, and the category is determined by using a threshold-based method or machine learning algorithm.

5. A robust spatial gaze mapping and embodied interaction method according to claim 1, characterized in that, This also includes adjusting interaction sensitivity, including: Process the interaction log file, extract the data from the first 64 frames of the interaction action as input samples to obtain the operation sensitivity; adjust the sliding distance and sliding speed according to the operation sensitivity.

6. A robust spatial gaze mapping and embodied interaction method according to claim 5, characterized in that, The formula for calculating sensitivity is as follows: in, The timestamp of the starting frame of the interactive action. For the timestamp of the interactive action, Maximum allowable response time; Adjust sensitivity based on user swipe speed distribution: if the average user swipe speed is higher than the group average, then reduce the sensitivity. To reduce sensitivity; conversely, to increase sensitivity. To improve sensitivity.

7. A robust spatial gaze mapping and embodied interaction system, characterized in that, A robust spatial line-of-sight mapping and embodied interaction method as described in any one of claims 1-6 is employed.

8. A robust spatial line-of-sight mapping and embodied interaction device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the program, it implements a robust spatial line-of-sight mapping and embodied interaction method as described in any one of claims 1-6.

Images