Robust space sight mapping and body interaction method, device and system

Abstract

The invention relates to the technical field of intelligent cabin man-machine interaction, in particular to a robust space sight mapping and body interaction method, device and system. The method comprises the following steps: acquiring passenger data, wherein the passenger data comprises a view image and a viewpoint coordinate of a driver collected by a head-mounted eye tracker; processing the passenger data, namely acquiring screen area information in a physical environment, and acquiring a screen area mask image and an origin coordinate of a screen area by adopting an improved YOLOv8 instance segmentation algorithm; using an edge contour detection suzuki algorithm to extract the edge contour of the mask image, and sending the edge contour to a corner detection model to obtain a corner coordinate list; determining a screen origin according to the angular point coordinates, and performing projection transformation on a quadrangle formed by the angular points; and interactive action recognition and response: processing the viewpoint coordinate sequence to obtain an interactive action category, and interacting with a user interface in a screen. The technical scheme is suitable for multiple types of screens, mapping of the space sight on the screen can be achieved, and interaction with a vehicle-mounted screen is achieved.

Description

Technical Field

[0001] The present invention relates to the field of human-vehicle interaction technology, and in particular to a robust spatial line of sight mapping and embodied interaction method, device and system. Background Art

[0002] In the field of vehicle human-machine interaction, traditional interaction methods such as touch screens and voice-activated screens have certain limitations. Touch screens require passengers to remove their hands from the steering wheel to operate, which not only distracts the driver but also poses a safety hazard in certain driving situations (such as at high speeds). While voice control frees up hands to a certain extent, its performance is susceptible to interference from noise inside and outside the vehicle, can conflict with other passengers' voices, and is relatively limited in its operation content, applicable only to a few preset operations.

[0003] With the development of technology, methods for interacting with in-vehicle screens using viewpoints have gradually emerged. However, these methods have poor robustness when dealing with curved screens, rotating screens, and incomplete screens.

[0004] Specifically, the prior art has the following problems: Low accuracy: Voice interaction is easily affected by ambient noise, resulting in low recognition rates. It can also be disrupted by other people's voices, leading to permission issues. Patent publication number CN109493305A describes a method and system for superimposing human gaze with a foreground image. While this method uses a telemetry camera to directly capture the face to determine the occupant's gaze direction, it also suffers from inaccurate gaze location. Because the face occupies a relatively small area in the camera image, resulting in low resolution and limited information in the eye area, the estimation of gaze direction is inaccurate.

[0005] Poor real-time performance: Patents CN113561988A, "A Voice Control Method Based on Eye Tracking," "A Vehicle, and a Readable Storage Medium," and CN115547332A, "A Wake-Up-Free Intention Recall Method Based on Eye Attention," require a step-by-step interaction process. For example, first estimating eye direction, then detecting voice for control, these two steps are not performed in parallel, resulting in poor real-time performance. Furthermore, the use of SLAM algorithms to establish spatial models involves a significant computational burden, which also affects real-time performance.

[0006] Low degree of freedom: The patented method, system, vehicle and storage medium for human-vehicle interaction with publication number CN110481419A, and the patented method, device, computer equipment and storage medium for vehicle-mounted screen control with publication number CN113064481A, can only perform simple control operations, have low control freedom, and cannot meet diverse interaction needs.

[0007] Low robustness: The corner detection model in the patented vehicle-mounted sliding screen gesture control method, device, vehicle and storage medium with publication number CN116820232A is easily affected by the screen content and the black border of the screen. The coordinate transformation process is also affected by the stability between cameras, resulting in low robustness of the method.

[0008] Highly intrusive: To facilitate segmentation of the display range, some methods add feature labels or display feature images around the display, affecting the original display or user interface.

[0009] No interactivity: The method and system for superimposing the human eye's line of sight with a foreground image in the patent publication number CN109493305A merely overlaps the line of sight information with the display, and cannot achieve true interactive action.

[0010] Poor applicability: The patented screen area detection method and screen area detection system with publication number US2011274353A1 rely on complete screen edge straight lines to detect the screen position, which is difficult to implement on curved screens or partially obscured screens. It cannot be well applied to various screen types and does not solve the problem of missing corner points. Summary of the Invention

[0011] The purpose of the present invention is to propose a robust spatial line of sight mapping and embodied interaction method, device and system, which can effectively use line of sight to capture multiple screens, and then locate the cursor on the screen by combining the viewpoint and the field of view image, so as to directly interact with the user interface in the vehicle screen.

[0012] To achieve the above objectives, in a first aspect, the present invention provides a robust spatial gaze mapping and embodied interaction method, comprising: Obtaining passenger data, including using a head-mounted eye tracker to capture the driver's field of view image and viewpoint coordinates; Processing of occupant data, including: Obtain screen area information and use the improved YOLOv8 instance segmentation algorithm to obtain the screen area mask image and the origin coordinates of the screen area; Use the Suzuki edge contour detection algorithm to extract the edge contour of the mask image and feed it into the corner detection model to obtain a list of corner point coordinates; Determine the screen origin based on the corner point coordinates, and perform projection transformation on the quadrilateral formed by the corner points to obtain a rectangle and projection matrix; Interactive action recognition and response, including: Process the viewpoint coordinate sequence to obtain the interactive action category; respond based on the interactive action category and the content displayed on the vehicle screen; Beneficial effects of the basic program: The improved YOLOv8 model used in this technical solution offers advantages over traditional object detection algorithms in terms of real-time performance and accuracy, enabling more accurate identification of screen boundaries (especially in partially occluded scenes in complex environments). The Suzuki algorithm, combined with corner detection, uses edge contour extraction and corner location, combined with projection transformation to map any quadrilateral screen into a standard rectangle. This solves the problem of interaction coordinate offset caused by the installation angle or curved surface design of in-vehicle screens, improving interaction accuracy.

[0013] For various display screens, the present invention uses an image segmentation method and additionally estimates the screen origin to achieve the same effect on rotating and curved screens as on normal screens. The present invention uses a head-mounted eye tracker to obtain line of sight. Because the head-mounted eye tracker is closer to the eyes, the eye area in the obtained image is larger and more accurate. The present invention has developed a self-developed corner detection algorithm that can effectively address the problem of missing corners. The present invention makes interactive adjustments for different passengers to enhance the user experience.

[0014] This technology reduces reliance on physical buttons or touchscreens, aligning with the trend toward "unconscious interaction" and reducing the risk of driver distraction. This technology allows drivers to quickly access vehicle status information through their line of sight. It also provides alternative interaction methods for drivers with physical disabilities and optimizes interaction complexity for elderly users.

[0015] As an implementable preferred solution, the improved YOLOv8 instance segmentation model adds a point output head on the basis of the original YOLOv8 algorithm. The point output head includes two CBS modules and a convolutional layer, and the output is batchx*reg_max*2*w*h.

[0016] As an implementable preferred solution, the corner detection model includes 3 CBRSA modules, a maxpooling module and two fully connected layers. The CBRSA module is a CONV2D layer, a BN layer and a Relu activation function connected in sequence, and then divided into two branches. Branch 1 includes a 1*C*1*1 convolution layer and an avgpool layer in parallel, and then connected to a 1*2*3*3 convolution layer. Then branches 1 and 2 are multiplied to obtain the output.

[0017] As an implementable preferred solution, the distance is calculated to determine the screen origin. The screen origin coordinates obtained by the segmentation model point output head and the four corner point coordinates are used to calculate the distance, and the corner point with the smallest distance is determined as the screen origin.

[0018] As an implementable preferred solution, a projection transformation is performed on the quadrilateral formed by the four corner points to obtain a rectangle, and the projection matrix is ​​calculated. The formula for the projection transformation is:

[0019] in, are the original coordinates, are the transformed coordinates, is the projection matrix.

[0020] As an implementable preferred solution, the viewpoint coordinate sequence is processed to obtain the interactive action category, and a threshold-based method or a machine learning algorithm is used for classification to determine the interactive action category.

[0021] As an implementable preferred solution, adjusting the interaction sensitivity includes: The interaction log files are processed and the data of the first 64 frames of the interaction action are extracted as input samples to obtain the operation sensitivity. The sliding distance and sliding speed are adjusted according to the operation sensitivity.

[0022] As an implementable preferred solution, the sensitivity calculation formula is as follows:

[0023] in, The timestamp of the start frame of the interactive action. The timestamp of executing the interaction action. is the maximum allowed response time; Adjust sensitivity based on user sliding speed distribution: if the average sliding speed of users is higher than the group mean, reduce To reduce the sensitivity; otherwise, increase To improve sensitivity.

[0024] In a second aspect, the embodiments of the present disclosure further provide a robust spatial vision mapping and embodied interaction system, which utilizes the above-mentioned robust spatial vision mapping and embodied interaction method.

[0025] On the third aspect, an embodiment of the present disclosure also provides a robust spatial visual line mapping and embodied interaction device, which includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the program, the above-mentioned robust spatial visual line mapping and embodied interaction method is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Shown is a schematic diagram of the device deployment of this method.

[0027] Figure 2 Shown is a simplified flow chart of the method.

[0028] Figure 3 Shown is a detailed flow chart of the method.

[0029] Figure 4Shown is the point output header of the improved model of this method.

[0030] Figure 5 Shown is the corner detection model diagram of this method.

[0031] Figure 6 Shown is the perspective transformation diagram of this method.

[0032] Figure 7 FIG. 1 is a schematic diagram showing the structure of an electronic device according to an embodiment.

[0033] Reference numerals: electronic device 500 , processor 501 , communication interface 502 , memory 503 , bus 504 . DETAILED DESCRIPTION

[0034] In order to make the technical solution and advantages of the present application clearer, the technical solution of the present invention will be further described in detail below with reference to the accompanying drawings. It will be understood that the specific embodiments described herein are only partial embodiments of the present invention, which are only used to explain the present application, rather than to limit the present application. It should be noted that the technical features or combinations of technical features described in the following embodiments should not be considered to be isolated, and they can be combined with each other to achieve better technical effects. The same reference numerals appearing in the drawings of the following embodiments represent the same features or components, which can be applied to different embodiments.

[0035] In addition, unless otherwise defined, technical or scientific terms used in the description of the present invention should have the common meanings understood by those skilled in the art in the art to which the present invention belongs.

[0036] The present invention will be further described in detail below with reference to the accompanying drawings: Example 1 Reference Figure 1 ,The hardware that this technical solution relies on includes a head-mounted eye tracker.

[0037] A head-mounted eye tracker can capture the driver's gaze direction. It uses high-precision cameras and sensors to obtain clear field of view images and precise viewpoint coordinates. During installation, calibration is required to align the eye tracker's coordinate system with the driver's actual gaze direction.

[0038] Reference Figure 2 and Figure 3 , a robust spatial gaze mapping and embodied interaction method, including: Step S100: Acquiring occupant data involves using a head-mounted eye tracker to capture the driver's field of view image and viewpoint coordinates at a set frame rate (e.g., 60 fps). The field of view image has a resolution of 640 x 640 pixels, and the viewpoint coordinates include the driver's viewpoint position (x, y) within the field of view image. During the acquisition process, the eye tracker transmits the data in real time to the onboard computer.

[0039] Perform data preprocessing, including image denoising and image normalization.

[0040] Step S200, processing passenger data, includes: Step S201, obtaining screen area information, including: The improved YOLOv8 instance segmentation algorithm is used to obtain the screen area mask image and the origin coordinates of the screen area, including: The preprocessed field of view image is input into the improved YOLOv8 instance segmentation model. The improved YOLOv8 instance segmentation model adds a point output head based on the original YOLOv8 algorithm. The point output head is responsible for predicting the origin coordinates of the screen area. Figure 4 , the point output head contains two CBS modules and a convolutional layer, and the output is batch*reg_ma*2*w*h.

[0041] The model analyzes the input image and outputs a mask image of the screen area within the field of view, along with the coordinates of the screen's origin. The mask image is a binary image, with the screen area white (pixel values ​​of 255) and the rest of the area black (pixel values ​​of 0). The origin is determined to accommodate scenarios where the screen's origin is not fixed to the top left corner when the screen is rotated.

[0042] Use the Suzuki edge contour detection algorithm to extract the edge contour of the mask image, including: The Suzuki algorithm is used to extract the edge contours of the acquired screen area mask image. The Suzuki algorithm can effectively extract the edge contours of the object and provide a basis for subsequent corner detection.

[0043] The extracted edge contour image is used as input and fed into the corner detection model. The input of the corner detection model is the black and white contour image and the position weight map, and the output is a list of corner coordinates. Figure 5The corner detection model consists of three CBRSA modules, a maxpooling module, and two fully connected layers. The CBRSA module consists of a CONV2D layer, a BN layer, and a ReLU activation function connected in sequence. It is then divided into two branches. Branch 1 includes a 1*C*1*1 convolutional layer and an avgpool layer in parallel, followed by a 1*2*3*3 convolutional layer. The output of branches 1 and 2 is then multiplied.

[0044] This method is helpful for detecting corners on curved screens, as it is impossible to obtain four straight lines on a curved screen and thus to use the intersection of straight lines to determine corners. Furthermore, by training on data with missing corners, it can also detect occluded corners and predict their locations.

[0045] Step S202, determining the screen origin, includes: Calculate the distance to determine the screen origin. Use the screen origin coordinates obtained by the segmentation model point output head and the four corner point coordinates to calculate the distance. Determine the corner point with the smallest distance as the screen origin.

[0046] Step S203, refer to Figure 6 , perform projection transformation on the quadrilateral formed by the four corner points to obtain a rectangle, and calculate the projection matrix. The formula for projection transformation is:

[0047] in, are the original coordinates, are the transformed coordinates, The projection matrix is ​​used to map the view coordinates into the quadrilateral coordinate system, that is, into the screen area.

[0048] Step S300, interactive action recognition and response, includes: Step S301, processing the viewpoint coordinate sequence to obtain the interactive action category, including: The acquired viewpoint coordinate sequence is analyzed and processed to determine the driver's interaction action type. For example, rapid movement of the viewpoint coordinates over a short period of time may indicate a sliding action; while prolonged lingering of the viewpoint coordinates in a certain area may indicate a selection action. This determination can be made using threshold-based methods or machine learning algorithms.

[0049] Save the coordinates and interaction action categories to an interaction log file. Each line in the log file contains the gaze coordinates and interaction action category for a frame, expressed as "x, y, cls", where x and y are the gaze coordinates and cls is the interaction category number. If no interaction is performed, the interaction category number is 0.

[0050] Step S302, responding according to the interactive action type and the vehicle screen display content, includes: If the interactive action type is up, down, left, or right sliding, the screen is observed to perform sliding operations based on the sliding speed and distance. For example, when a right sliding action is detected, the content on the screen is scrolled to the right accordingly based on the sliding speed and distance.

[0051] If the interactive action category is selected, the action of clicking the button in the area where the viewpoint is located on the observation screen will be executed. The system will determine the click position based on the viewpoint coordinates and trigger the corresponding function.

[0052] If the interaction action category is determined to be unselected, no action will be performed.

[0053] Step S400, adjusting the interaction sensitivity, includes: Step S401, processing the interaction log file to obtain the operation sensitivity, includes: The recorded interaction log files are processed to analyze the driver's interaction habits. The first 64 frames of data for the interaction are extracted as input samples, with a shape of 64 x 2 (2 represents x and y). Labels are one-hot encoded, where the value under that category is the sensitivity, and all other values ​​are 0.

[0054] The sensitivity of the interaction is obtained by training a neural network model. The sensitivity calculation formula is as follows:

[0055] in, The timestamp of the start frame of the interactive action. The timestamp of executing the interaction action. The maximum allowed response time (the default value is 64 frames).

[0056] A neural network model is trained using labeled log files to obtain interaction sensitivity. The trained neural network model is then used to process labeled and unlabeled log file data to obtain sensitivity values ​​for all categories. The sensitivity is then normalized to a range of 0 to 1. The closer the sensitivity value is to 1, the more sensitive it is.

[0057] Step S402: Adjust the sliding distance and sliding speed according to the sensitivity. Adjust the sliding distance and sliding speed according to the calculated sensitivity. For example, when the sensitivity is 0.9, it means that the user is accustomed to sensitive operation, and the sliding distance and speed can be appropriately increased; when the sensitivity is 0.1, the sliding distance and speed can be appropriately reduced. Adjust the sensitivity according to the user's sliding speed distribution: if the user's average sliding speed is higher than the group mean, reduce the sensitivity. To reduce the sensitivity; otherwise, increase To improve sensitivity.

[0058] By analyzing historical operation data (such as sliding distance and click frequency), the sensitivity parameters are dynamically adjusted to adapt to different user habits (such as the elderly needing a larger sliding area) or scenario requirements (such as reducing sensitivity when driving at high speed to reduce misoperation).

[0059] The embodiments of the present disclosure provide a robust spatial line of sight mapping and embodied interaction system, which utilizes the above-mentioned robust spatial line of sight mapping and embodied interaction method.

[0060] Those skilled in the art will understand that all or part of the processes in a robust spatial line of sight mapping and embodied interaction method can be implemented by instructing related hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of various embodiments of a robust spatial line of sight mapping and embodied interaction method. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0061] This embodiment of the present application also provides a robust spatial line of sight mapping and embodied interaction device (electronic device 500), which utilizes the aforementioned robust spatial line of sight mapping and embodied interaction method. The device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the aforementioned robust spatial line of sight mapping and embodied interaction method are implemented. In this embodiment of the present application, the processor serves as the control center of the computer method and can be a processor of a physical machine or a processor of a virtual machine.

[0062] Reference Figure 7The electronic device 500 includes: at least one processor 501, at least one communication interface 502, at least one memory 503, and at least one bus 504. The bus 504 is used to implement communication between these components, the communication interface 502 is used to communicate signaling or data with other node devices, and the memory 503 stores machine-readable instructions executable by the processor 501. When the electronic device 500 is running, the processor 501 communicates with the memory 503 via the bus 504. When the machine-readable instructions are called by the processor 501, the steps of the above-mentioned robust spatial line of sight mapping and embodied interaction method are executed.

[0063] The above contents are merely embodiments of the present invention. Common knowledge such as the known specific structures and characteristics in the scheme is not described in detail here. A person of ordinary skill in the art is aware of all common technical knowledge in the technical field to which the invention belongs before the filing date or priority date, is able to obtain all existing technologies in the field, and has the ability to apply conventional experimental means before that date. A person of ordinary skill in the art can, under the guidance of this application, improve and implement this scheme in combination with his or her own abilities. Some typical known structures or known methods should not become an obstacle for a person of ordinary skill in the art to implement this application. It should be pointed out that for a person of ordinary skill in the art, several variations and improvements can be made without departing from the structure of the present invention, which should also be regarded as the scope of protection of the present invention, and these will not affect the effect of the implementation of the present invention and the practicality of the patent. The scope of protection claimed in this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. A robust spatial gaze mapping and embodied interaction method, characterized by: include: Obtaining passenger data, including using a head-mounted eye tracker to capture the driver's field of view image and viewpoint coordinates; Processing of occupant data, including: Obtain screen area information and use the improved YOLOv8 instance segmentation algorithm to obtain the screen area mask image and the origin coordinates of the screen area; Use the Suzuki edge contour detection algorithm to extract the edge contour of the mask image and feed it into the corner detection model to obtain a list of corner point coordinates; Determine the screen origin based on the corner point coordinates, and perform projection transformation on the quadrilateral formed by the corner points to obtain a rectangle and projection matrix; Interactive action recognition and response, including: Process the viewpoint coordinate sequence to obtain the interactive action category; and respond according to the interactive action category and the content displayed on the vehicle screen.

2. A robust spatial gaze mapping and embodied interaction method according to claim 1, characterized in that: The improved YOLOv8 instance segmentation model adds a point output head on the basis of the original YOLOv8 algorithm. The point output head includes two CBS modules and a convolutional layer, and the output is batchx*reg_max*2*w*h.

3. A robust spatial gaze mapping and embodied interaction method according to claim 1, characterized in that: The corner detection model includes three CBRSA modules, a maxpooling module and two fully connected layers. The CBRSA module is a CONV2D layer, a BN layer and a Relu activation function connected in sequence, and then divided into two branches. Branch 1 includes a 1*C*1*1 convolutional layer and an avgpool layer in parallel, followed by a 1*2*3*3 convolutional layer, and then branches 1 and 2 are multiplied to obtain the output.

4. A robust spatial gaze mapping and embodied interaction method according to claim 1, characterized in that: Calculate the distance to determine the screen origin. Use the screen origin coordinates obtained by the segmentation model point output head and the four corner point coordinates to calculate the distance. Determine the corner point with the smallest distance as the screen origin.

5. A robust spatial gaze mapping and embodied interaction method according to claim 1, characterized in that: Perform projection transformation on the quadrilateral formed by the four corner points to obtain a rectangle, and calculate the projection matrix. The formula for projection transformation is: in, are the original coordinates, are the transformed coordinates, is the projection matrix.

6. A robust spatial gaze mapping and embodied interaction method according to claim 1, characterized in that: The viewpoint coordinate sequence is processed to obtain the interactive action category, and a threshold-based method or machine learning algorithm is used for classification to determine the interactive action category.

7. A robust spatial sightline mapping and embodied interaction method according to claim 1, characterized in that: Also included is adjusting interaction sensitivity, including: The interaction log files are processed and the data of the first 64 frames of the interaction action are extracted as input samples to obtain the operation sensitivity. The sliding distance and sliding speed are adjusted according to the operation sensitivity.

8. A robust spatial sightline mapping and embodied interaction method according to claim 7, characterized in that: The sensitivity calculation formula is as follows: in, The timestamp of the start frame of the interactive action. The timestamp of executing the interaction action. is the maximum allowed response time; Adjust sensitivity based on user sliding speed distribution: if the average sliding speed of users is higher than the group mean, reduce To reduce sensitivity; otherwise, increase To improve sensitivity.

9. A robust spatial gaze mapping and embodied interaction system, characterized by: A robust spatial line of sight mapping and embodied interaction method as described in any one of claims 1-8 is used.

10. A robust spatial gaze mapping and embodied interaction device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, wherein when the processor executes the program, a robust spatial sightline mapping and embodied interaction method as described in any one of claims 1 to 8 is implemented.

Images