Control terminal, control device and control system

Abstract

This invention provides a control terminal, control device, and control system that improve system safety when remotely controlling a mobile device. The control terminal of this invention is carried by a user and used for remotely moving a mobile device capable of communicating with the control terminal. The control terminal includes: a gaze information detection unit that detects the user's gaze information towards the control terminal; a gaze state determination unit that determines the user's gaze state towards the control terminal based on the gaze information; a gaze state control unit that performs control based on the gaze state determined by the gaze state determination unit; and a notification unit that notifies the user. Furthermore, the gaze state control unit performs a first determination to determine whether the continuous gaze period of the gaze state exceeds the first period. When the result of the first determination is that the continuous gaze period exceeds the first period, it instructs the notification unit to notify the user to look at the mobile device.

Description

Technical Field

[0001] This invention relates to a control terminal, control device, and control system, and particularly to a terminal, device, and system for remote vehicle control with safety monitoring functions. Background Technology

[0002] In recent years, there has been a demand to improve traffic safety in order to make cities and human settlements more inclusive, safe, resilient, and sustainable. Regarding vehicles, there is a requirement to ensure traffic safety even in designated parking spaces.

[0003] With the development of autonomous driving technology, remote parking systems (hereinafter referred to as "remote summoning") that allow users to remotely operate vehicles using smartphones to park them in designated parking spaces or summon them from parking spaces have gradually been applied in real-world scenarios. For example, users can use their smartphones at the entrance of a shopping mall parking garage to have their parked vehicles automatically drive to their location. Summary of the Invention

[0004] The technical problem that the invention aims to solve

[0005] Current vehicle remote control technology requires users to operate on terminal devices such as smartphones (e.g., clicking, swiping, swiping and holding). During operation, users need to frequently look at the screen, which can easily cause them to ignore their surroundings.

[0006] With the increasing number of vehicles, even the roads in parking lots are becoming more and more complex, posing many safety risks. For example, pedestrians or other moving obstacles may suddenly appear around the vehicle, the vehicle may unexpectedly sway or deviate from the intended path, and other vehicles in the parking lot may suddenly start or open their doors. Users need to detect these situations in time and take appropriate measures.

[0007] While existing autonomous driving technologies have developed techniques for obstacle avoidance and path correction, misjudgments can still occur due to factors such as potential blind spots in the sensors upon which these technologies rely, incomplete camera coverage of the vehicle's surroundings, and image transmission delays that may cause the images to lag behind reality. Therefore, when remotely summoning a vehicle, the user outside the car must have direct visual contact with the vehicle and take evasive action in emergency situations (such as releasing a finger to stop autonomous driving).

[0008] Furthermore, current remote summoning technology is generally considered a Level 2 autonomous driving function (a classification standard defined by SAE (Society of Automotive Engineers), which is divided into six levels from L0 to L5), characterized by the lack of fully autonomous decision-making capabilities. As a requirement of Level 2 systems, users must constantly monitor the system's operating status and remotely control the vehicle from outside the car via mobile terminals such as smartphones, and must be ready to completely take over vehicle control at any time.

[0009] Therefore, the current operating method, which requires users to frequently look at their smartphone screens during operation, contradicts the need for secure monitoring of vehicle movement.

[0010] The purpose of this invention is to provide a control terminal, control device, and control system that can improve the safety of system use when remotely controlling a mobile device.

[0011] Methods for solving technical problems

[0012] This invention is a control terminal, a user-carried mobile device capable of remote movement and communication with the control terminal, wherein...

[0013] The control terminal includes:

[0014] A gaze detection unit that detects the user's gaze at the control terminal;

[0015] A gaze state determination unit that determines the user's gaze state towards the control terminal based on the gaze information; a gaze state control unit that performs control based on the gaze state determined by the gaze state determination unit; and

[0016] The notification department that notifies the user, and,

[0017] The gaze state control unit

[0018] A first determination is made to determine whether the duration of the sustained gaze in the gaze state exceeds a first period.

[0019] When the result of the first determination is that the continuous gaze period exceeds the first period, the notification unit is instructed to issue a notification prompting the user to look at the mobile device.

[0020] This invention relates to a control device that is communicatively connected to a user-carried control terminal and enables a mobile device to move remotely based on instructions from the control terminal.

[0021] The control terminal includes:

[0022] A gaze detection unit that detects the user's gaze at the control terminal, and

[0023] The notification department that notifies the user.

[0024] The control device includes:

[0025] A gaze determination unit that determines the user's gaze state towards the control terminal based on the gaze information obtained through communication with the control terminal, and...

[0026] A gaze state control unit that performs control based on the gaze state determined by the gaze state determination unit, and

[0027] The gaze state control unit

[0028] A first determination is made to determine whether the duration of the sustained gaze in the gaze state exceeds a first period.

[0029] When the result of the first determination is that the continuous gaze period exceeds the first period, the notification unit of the control terminal is instructed to notify the user to look at the mobile device through communication with the control terminal.

[0030] This invention is a control system, comprising a user-carried control terminal and a control device communicatively connected to the control terminal for enabling remote movement of a mobile device, wherein...

[0031] The control terminal includes:

[0032] A gaze information detection unit that detects the user's gaze at the control terminal.

[0033] Based on the gaze information, a gaze state determination unit determines the user's gaze state towards the control terminal; a gaze state control unit performs control based on the gaze state determined by the gaze state determination unit; and a notification unit notifies the user.

[0034] The gaze state control unit

[0035] A first determination is made to determine whether the duration of the sustained gaze in the gaze state exceeds a first period.

[0036] When the result of the first determination is that the continuous gaze period exceeds the first period, the notification unit is instructed to issue a notification prompting the user to look at the mobile device.

[0037] This invention is a control system, comprising a user-carried control terminal and a control device communicatively connected to the control terminal for enabling remote movement of a mobile device, wherein...

[0038] The control terminal includes:

[0039] A gaze detection unit that detects the user's gaze at the control terminal, and

[0040] The notification department that notifies the user.

[0041] The control device includes:

[0042] A gaze determination unit that determines the user's gaze state towards the control terminal based on the gaze information obtained through communication with the control terminal, and...

[0043] A gaze state control unit that performs control based on the gaze state determined by the gaze state determination unit, and,

[0044] The gaze state control unit

[0045] A first determination is made to determine whether the duration of the sustained gaze in the gaze state exceeds a first period.

[0046] When the result of the first determination is that the continuous gaze period exceeds the first period, the notification unit of the control terminal is instructed to notify the user to look at the mobile device through communication with the control terminal.

[0047] The effects of the invention

[0048] According to the present invention, a control terminal, control device, and control system can be provided to improve the safety of system use when remotely controlling a mobile device. Attached Figure Description

[0049] Figure 1 This diagram illustrates a scenario where a smartphone is used to control a vehicle's exit from a parking space.

[0050] Figure 2 yes Figure 1 The diagram shows the internal structure of a smartphone.

[0051] Figure 3 This is the main flowchart of the automated outbound process executed on a smartphone.

[0052] Figure 4 This is a schematic diagram illustrating an example of an interface for preparing for automatic warehouse dispatch.

[0053] Figure 5 This is a schematic diagram illustrating an example of an automated outbound operation interface.

[0054] Figure 6 This is a flowchart of the gaze monitoring steps performed on a smartphone.

[0055] Figure 7 This is another example of an automatic inventory checkout interface.

[0056] Figure 8 yes Figure 1 The diagram shows a block diagram of the vehicle's internal structure.

[0057] Explanation of reference numerals in the attached figures

[0058] 1. Smartphone (control terminal)

[0059] 11. Visual Information Detection Department

[0060] 111 Front-facing camera

[0061] 112. Infrared camera

[0062] 113, 212 Ambient Light Sensors

[0063] 114, 213 Human Distance Sensors

[0064] 115 Posture Sensor

[0065] 12, 281 Gaze Status Determination Unit

[0066] Image preprocessors 121 and 2811

[0067] 122, 2812 Facial Feature Extractor 123, 2813 Eye-tracking processors 124 and 2814 feature comparison units 13, 282 Gaze Status Control Unit

[0068] 131, 2821 Gaze state timer 132, 2822 Non-gaze timer 133, 2823 Time Comparison Unit 14 Notification Department

[0069] 141 Notification Content Generation Department

[0070] 142 Display Screen

[0071] 143 Vibration Motor

[0072] 144 speakers

[0073] 145 LED indicator lights 15-position detection department

[0074] 151 GPS receiver

[0075] 16 Ministry of Communications 2. Vehicles (mobile devices)

[0076] 21 External sensor group

[0077] 211 camera group

[0078] 2111 Front-facing camera

[0079] 2112 Rear Side Camera

[0080] 2113 Left-side camera

[0081] 2114 Right-side camera

[0082] 22 Vehicle sensor groups

[0083] 221 Vehicle speed sensor

[0084] 222 Accelerometer

[0085] 223 Azimuth Sensor

[0086] 23 Navigation devices

[0087] 231 GPS Receiver

[0088] 232 Map Storage Department

[0089] 24 Propulsion device

[0090] 25 Braking device

[0091] 26. Steering mechanism

[0092] 27 Ministry of Communications

[0093] 28. Control device

[0094] 2815 Standing Posture Feature Extractor

[0095] 283 Automatic Outbound Control Department

[0096] 2831 Trajectory Planning Department

[0097] 2832 Driving Control Unit

[0098] 284 Storage Unit

[0099] 285 Input / Output Section

[0100] M user Detailed Implementation

[0101] The following describes one embodiment of the control terminal, control device, and control system of the present invention based on the accompanying drawings. The following embodiment does not limit the invention to the scope of the patent claims, and not all combinations of features described in the embodiment are essential to the invention. Two or more features described in the embodiment can be combined arbitrarily. Furthermore, identical or similar structures are labeled with the same numbers, and repeated descriptions are omitted.

[0102] One embodiment of the vehicle control system of the present invention includes a control terminal and a mobile device.

[0103] A control terminal is a wireless terminal carried by a user, capable of communicating with a mobile device from the user's location via a communication module. In this embodiment, the control terminal is, for example, a portable information processing device such as a smartphone 1. By pre-installing a specified application on the control terminal, the control terminal can communicate with the mobile device and issue commands. The mobile device, such as a vehicle 2, can communicate with the user-carried control terminal, receive commands from the control terminal, and move according to those commands.

[0104] Figure 1 This is a schematic diagram illustrating an application scenario of a remote mobility control system according to an embodiment of the present invention. User M is a certain distance from vehicle 2. By operating a smartphone 1 held by user M, user M instructs vehicle 2 to automatically park in a designated parking lot, or instructs vehicle 2 to automatically leave the parking space and return to user M's location, facilitating user M's driving.

[0105] The structure and processing flow of the smartphone 1 and vehicle 2 of the present invention, which realize automatic parking, will be described below. It should be noted that although the automatic parking process is described in detail below as an example, it can be equally applied to the automatic parking process.

[0106] <Structure of a Smartphone>

[0107] like Figure 2 As shown, the smartphone 1 is configured to include: a gaze information detection unit 11, a gaze state determination unit 12, a gaze state control unit 13, and a notification unit 14.

[0108] The gaze information detection unit 11 includes a front-facing camera 111, which is responsible for capturing facial images of user M, especially image data of user M's eyes. Preferably, the front-facing camera 111 captures images at a frequency of not less than 10 frames per second (i.e., sampling once every 0.1 seconds). A higher sampling frequency can avoid misjudgments caused by normal blinking or the user M's gaze shifting away from the screen and immediately returning to it (looking up) when looking at the phone screen.

[0109] Preferably, the gaze information detection unit 11 may further include: an infrared camera 112, an ambient light sensor 113, a human body distance sensor 114, and a posture sensor 115. Specifically: the infrared camera 112 can work synchronously with the front-facing camera 111, providing all-weather facial feature recognition capabilities through image fusion algorithms; the ambient light sensor 113 can automatically adjust the backlight of the display screen 142 to supplement illumination when insufficient light is detected in the facial area, and adjust the camera's exposure parameters when the light is too strong, which helps the front-facing camera 111 obtain a clearer facial image of the user M; the human body distance sensor 114 and the posture sensor 115 accurately measure the distance between the user M's face and the smartphone 1, as well as the user M's posture while holding the smartphone 1, and can provide distance and posture information to the front-facing camera 111 for automatic focusing.

[0110] The gaze determination unit 12 is used to analyze and compare the facial image of user M captured by the front camera 111 to determine whether user M is looking at the screen.

[0111] The gaze determination unit 12 includes an image preprocessor 121, a facial feature extractor 122, an eye-tracking processor 123, and a feature comparison unit 124. The image preprocessor 121 performs preprocessing such as noise reduction and enhancement on images acquired by the front-facing camera 111 and / or the infrared camera 112. The facial feature extractor 122 extracts facial feature points of user M, particularly feature points in the eye region, from the preprocessed images. The eye-tracking processor 123 analyzes the trajectory of user M's eye movements based on the extracted feature points. The feature comparison unit 124 compares the facial feature points and the eye movement trajectory with a preset facial feature model, outputting a result indicating whether the user is in a gaze state. The facial feature model is a model built by collecting a large amount of image data of human facial states when gazing at a mobile phone screen in different postures, organizing this data, extracting features, and establishing the model. In a preferred embodiment, a data model can also be created based on the facial image state of user M holding the smartphone 1 when gazing at the screen, for later comparison with real-time captured facial images of user M.

[0112] The determination result of the gaze state determination unit 12 includes gaze state and non-gaze state. The gaze state means that the user M's gaze is on the mobile phone screen, that is, the user M is looking at the mobile phone screen. The non-gaze state means that the user M's gaze is not on the mobile phone screen, but is paying attention to the vehicle and / or the vehicle's surrounding environment.

[0113] The gaze state determination unit 12 continuously sends the determination result, i.e., the information representing "gaze state" and "non-gaze state", to the gaze state control unit 13 at a frequency no less than the sampling frequency of the front camera 111.

[0114] The gaze state control unit 12 includes at least: a gaze state timer 131, a non-gaze state timer 132, and a time comparison unit 133.

[0115] Preferably, the gaze state timer 131 includes a high-precision timer (not shown) and a buffer timer (not shown).

[0116] A high-precision timer continuously records the time for receiving "gaze state" information, while a buffer timer remains in standby mode. Upon receiving "non-gaze state" information (i.e., user M's gaze at the phone screen is interrupted), the buffer timer immediately starts timing, and the high-precision timer pauses to maintain its current accumulated value. If the buffer timer has not reached a preset buffer time threshold (e.g., 0.5 seconds) before receiving "gaze state" information (i.e., user M's gaze quickly returns to the phone screen), the buffer timer stops and resets, while the high-precision timer continues recording based on the previously paused accumulated value. If the buffer timer exceeds the preset buffer time threshold, the high-precision timer resets to 0. This avoids interruptions in the gaze state timing caused by user M's physiological blinking or immediate return to the phone screen after looking away (e.g., glancing up briefly and then looking back), which could lead to the mistaken impression that user M's gaze time was shorter than expected.

[0117] The non-attention timer 132 also includes a high-precision timer (not shown) that counts the duration of continuously received "non-attention" information. Like the attention timer, the non-attention timer 132 may also include a buffer timer (not shown). However, preferably, the buffer timer is not set, and the high-precision timer is immediately reset to 0 upon receiving "attention" information (i.e., user M's gaze is on the phone screen). In this way, by only counting the time when user M is highly focused on vehicle 2 and completely ignores the phone screen as non-attention time, it is possible to ensure automatic exit from the parking space when user M is indeed looking at vehicle 2, thus improving the quality of the user gaze monitoring program.

[0118] The time comparison unit 133 receives the duration of the gaze state (i.e., the continuous gaze period) sent from the gaze state timer 131 and compares them with a plurality of preset time thresholds stored therein. These plurality of time thresholds may be, for example, a first period, a second period, a third period, and a fourth period.

[0119] When the time comparison unit 133 obtains the result that the duration of the gaze state exceeds the first period, the gaze state control unit 13 instructs the notification unit 14 to issue a notification prompting the user M to look at the vehicle 2.

[0120] When the time comparison unit 133 receives a result indicating that the duration of the gaze state exceeds the second period, the gaze state control unit 13, through communication with the vehicle 2, sends an instruction to the vehicle 2 to limit its speed to below a specified speed, i.e., to cause the vehicle 2 to slow down. This specified speed is, for example, below 10 km / h, preferably below 8 km / h, more preferably below 6 km / h, and even more preferably below 3 km / h. At the same time, the notification unit 14 may also simultaneously issue a notification prompting the user M to look at the vehicle 2.

[0121] When the time comparison unit 133 obtains the result that the duration of the gaze state exceeds the third period, the gaze state control unit 13 sends an instruction to the vehicle 2 to stop driving through communication with the vehicle 2, that is, to set the driving speed of the vehicle 2 to 0.

[0122] When the time comparison unit 133 obtains the result that the gaze state duration exceeds the fourth period, the gaze state control unit 13 sends an instruction to the vehicle 2 to stop the automatic exit operation through communication with the vehicle 2.

[0123] Preferably, the first period, second period, third period, and fourth period are preset to increase sequentially. For example, in a preferred embodiment, the first period can be set to 2 seconds, the second period can be set to 4 seconds, the third period can be set to 6 seconds, and the fourth period can be set to 8 seconds.

[0124] The time comparison unit 133 also receives the duration of the non-attention state sent from the non-attention state timer 132 and compares it with a preset predetermined time. When the comparison result shows that the duration of the non-attention state is greater than the predetermined time, the attention state control unit 13 will, through communication with the vehicle 2, instruct the vehicle 2 to release a series of restrictive operations performed during the exit process due to the user M's excessive screen-gazing time, including: releasing the instruction to have the notification unit 14 issue a notification; releasing the instruction to have the vehicle 2 slow down; and releasing the instruction to have the vehicle 2 stop driving.

[0125] For example, the specified time can be set to be greater than or equal to the first period and less than the second period. For instance, if the first period can be set to 2 seconds and the second period can be set to 4 seconds, the specified time can be set to 3 seconds. By setting the specified time to be greater than or equal to the first period, it can be ensured that vehicle 2 is instructed to automatically exit the vehicle while user M is reliably and continuously monitoring the vehicle. Furthermore, by setting the specified time to be less than the second period, the automatic exit operation can be quickly resumed in response to user M visually observing vehicle 2, resulting in a better user experience.

[0126] The notification unit 14 may include: a notification content generation unit 141, a display screen 142, a vibration motor 143 providing haptic feedback, a speaker 144, and an LED indicator 145. For example, when the notification unit 14 receives a notification from the gaze state control unit 13 prompting the user to look at the vehicle, or receives a notification from the vehicle 2 prompting the user to look at the vehicle, it may, for example, perform the following notifications: the notification content generation unit 141 generates text, images, or video information of "Please look at the vehicle" and displays it on the display screen 142; the vibration motor 143 emits vibrations of a specific rhythm and the LED indicator 145 flashes to remind the user M that there is an abnormal situation that needs to be handled; the speaker 144 plays the voice message "Please look at the vehicle," etc.

[0127] The location detection unit 15 is capable of acquiring the location information of the smartphone 1. For example, the location detection unit 15 can acquire the location of the smartphone 1 by receiving signals from geodetic satellites (GPS satellites) via the GPS receiver 151. In addition, by communicating with the vehicle 2 via the communication unit 16, the location detection unit 15 can also acquire information including the relative position of the smartphone 1 with respect to the vehicle 2.

[0128] The communication unit 16 enables wireless communication between the smartphone 1 and the vehicle 2. That is, the smartphone 1 can communicate with the vehicle 2 via the communication unit 16, for example, using infrared communication or Bluetooth (registered trademark).

[0129] In addition, smartphone 1 also includes a power management unit, memory, and a main processor (none of which are shown). The power management unit is responsible for power supply control, the memory stores programs and data, and the main processor executes overall program control. These components work together to ensure the normal operation of smartphone 1.

[0130] <Process for Smartphones to Instruct Vehicles to Automatically Exit the Parking Lot>

[0131] Figure 3 This is the main flowchart of the automated outbound program running on smartphone 1.

[0132] User M finds and opens the pre-installed application on smartphone 1 and clicks the "Start Automatic Outbound" button.

[0133] First, smartphone 1 determines whether it has received a request from user M to start the automatic departure procedure of vehicle 2 (step S10).

[0134] In step S10, if no application launch operation is received (step S10: No), the smartphone 1 repeatedly executes the processing of step S10 and waits until an application launch operation is received.

[0135] In step S10, if an application launch operation is received (step S10: Yes), the smartphone 1 displays the regulations and precautions (disclaimers) related to automatic delivery on the display screen 142 (step S20).

[0136] For example, Figure 4 This is an example of an interface for preparing for automatic outbound shipment, in which a disclaimer is displayed on the screen 142 of smartphone 1. Figure 4 As shown, the display screen 142 displays a disclaimer such as: "Note: This function is not fully autonomous driving...". Additionally, the disclaimer interface may include a button for user M to confirm their agreement to the disclaimer; this button can be either a clickable or swipe-to-click button. Along with this button, text information guiding user M's action, such as "Swipe right to agree to the disclaimer," may also be displayed.

[0137] In step S30, it is determined whether a consent disclaimer has been received from user M. If no consent disclaimer has been received from user M (step S30: no), the process of step S30 is repeated and the user continues to wait.

[0138] When the information of consent to the disclaimer is received from user M (step S30: Yes), smartphone 1 obtains the generated information related to automatic delivery based on the communication connection established with vehicle 2, and then displays the operation interface for preparing automatic delivery on display screen 142 (step S40).

[0139] like Figure 5As shown, the interface for preparing for automatic vehicle departure may include: a map showing the expected driving trajectory for automatic departure planned based on the GPS positioning of smartphone 1 and vehicle 2; the estimated time required for the vehicle to reach the user M's location calculated based on the expected driving trajectory; a virtual button for starting vehicle movement (i.e., a "summon" button); and prompts for user M to operate the button. For example, the interface for preparing for automatic departure may display text messages such as "Please press and hold here to instruct the vehicle to perform automatic departure" or "Release your finger to pause the vehicle's movement," guiding user M to operate the system.

[0140] In step S50, it is determined whether a user M has been instructed by the user interface to automatically exit the vehicle.

[0141] Only when the smartphone 1 continuously receives instructions from user M to automatically exit the parking space (i.e., continuously press and hold the "call" button) (step S50: Yes) will the smartphone 1 send an instruction to the vehicle 2 to perform automatic exit (step S60), and then immediately start the gaze state detection program (step S70).

[0142] If no instruction from user M is received for the vehicle to automatically exit the warehouse (i.e., the user releases their finger) (step S50: No), then return to step S40 and return to the operation interface for preparing for automatic exit.

[0143] In a preferred embodiment, in step S60, when the user M instructs the vehicle to automatically leave the parking space, that is, when the user M presses and holds the "summon" button, the state of the summon button can be changed, and unnecessary information such as the expected driving trajectory map and countdown will no longer be displayed, so as not to distract the user M.

[0144] In step S80, it is continuously determined whether automatic outbound has been completed. If automatic outbound has not been completed, the gaze status monitoring step S70 continues until the end.

[0145] <Processing flow for smartphone gaze monitoring>

[0146] The following is for reference Figure 6 The flowchart provides a detailed explanation of the gaze state monitoring step S70.

[0147] In step S701, the time that user M gazes at display screen 142 is counted to obtain the duration of continuous gaze.

[0148] In step S702, it is determined whether the duration of sustained gaze exceeds the first period.

[0149] If the continuous gaze period exceeds the first period (step 702: Yes), the smartphone 1 sends a notification to the user M via the notification unit 14, prompting the user M to look at the vehicle 2 through the display screen, sound, or vibration (step S703). Figure 7 This is one embodiment of notifying the user M to look at vehicle 2 via text on a display screen.

[0150] If the sustained fixation period does not exceed the first period (step 702: No), then the statistical fixation time processing in step S701 is repeated.

[0151] After step S703, the step of determining whether the continuous gaze period exceeds the second period is performed (step S704).

[0152] If the continuous gaze period exceeds the second period (step S704: Yes), the smartphone 1 sends a command to the vehicle 2 to limit the driving speed, causing the vehicle 2 to slow down (step S705).

[0153] If the sustained fixation period does not exceed the second period (step 704: No), return to the processing of statistical fixation time in step S701.

[0154] After step S705, the step of determining whether the duration of the non-gazing state exceeds a predetermined time is performed (step S706).

[0155] If the non-gazing state lasts for more than the specified time (step S706: Yes), then the smartphone 1 sends a release command to the vehicle 2 to slow down the vehicle 2 (step S711), and then returns to the processing of the statistical gaze time in step S701.

[0156] If the non-gazing state does not exceed the specified time (step S706: No), then proceed to the step of determining whether the continuous gazing period exceeds the third period (step S707).

[0157] If the continuous gaze period exceeds the third period (step S707: Yes), the smartphone 1 sends a command to the vehicle 2 to stop driving, causing the vehicle 2 to temporarily stop driving, that is, to set the target value of the driving speed to 0 (step S708).

[0158] If the sustained fixation period does not exceed the third period (step 707: No), return to the processing of statistical fixation time in step S701.

[0159] After step S708, the step of determining whether the duration of the non-gazing state exceeds a predetermined time is performed (step S709).

[0160] If the non-gazing state lasts for more than the specified time (step S709: Yes), then a command to release the vehicle 2 from the pause is sent to the vehicle 2, in other words, a command to resume the vehicle 2 from the pause is issued (step S712), and then the process of calculating the gaze time in step S701 is returned.

[0161] If the non-gazing state does not last longer than the specified time (step S709: No), then proceed to the step of determining whether the continuous gazing period exceeds the fourth period (step S710).

[0162] If the continuous gaze duration exceeds the fourth period (step S710: Yes), then smartphone 1 sends a command to vehicle 2 to stop the exit operation (step S713). If the continuous gaze duration does not exceed the fourth period (step S710: No), then return to the processing of statistical gaze time in step S701.

[0163] By detecting that user M is looking at the display screen 142 of smartphone 1 for a certain period of time (the first period) without looking at vehicle 2, a notification is issued to remind user M. Then, if user M continues to look at the display screen 142 without looking at vehicle 2, as the duration of not looking at vehicle 2 increases, vehicle 2 is gradually slowed down, stopped, and eventually stopped from automatically leaving the parking space. This can prevent vehicle 2 from driving without monitoring and causing unexpected consequences.

[0164] <Vehicle Structure>

[0165] Vehicle 2 has an external sensor group 21, a vehicle sensor group 22, a navigation device 23, a propulsion device 24, a braking device 25, a steering device 26, a communication unit 27, and a control device 28.

[0166] The various components of vehicle 2 are connected by an onboard communication network such as CAN (Controller Area Network) to enable signal transmission.

[0167] The propulsion device 24 is a device that provides driving force to the vehicle, and includes, for example, a power source and a transmission. The power source has at least one of an internal combustion engine and an electric motor, such as a gasoline engine or a diesel engine. The braking device 25 is a device that provides braking force to the vehicle, and includes, for example, a brake caliper that presses brake pads against a brake disc and an electric cylinder that supplies hydraulic pressure to the brake caliper. The braking device 25 includes a parking brake that limits wheel rotation via a steel cable. The steering device 26 is a device for changing the wheel steering angle, and includes, for example, a rack and pinion mechanism that steers the wheels and an electric motor that drives the rack and pinion mechanism. The propulsion device 24, the braking device 25, and the steering device 26 are controlled by a control device 28.

[0168] The external sensor group 21 is a sensor that detects objects and other objects around the vehicle 2. The external sensor group 21 includes a camera group 211, an ambient light sensor 212, a human distance sensor 213, etc., and outputs the detection results to the control device 28.

[0169] Camera group 211 is a device for capturing images around the vehicle, such as a digital camera using a solid-state imaging element like a CCD or CMOS. Camera 211 includes a front-side camera 2111 for capturing images in front of the vehicle and a rear-side camera 2112 for capturing images behind the vehicle. Camera 18 may also include a left-side camera 2113 and a right-side camera 2114 located near the mounting position of the vehicle's door mirrors for capturing images on the left and right sides.

[0170] The vehicle sensor group 22 includes a vehicle speed sensor 221 for detecting vehicle speed, an acceleration sensor 222 for detecting acceleration, and an orientation sensor 223 for detecting vehicle direction.

[0171] The communications department 27 establishes wireless communication between vehicle 2 and smartphone 1.

[0172] The navigation device 23 is a device that obtains the current position of the vehicle 2 and provides destination route guidance, etc., and includes a GPS receiver 231 and a map storage unit 232. The GPS receiver 231 determines the vehicle's position (latitude and longitude) based on signals received from artificial satellites (positioning satellites). The map storage unit 232 is composed of a storage device such as flash memory or a hard disk, and stores map information.

[0173] The control unit 28 is an electronic control unit (ECU) that includes a CPU, non-volatile memory (ROM), and volatile memory (RAM). The control unit 28 performs program-based computation through the CPU, enabling it to perform various vehicle controls. At least a portion of the functional parts of the control unit 28 can be implemented using hardware such as LSI, ASIC, or FPGA, or through a combination of software and hardware.

[0174] The control device 28 includes a gaze state determination unit 281, a gaze state control unit 282, an automatic exit control unit 283, a storage unit 284, an input / output unit 285, etc. The control device 28 can control the driving of the vehicle 2, and can also control the driving of the vehicle 2 according to the instructions from the smartphone 1.

[0175] When the automatic delivery control unit 283 receives an operation command to activate the automatic delivery control from the smartphone 1 carried by user M, it communicates with user M's smartphone 1 and controls the vehicle 2 to automatically drive to the location specified by user M. This location specified by user M can be a location selected by user M on a map, or it can be user M's current location.

[0176] The trajectory planning unit 2831 calculates the trajectory as the vehicle's driving path as needed.

[0177] When the system receives an automatic exit command from user M, the trajectory planning unit 2831 obtains the current location of vehicle 2 from the GPS receiver 231, obtains a map of the parking lot from the map storage unit 232, and obtains the location specified by user M through communication with the smartphone 1. Next, the trajectory planning unit 2831 calculates the expected driving trajectory from the current location of vehicle 2 to the location specified by user M, and sends the expected driving trajectory to the driving control unit 2832.

[0178] In addition, the trajectory planning unit 2831 can also calculate the time it takes for vehicle 2 to reach the location specified by user M based on the vehicle speed preset by the automatic exit program.

[0179] The driving control unit 2832, based on the trajectory of the predetermined driving path obtained from the trajectory planning unit 2831 and the vehicle speed detected in real time by the vehicle speed sensor 221, controls the operation of the propulsion device 24, the braking device 25, and the steering device 26, so that the vehicle travels along the predetermined trajectory and speed. By controlling the driving control unit 2832 to automatically execute the steering, acceleration, and braking operations of the vehicle 2, the automatic exit function of the vehicle 2 is realized.

[0180] In summary, the automatic exit control unit 283 can execute automatic exit control to drive the vehicle from its current parking position to the end of the specified path based on the vehicle status data obtained by the vehicle sensor group 22, the location information provided by the navigation device 23, and the target exit path specified by the user M via the smartphone 1.

[0181] Furthermore, during the automatic exit operation, the automatic exit control unit 283 continuously detects whether it continuously receives the automatic exit command sent from the smartphone 1 (i.e., the user M keeps pressing the "call" button). Once the command is not received, the automatic exit control unit 283 will immediately send a pause command to the driving control unit 2832 to stop the vehicle from exiting the parking lot.

[0182] In addition, the driving control unit 2832 also receives slow-down commands, stop driving commands (i.e., setting the driving speed to 0), and stop exiting the parking space commands from the smartphone 1. When a slow-down command is received, the driving control unit 2832 controls the propulsion device 24 and, if necessary, the braking device 25 to limit the vehicle speed below a specified value. When a stop command is received, the propulsion device 24 is stopped, and the vehicle is brought to a stop by the braking device 25. When a stop command is received, the maximum braking force is applied by the braking device 25 to bring the vehicle to an emergency stop.

[0183] When the control device 28 receives an automatic delivery instruction from the smartphone 1, it requests the facial image data of the user M collected by the smartphone 1 via the communication unit 27.

[0184] In addition to image data acquired from smartphone 1, control device 28 can also acquire image data of user M from the external sensor group 21 located on vehicle 2 itself. Specifically, it can capture images of the surrounding environment using front camera 2111, rear camera 2112, left camera 2113, and right camera 2114 positioned at four locations on vehicle 2, and analyze faces appearing in the environmental images. When a face image of user M is detected in the image captured by one of the cameras, it requests that camera to capture a full-body image of user M, including the face, at a frequency of not less than 10 frames per second (i.e., sampling once every 0.1 seconds).

[0185] Furthermore, the external sensor group 21 may also include an ambient light sensor 212 and a human distance sensor 213. The ambient light sensor 212 can automatically adjust the exposure parameters of the camera group 211 when insufficient ambient light is detected, which helps the camera group 211 obtain a clearer facial image of the user M. The human distance sensor 215 can accurately measure the distance between the user M and the vehicle 2 and provide distance information to the camera group 211 for autofocus, in order to capture a clearer image of the user M.

[0186] The gaze state determination unit 281 of the control device 28 can analyze and compare the facial image of user M based on the facial image data of user M sent by smartphone 1 and the facial image of user M captured by camera group 211, so as to determine whether user M is looking at the display screen 142 of smartphone 1.

[0187] Specifically, the gaze state determination unit 281 includes: an image preprocessor 2811, a facial feature extractor 2812, an eye-tracking processor 2813, and a feature comparison unit 114. The image preprocessor 2811 performs preprocessing such as noise reduction and enhancement on images sent by the smartphone 1 and / or images captured by the camera group 211. The facial feature extractor 2812 extracts facial feature points of the user M, particularly feature points in the eye region, from the preprocessed images. The eye-tracking processor 2813 analyzes the trajectory of the user M's eye movements based on the extracted feature points. The feature comparison unit 2814 compares the facial feature points and the eye movement trajectory with a preset facial feature model, outputting a result indicating whether the user is in a gaze state. The facial feature model is a model built by collecting a large amount of image data of human facial states when gazing at a mobile phone screen in different postures, organizing this data, extracting features, and establishing the model. In a preferred embodiment, the facial image state of the user M holding the smartphone 1 when gazing at the screen can also be individually collected and a data model can be created for later comparison with real-time captured facial images of the user M.

[0188] In another preferred embodiment, the gaze determination unit 281 further includes a posture feature extraction unit 2815, which extracts the posture features of user M, especially head turning features, from the images captured by the camera group 211, and compares them with a preset posture feature model. The preset posture feature model may include image data of user M standing and looking around the vehicle, user M looking down at the smartphone, etc., and the model is established by processing and extracting posture features from this data. When the posture of user M holding the smartphone 1 causes the front camera 111 to not cover user M's face (for example, the front camera 111 only captures the sky or ceiling), the feature comparison unit 2814 can also determine whether user M is looking down at the phone screen or looking at the vehicle based on the posture of user M extracted by the posture feature extraction unit 2815.

[0189] The determination result of the gaze state determination unit 281 includes gaze state and non-gaze state. The gaze state means that the user M's gaze is on the mobile phone screen, that is, the user M is looking at the mobile phone screen. The non-gaze state means that the user M's gaze is not on the mobile phone screen, but is paying attention to the vehicle and / or the vehicle's surrounding environment.

[0190] The gaze state determination unit 281 continuously sends the determination results, namely the "gaze state" and "non-gaze state" information, to the gaze state control unit 282 at a frequency no less than the sampling frequency of the front camera 111 and the camera group 211.

[0191] The gaze state control unit 281 includes at least: a gaze state timer 2821, a non-gaze state timer 2822, and a time comparison unit 2823.

[0192] Preferably, the gaze state timer 2821 includes a high-precision timer (not shown) and a buffer timer (not shown).

[0193] The high-precision timer continuously counts the time for receiving "gaze state" information, while the buffer timer remains in standby mode. Upon receiving "non-gaze state" information (i.e., user M's gaze at the phone screen is interrupted), the buffer timer immediately starts timing, and the high-precision timer pauses to maintain its current accumulated value. If the buffer timer has not reached a preset buffer time threshold (e.g., 0.5 seconds) before receiving "gaze state" information (i.e., user M's gaze quickly returns to the phone screen), the buffer timer 2821 stops timing and resets, while the high-precision timer continues recording based on the previously provisional accumulated value. If the buffer timer's timing exceeds the preset buffer time threshold, the high-precision timer resets to 0. This avoids interruptions in the gaze state timing caused by user M's physiological blinking or immediate return to the phone screen after looking away (e.g., glancing up briefly and then looking back), which could lead to the mistaken impression that user M's gaze at the phone screen was not long enough.

[0194] The non-attention timer 2822 also includes a high-precision timer (not shown) that counts the duration of continuously received "non-attention" information. Like the attention timer 2821, the non-attention timer 2822 may also include a buffer timer (not shown). However, preferably, the buffer timer is not set, and the high-precision timer is immediately reset to 0 upon receiving "attention" information (i.e., user M's gaze is on the phone screen). In this way, by only counting the time when user M is highly focused on vehicle 2 and completely ignores the phone screen as non-attention time, it is possible to ensure automatic exit from the parking space when user M is indeed looking at vehicle 2, thus improving the quality of the user gaze monitoring program.

[0195] The time comparison unit 2823 receives the duration of the gaze state (i.e., the continuous gaze period) sent from the gaze state timer 2821 and compares them with a plurality of preset time thresholds stored therein. These multiple thresholds may be, for example, a first period, a second period, a third period, and a fourth period.

[0196] When the time comparison unit 2823 obtains the result that the duration of the gaze state exceeds the first period, the gaze state control unit 282 sends an instruction to the smartphone 1 via the communication unit 27 to prompt the user M to gaze at the vehicle 2.

[0197] When the time comparison unit 2823 determines that the duration of the gaze state exceeds the second period, the gaze state control unit 282 sends an instruction to the driving control unit 2832 to limit the driving speed to below a specified speed, that is, to slow down the vehicle 2. This specified speed is, for example, below 10 km / h, preferably below 8 km / h, more preferably below 6 km / h, and even more preferably below 3 km / h. At the same time, a command instructing the user M to look at the vehicle 2 can also be sent to the smartphone 1 via the communication unit 27.

[0198] When the time comparison unit 2823 obtains the result that the duration of the gaze state exceeds the third period, the gaze state control unit 282 sends an instruction to the driving control unit 2832 to suspend driving, that is, to set the driving speed of the vehicle 2 to 0.

[0199] When the time comparison unit 2823 obtains the result that the gaze state duration exceeds the fourth period, the gaze state control unit 282 sends an instruction to the driving control unit 2832 to stop the automatic exit operation.

[0200] Preferably, the first period, second period, third period, and fourth period are preset to increase sequentially. For example, in a preferred embodiment, the first period can be set to 2 seconds, the second period can be set to 4 seconds, the third period can be set to 6 seconds, and the fourth period can be set to 8 seconds.

[0201] The time comparison unit 2823 also receives the duration of the non-attention state sent from the non-attention state timer 2822 and compares it with a preset predetermined time. When the comparison result shows that the duration of the non-attention state is greater than the predetermined time, the attention state control unit 282 sends an instruction to the smartphone 1 to release the notification prompting the user M to look at the vehicle 2 through communication with the smartphone 1. Furthermore, it instructs the driving control unit 2832 to release a series of restrictive operations performed during the automatic exit process due to the user M's excessive screen viewing time, including: releasing the instruction to slow down the vehicle 2; and releasing the instruction to stop the vehicle 2.

[0202] For example, the specified time can be set to be greater than or equal to the first period and less than the second period. For instance, if the first period can be set to 2 seconds and the second period can be set to 4 seconds, the specified time can be set to 3 seconds. By setting the specified time to be greater than or equal to the first period, it is possible to ensure that the vehicle is instructed to automatically exit the parking space while user M is reliably and continuously monitoring the vehicle. Furthermore, by setting the specified time to be less than the second period, it is possible to quickly respond to user M's visual observation of vehicle 2 and promptly resume the automatic exit operation, resulting in a better user experience.

[0203] Furthermore, while the above embodiments illustrate the use of a vehicle (four-wheeled car) as an example of a mobile device, the invention is not limited to this. For example, it could also be a two-wheeled vehicle, a Segway, or other similar vehicle. Moreover, the concept of this invention is not limited to vehicles and can also be applied to robots, ships, aircraft, and the like, which are equipped with a drive source and can move under the power of that drive source.

[0204] Furthermore, the control method described in the foregoing embodiments can be implemented by executing a pre-prepared control program. This control program is recorded in a computer-readable storage medium and is executed by reading it from the storage medium. Additionally, this control program can be provided in the form of storage on a non-transitory storage medium such as flash memory, or it can be provided via a network such as the Internet. The computer executing this control program can be included in a control device, or in an electronic device such as a smartphone, tablet terminal, or personal computer capable of communicating with the control device, or in a server device capable of communicating with these control devices and electronic devices.

[0205] In addition, the present invention includes at least the following items, wherein the items in parentheses represent the corresponding components or likes in the above embodiments, but are not limited thereto.

[0206] <Option 1>

[0207] A control terminal (smartphone 1), carried by a user, is used for remotely moving a mobile device (vehicle 2) capable of communicating with the control terminal (1), wherein,

[0208] The control terminal (1) includes:

[0209] A gaze information detection unit (11) that detects the user's gaze information toward the control terminal (1);

[0210] The gaze determination unit (12) determines the user's gaze state towards the control terminal (1) based on the gaze information;

[0211] A gaze state control unit (13) that performs control based on the gaze state determined by the gaze state determination unit (12); and

[0212] The notification unit (14) that notifies the user, and,

[0213] The gaze state control unit (13),

[0214] A first determination is made to determine whether the duration of the sustained gaze in the gaze state exceeds a first period.

[0215] When the result of the first determination is that the continuous gaze period exceeds the first period, the notification unit (14) is instructed to issue a notification prompting the user to look at the mobile device (2).

[0216] The control terminal of Option 1, through an intelligent gaze monitoring and feedback mechanism, ensures the convenience of remote operation while proactively guiding users to maintain necessary attention to the vehicle. It can even remind users when they are not looking at the vehicle, thus achieving an organic balance between safety and ease of use.

[0217] <Option 2>

[0218] In the control terminal (1) described in Scheme 1,

[0219] The mobile device (2) includes a driving control unit (2832) for driving control of the mobile device (2), and,

[0220] The gaze state control unit (13),

[0221] A second determination is made to determine whether the sustained gaze period exceeds a second period that is longer than the first period.

[0222] When the result of the second determination is that the continuous gaze period exceeds the second period, the notification unit (14) is instructed to notify the user to look at the mobile device (2), and through communication with the mobile device (2), the driving control unit (2832) of the mobile device (2) is instructed to limit the driving speed of the mobile device (2) to below a specified speed.

[0223] The control terminal of Option 2 improves the level of gaze monitoring and feedback mechanisms, and realizes progressive intervention from warning to speed limit. It ensures that the system can intelligently adjust vehicle behavior when the user is continuously looking at the terminal, and effectively reduce safety risks without affecting normal operation.

[0224] <Option 3>

[0225] In the control terminal (1) described in Scheme 2,

[0226] The gaze state control unit (13),

[0227] A third determination is made to determine whether the sustained gaze period exceeds a third period that is longer than the second period.

[0228] When the result of the third determination is that the continuous gaze period exceeds the third period, the notification unit (14) is instructed to notify the user to look at the mobile device (2), and through communication with the mobile device (2), the driving control unit (2832) of the mobile device (2) is instructed to temporarily stop the mobile device (2).

[0229] The control terminal of Solution 3 further enhances the level of gaze monitoring and feedback mechanisms. When a user continues to look at their mobile phone for a longer period of time, the system will not only issue a warning but also instruct the vehicle to stop temporarily, thus constructing a complete safety guarantee system from reminders and speed limits to stopping, effectively preventing the cumulative effect of safety hazards.

[0230] <Option 4>

[0231] In the control terminal (1) described in Scheme 3,

[0232] The gaze state control unit (13),

[0233] A fourth determination is made to determine whether the sustained gaze period exceeds a fourth period that is longer than the third period.

[0234] When the result of the fourth determination is that the continuous gaze period exceeds the fourth period, the notification unit (14) is instructed to notify the user that the remote movement operation of the mobile device (2) has been suspended, and through communication with the mobile device (2), the driving control unit (2832) of the mobile device (2) is instructed to stop the driving of the mobile device (2).

[0235] The control terminal of Scheme 4 also sets a final security level. When the user stares at the mobile phone for too long, the system will terminate the remote operation, thus constructing a complete security control closed loop from warning, speed limit, pause to complete termination, ensuring absolute security during the automatic remote operation process.

[0236] <Option 5>

[0237] In the control terminal (1) described in Scheme 2,

[0238] The gaze state control unit (13),

[0239] When the result of the second determination is that the continuous gaze period exceeds the second period, the result of the second determination is sent to the mobile device (2).

[0240] <Option 6>

[0241] In the control terminal (1) described in Scheme 2,

[0242] The gaze state control unit (13),

[0243] When the result of the second determination is that the continuous gaze period exceeds the second period, the prescribed limit value of the travel speed of the mobile device (2) is sent to the mobile device (2), wherein the limit value of the travel speed is a prescribed speed value.

[0244] <Option 7>

[0245] As in the control terminal (1) described in Scheme 2,

[0246] The gaze state control unit (13),

[0247] When the result of the second determination is that the continuous gaze period exceeds the second period, the result of the second determination and the prescribed limit value of the travel speed of the mobile device (2) are sent to the mobile device (2), wherein the limit value of the travel speed is a prescribed speed value.

[0248] The control terminals in schemes 5, 6, and 7 achieve coordinated control between the terminal and the vehicle by establishing a direct transmission mechanism for the judgment results between the control terminal and the mobile device, thereby improving the real-time performance and reliability of the vehicle system response.

[0249] <Option 8>

[0250] In the control terminal (1) described in Scheme 3,

[0251] When the result of the third determination is that the continuous gaze period exceeds the third period, the gaze state control unit (13) sends the result of the third determination to the mobile device (2).

[0252] <Option 9>

[0253] In the control terminal (1) described in Scheme 3,

[0254] When the result of the third determination is that the continuous gaze period exceeds the third period, the gaze state control unit (13) sends a predetermined limit value of the travel speed of the mobile device (2) with the speed set to 0 to the mobile device (2).

[0255] <Option 10>

[0256] In the control terminal (1) described in Scheme 3,

[0257] When the result of the third determination is that the continuous gaze period exceeds the third period, the gaze state control unit (13) sends the result of the third determination and the predetermined limit value of the travel speed of the mobile device (2) set to 0 to the mobile device (2).

[0258] The control terminals in schemes 8, 9, and 10 achieve coordinated control between the terminal and the vehicle by establishing a direct transmission mechanism for the judgment results between the control terminal and the mobile device, thereby improving the real-time performance and reliability of the vehicle system response.

[0259] <Option 11>

[0260] In the control terminal (1) described in Scheme 2,

[0261] When the result of the second determination is that the continuous gaze period has not exceeded the second period, and the gaze determination unit (12) determines that the user's non-gaze state on the control terminal (1) has lasted for more than a specified time, the gaze state control unit (13) instructs the driving control unit (2832) of the mobile device (2) to release the restriction on the driving speed of the mobile device (2) through communication with the mobile device (2).

[0262] The control terminal of Scheme 11 automatically removes the speed limit by detecting the duration for which the user shifts their gaze from the terminal to the vehicle, thus realizing an intelligent recovery mechanism based on behavior confirmation, which ensures both safety and provides a smooth operating experience.

[0263] <Option 12>

[0264] In the control terminal (1) described in Scheme 3,

[0265] When the result of the third determination is that the continuous gaze period has not exceeded the third period, and the gaze state determination unit (12) determines that the user's (1) non-gaze state on the control terminal has lasted for more than a specified time, the gaze state control unit (13) instructs the driving control unit (2832) of the mobile device (2) to continue to perform the driving operation of the mobile device (2) through communication with the mobile device (2).

[0266] The control terminal of Scheme 12 automatically lifts the restriction on pausing automatic driving by detecting the duration for which the user shifts their gaze from the terminal to the vehicle, thus realizing an intelligent recovery mechanism based on behavior confirmation, which ensures both safety and provides a smooth operating experience.

[0267] <Option 13>

[0268] In the control terminal (1) of any one of schemes 1 to 4,

[0269] The notifications made by the notification unit (14) are made through at least one of sound, display and vibration.

[0270] The control terminal of Solution 13 enables warning information to be conveyed to users through multiple sensing channels, ensuring the effectiveness and timeliness of the notification.

[0271] <Option 14>

[0272] A control device (28) is a control device (28) that is communicatively connected to a control terminal (1) carried by a user, and remotely moves a mobile device (2) based on instructions from the control terminal (1), wherein,

[0273] The control terminal (1) includes:

[0274] A gaze information detection unit (11) that detects the user's gaze information toward the control terminal (1), and

[0275] The notification unit (14) that notifies the user,

[0276] The control device (28) includes:

[0277] A gaze determination unit (282) that determines the user's gaze state towards the control terminal (1) based on the gaze information obtained through communication with the control terminal (1), and

[0278] The gaze state control unit (282) performs control based on the gaze state determined by the gaze state determination unit (282), and

[0279] The gaze state control unit (282),

[0280] A first determination is made to determine whether the duration of the sustained gaze in the gaze state exceeds a first period.

[0281] When the result of the first determination is that the continuous gaze period exceeds the first period, the notification unit (14) of the control terminal (1) is instructed to notify the user to look at the mobile device (2) through communication with the control terminal (1).

[0282] The control device of Scheme 14 transfers the gaze determination function to the vehicle's control device, and performs centralized processing and control based on the gaze information transmitted from the terminal, thereby reducing the terminal load and improving the overall control reliability.

[0283] <Option 15>

[0284] In the control device (28) described in scheme 14,

[0285] The mobile device (2) includes a driving control unit (2832) for driving control of the mobile device (2).

[0286] The gaze state control unit

[0287] A second determination is made to determine whether the sustained gaze period exceeds a second period that is longer than the first period.

[0288] When the result of the second determination is that the continuous gaze period exceeds the second period, the notification unit (14) of the control terminal (1) is instructed to notify the user to look at the mobile device (2) through communication with the control terminal (1), and the driving control unit (2832) is instructed to limit the driving speed of the mobile device (2) to below a specified speed through communication with the mobile device (2).

[0289] The control device in Scheme 15 improves the level of gaze monitoring and feedback mechanisms, enabling progressive intervention from warnings to speed limits. This ensures that the system can intelligently adjust vehicle behavior when the user is continuously looking at the terminal, effectively reducing safety risks without affecting normal operation.

[0290] <Option 16>

[0291] In the control device (28) described in scheme 15,

[0292] The gaze state control unit

[0293] A third determination is made to determine whether the sustained gaze period exceeds a third period that is longer than the second period.

[0294] When the result of the third determination is that the continuous gaze period exceeds the third period, the notification unit (14) of the control terminal (1) is instructed to notify the user to look at the mobile device (2) through communication with the control terminal, and the driving control unit (2832) is instructed to temporarily stop the mobile device (2) through communication with the mobile device (2).

[0295] The control device in Scheme 16 further enhances the level of gaze monitoring and feedback mechanisms. When a user continues to look at their mobile phone for an extended period of time, the system will not only issue a warning but also instruct the vehicle to temporarily stop, thus constructing a complete safety assurance system from reminders and speed limits to stopping, effectively preventing the cumulative effect of safety hazards.

[0296] <Option 17>

[0297] In the control device (28) described in scheme 16,

[0298] The gaze state control unit

[0299] A fourth determination is made to determine whether the sustained gaze period exceeds a fourth period that is longer than the third period.

[0300] When the result of the fourth determination is that the continuous gaze period exceeds the fourth period, the notification unit (14) of the control terminal (1) is instructed to notify the mobile device (2) to stop remote movement by communicating with the control terminal, and the driving control unit (2832) is instructed to stop the driving of the mobile device (2) by communicating with the mobile device (2).

[0301] The control device in Scheme 17 also sets a final safety level. When the user stares at the mobile phone for too long, the system will terminate the remote operation, thus constructing a complete safety control closed loop from warning, speed limit, pause to complete termination, ensuring absolute safety during the automatic remote operation process.

[0302] <Option 18>

[0303] In the control device (28) described in scheme 16,

[0304] When the result of the second determination is that the continuous gaze period has not exceeded the second period, and the gaze determination unit (281) also determines that the user's non-gaze state on the control terminal (1) has lasted for more than a specified time, the gaze control unit instructs the driving control unit (2832) to release the restriction on the driving speed of the mobile device (2).

[0305] The control device in Scheme 18 automatically removes the speed limit by detecting the duration for which the user shifts their gaze from the terminal to the vehicle, thus realizing an intelligent recovery mechanism based on behavior confirmation, which ensures both safety and provides a smooth operating experience.

[0306] <Option 19>

[0307] In the control device (28) described in scheme 17,

[0308] When the result of the third determination is that the continuous gaze period has not exceeded the third period, and the gaze state determination unit (281) also determines that the user's non-gaze state on the control terminal (1) has lasted for more than a specified time, the driving control unit (2832) instructs the driving control unit (2832) to release the temporary stop of the driving of the mobile device (2).

[0309] The control device of Scheme 19 automatically lifts the restriction on pausing automatic driving by detecting the duration for which the user shifts their gaze from the terminal to the vehicle, thus realizing an intelligent recovery mechanism based on behavior confirmation, which ensures both safety and provides a smooth operating experience.

[0310] <Option 20>

[0311] In the control device (28) of any one of schemes 14 to 17,

[0312] The notifications made by the notification unit (14) are made through at least one of sound, display and vibration.

[0313] The control device of Solution 20 enables warning information to be conveyed to users through multiple sensing channels, ensuring the effectiveness and timeliness of the notification.

[0314] <Option 21>

[0315] A control system includes a user-carried control terminal (1) and a control device (28) communicatively connected to the control terminal (1) for remotely moving a mobile device (2), wherein...

[0316] The control terminal (1) includes:

[0317] A gaze information detection unit (11) that detects the user's gaze towards the control terminal (1)

[0318] The gaze determination unit (12) determines the user's gaze state towards the control terminal (1) based on the gaze information.

[0319] A gaze state control unit (13) that performs control based on the gaze state determined by the gaze state determination unit (12), and

[0320] The notification unit (14) that notifies the user, and

[0321] The gaze state control unit (13),

[0322] A first determination is made to determine whether the duration of the sustained gaze in the gaze state exceeds a first period.

[0323] When the result of the first determination is that the continuous gaze period exceeds the first period, the notification unit (14) is instructed to issue a notification prompting the user to look at the mobile device (2).

[0324] The control system of Scheme 21 integrates the control terminal of the mobile phone and the control device of the vehicle to build a complete remote control system architecture, realizing the systematization of line-of-sight monitoring, status determination and notification feedback, and providing comprehensive technical support for the safety management of remote operation.

[0325] <Option 22>

[0326] A control system includes a user-carried control terminal (1) and a control device (28) communicatively connected to the control terminal (1) for remotely moving a mobile device (2), wherein...

[0327] The control terminal (1) includes:

[0328] A gaze information detection unit (11) that detects the user's gaze information toward the control terminal (1), and

[0329] The notification unit (14) that notifies the user,

[0330] The control device (28) includes:

[0331] A gaze determination unit (281) that determines the user's gaze state towards the control terminal (1) based on the gaze information obtained through communication with the control terminal (1), and

[0332] A gaze state control unit (282) performs control based on the gaze state determined by the gaze state determination unit (281), and...

[0333] The gaze state control unit (282),

[0334] A first determination is made to determine whether the duration of the sustained gaze in the gaze state exceeds a first period.

[0335] When the result of the first determination is that the continuous gaze period exceeds the first period, the notification unit (14) of the control terminal (1) is instructed to notify the user to look at the mobile device (2) through communication with the control terminal (1).

[0336] The control system of Scheme 22 adjusts the division of labor between the control terminal via mobile phone and the vehicle control device. While the mobile phone focuses on information collection and notification feedback, the core judgment and control functions are concentrated in the vehicle control device, achieving more efficient and reliable remote control safety management.

Claims

1. A control terminal, carried by a user, for remotely moving a mobile device capable of communicating with the control terminal, wherein, The control terminal includes: A gaze detection unit that detects the user's gaze at the control terminal; A gaze determination unit that determines the user's gaze state toward the control terminal based on the gaze information; A gaze state control unit that performs control based on the gaze state determined by the gaze state determination unit; as well as The notification department that notifies the user, and, The gaze state control unit A first determination is made to determine whether the duration of the sustained gaze in the gaze state exceeds a first period. When the result of the first determination is that the continuous gaze period exceeds the first period, the notification unit is instructed to issue a notification prompting the user to look at the mobile device.

2. The control terminal as described in claim 1, wherein, The mobile device includes a driving control unit for driving control of the mobile device, and, The gaze state control unit A second determination is made to determine whether the sustained gaze period exceeds a second period that is longer than the first period. When the result of the second determination is that the continuous gaze period exceeds the second period, the notification unit is instructed to notify the user to look at the mobile device, and, through communication with the mobile device, the driving control unit of the mobile device is instructed to limit the driving speed of the mobile device to below a specified speed.

3. The control terminal as described in claim 2, wherein, The gaze state control unit A third determination is made to determine whether the sustained gaze period exceeds a third period that is longer than the second period. When the result of the third determination is that the continuous gaze period exceeds the third period, the notification unit is instructed to notify the user to look at the mobile device, and through communication with the mobile device, the driving control unit of the mobile device is instructed to temporarily stop the mobile device from moving.

4. The control terminal as described in claim 3, wherein, The gaze state control unit A fourth determination is made to determine whether the sustained gaze period exceeds a fourth period that is longer than the third period. When the result of the fourth determination is that the continuous gaze period exceeds the fourth period, the notification unit is instructed to notify the user that the remote movement operation of the mobile device has been suspended, and through communication with the mobile device, the driving control unit of the mobile device is instructed to stop the mobile device from driving.

5. The control terminal as described in claim 2, wherein, The gaze state control unit When the result of the second determination is that the continuous gaze period exceeds the second period, the result of the second determination is sent to the mobile device.

6. The control terminal as described in claim 2, wherein, The gaze state control unit When the result of the second determination is that the continuous gaze period exceeds the second period, a predetermined limit value for the travel speed of the mobile device is sent to the mobile device, wherein the limit value for the travel speed is a predetermined speed value.

7. The control terminal as described in claim 2, wherein, The gaze state control unit When the result of the second determination is that the continuous gaze period exceeds the second period, the result of the second determination and the predetermined limit value of the travel speed of the mobile device are sent to the mobile device, wherein the limit value of the travel speed is a predetermined speed value.

8. The control terminal as described in claim 3, wherein, When the result of the third determination is that the continuous gaze period exceeds the third period, the gaze state control unit sends the result of the third determination to the mobile device.

9. The control terminal as described in claim 3, wherein, When the result of the third determination is that the continuous gaze period exceeds the third period, the gaze state control unit sends a predetermined limit value of the travel speed of the mobile device, which is set to 0, to the mobile device.

10. The control terminal as described in claim 3, wherein, When the result of the third determination is that the continuous gaze period exceeds the third period, the gaze state control unit sends the result of the third determination and a predetermined limit value of the travel speed of the mobile device, which is set to 0, to the mobile device.

11. The control terminal as described in claim 2, wherein, When the result of the second determination is that the continuous gaze period has not exceeded the second period, and the gaze state determination unit determines that the user's non-gaze state on the control terminal has lasted for more than a specified time, the gaze state control unit instructs the driving control unit of the mobile device to release the speed limit on the mobile device through communication with the mobile device.

12. The control terminal as described in claim 3, wherein, When the result of the third determination is that the continuous gaze period has not exceeded the third period, and the gaze state determination unit determines that the user's non-gaze state on the control terminal has lasted for more than a specified time, the gaze state control unit instructs the driving control unit of the mobile device to continue to perform the driving operation of the mobile device through communication with the mobile device.

13. The control terminal as described in any one of claims 1 to 4, wherein, The notifications issued by the notification unit are delivered via at least one of sound, display, and vibration.

14. A control device that is communicatively connected to a control terminal carried by a user, and that enables a mobile device to move remotely based on instructions from the control terminal, wherein... The control terminal includes: A gaze detection unit that detects the user's gaze at the control terminal, and The notification department that notifies the user. The control device includes: A gaze determination unit that determines the user's gaze state towards the control terminal based on the gaze information obtained through communication with the control terminal, and... A gaze state control unit that performs control based on the gaze state determined by the gaze state determination unit, and the gaze state control unit, A first determination is made to determine whether the duration of the sustained gaze in the gaze state exceeds a first period. When the result of the first determination is that the continuous gaze period exceeds the first period, the notification unit of the control terminal is instructed to notify the user to look at the mobile device through communication with the control terminal.

15. The control device as claimed in claim 14, wherein, The mobile device includes a driving control unit for driving control of the mobile device. The gaze state control unit A second determination is made to determine whether the sustained gaze period exceeds a second period that is longer than the first period. When the result of the second determination is that the continuous gaze period exceeds the second period, the notification unit of the control terminal is instructed to notify the user to look at the mobile device through communication with the control terminal, and the driving control unit is instructed to limit the driving speed of the mobile device to below a specified speed through communication with the mobile device.

16. The control device as claimed in claim 15, wherein, The gaze state control unit A third determination is made to determine whether the sustained gaze period exceeds a third period that is longer than the second period. When the result of the third determination is that the continuous gaze period exceeds the third period, the notification unit of the control terminal is instructed to notify the user to look at the mobile device through communication with the control terminal, and the driving control unit is instructed to temporarily stop the mobile device through communication with the mobile device.

17. The control device as claimed in claim 16, wherein, The gaze state control unit A fourth determination is made to determine whether the sustained gaze period exceeds a fourth period that is longer than the third period. When the result of the fourth determination is that the continuous gaze period exceeds the fourth period, the notification unit of the control terminal is instructed to notify the mobile device to stop remote movement by communicating with the control terminal, and the driving control unit is instructed to stop the mobile device by communicating with the mobile device.

18. The control device as claimed in claim 16, wherein, When the result of the second determination is that the continuous gaze period has not exceeded the second period, and the gaze state determination unit also determines that the user's non-gaze state on the control terminal has lasted for more than a specified time, the driving control unit instructs the driving control unit to release the speed limit on the mobile device.

19. The control device as claimed in claim 17, wherein, When the result of the third determination is that the continuous gaze period has not exceeded the third period, and the gaze state determination unit determines that the user's non-gaze state on the control terminal has lasted for more than a specified time, the driving control unit instructs the driving control unit to release the temporary stop of the mobile device.

20. The control device according to any one of claims 14 to 17, wherein, The notifications issued by the notification unit are delivered via at least one of sound, display, and vibration.

21. A control system comprising a user-carried control terminal and a control device communicatively connected to the control terminal for remotely moving a mobile device, wherein, The control terminal includes: A gaze information detection unit that detects the user's gaze at the control terminal. The gaze determination unit determines the user's gaze state towards the control terminal based on the gaze information. A gaze state control unit that performs control based on the gaze state determined by the gaze state determination unit, and The notification department that notifies the user, and The gaze state control unit A first determination is made to determine whether the duration of the sustained gaze in the gaze state exceeds a first period. When the result of the first determination is that the continuous gaze period exceeds the first period, the notification unit is instructed to issue a notification prompting the user to look at the mobile device.

22. A control system comprising a user-carried control terminal and a control device communicatively connected to the control terminal for enabling remote movement of a mobile device, wherein, The control terminal includes: A gaze detection unit that detects the user's gaze at the control terminal, and The notification department that notifies the user. The control device includes: A gaze determination unit that determines the user's gaze state towards the control terminal based on the gaze information obtained through communication with the control terminal, and... A gaze state control unit that performs control based on the gaze state determined by the gaze state determination unit, and, The gaze state control unit A first determination is made to determine whether the duration of the sustained gaze in the gaze state exceeds a first period. When the result of the first determination is that the continuous gaze period exceeds the first period, the notification unit of the control terminal is instructed to notify the user to look at the mobile device through communication with the control terminal.

Images