Unmanned mobile vehicle control system, information processing device, program, and information processing method

Abstract

This system provides a control system that allows for easy configuration of information necessary for operating a wide variety of unmanned mobile vehicles. [Solution] In one aspect of the present disclosure, an unmanned mobile vehicle control system is provided, comprising: an unmanned mobile vehicle equipped with a camera; a ride-on control device for a user to operate the unmanned mobile vehicle; a video output device that outputs video captured by the camera; and a tablet terminal that receives input from the user of information necessary for operating the unmanned mobile vehicle.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to an unmanned mobile vehicle control system, an information processing device, a program, and an information processing method.

Background Art

[0002] A control system for an unmanned mobile vehicle equipped with an on-vehicle control device is known.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] On the other hand, in the conventional control system, information necessary for controlling the unmanned mobile vehicle had to be set for each actual machine of the unmanned mobile vehicle, and there was room for improvement from the viewpoint of versatility.

[0005] The problem to be solved by the present disclosure is to provide a control system capable of easily setting information necessary for controlling various actual machines of unmanned mobile vehicles.

Means for Solving the Problems

[0006] As one aspect of the present disclosure, there is provided an unmanned mobile vehicle control system including an unmanned mobile vehicle equipped with a photographing device, an on-vehicle control device for a user to board and control the unmanned mobile vehicle, a video output device for outputting video captured by the photographing device, and a tablet terminal for receiving an input of information necessary for controlling the unmanned mobile vehicle from the user.

Brief Description of the Drawings

[0008] <1. Embodiments> The following describes an unmanned mobile vehicle control system 100 as an embodiment of the present disclosure with reference to the drawings. In this specification and in each drawing, elements similar to those already described are denoted by the same reference numerals, and detailed descriptions are not repeated. In this disclosure, "user" refers to a person (operator) who controls the unmanned mobile vehicle 10 using the unmanned mobile vehicle control system 100. "Actual vehicle" means an automobile that is actually driven by a person. "Actual machine" means each individual machine of the unmanned mobile vehicle 10. "Unmanned" means that the operator is not on board the mobile vehicle, regardless of whether there are passengers who are not involved in controlling the mobile vehicle. For example, even if a person is on board the driver's seat (or other seat) of the mobile vehicle, if they are not involved in controlling the mobile vehicle, the mobile vehicle is included in the definition of an unmanned mobile vehicle.

[0009] (1.1. Overview of the Unmanned Mobile Vehicle Control System 100) As shown in Figure 1, the unmanned mobile vehicle control system 100 is a system for a user to control a small unmanned mobile vehicle 10 with a sense of realism, as if driving a real vehicle. It consists of an unmanned mobile vehicle 10 equipped with a camera, a ride-on control device 70 (hereinafter also simply referred to as the control device 70) that receives user input, AR glasses 41 that output video from the camera, and a tablet terminal 60 that receives input of information necessary for control. Control signals and video signals are transmitted bidirectionally via wireless or wired communication, enabling remote control that is in line with the actual driving environment.

[0010] In this embodiment, the unmanned mobile vehicle 10 is a small (for example, 1 / 10 the size of a real car) four-wheeled vehicle that can be driven by remote control, and is equipped with a video transmitter 20, an on-board camera 21 as a filming device, a signal transceiver 30, a steering servo motor 31, an ESC motor 32 for driving, and a camera servo motor 33. The unmanned mobile vehicle 10 receives control signals for operation transmitted from a tablet terminal 60 via the signal transceiver 30 and controls the operation of each mechanism. This enables real-time driving and filming in the real world.

[0011] The video transmitter 20 is a communication device for transmitting video data obtained from an on-board camera 21 mounted on an unmanned mobile vehicle 10 to a video receiver 40 installed at a predetermined location via wireless communication. For example, wireless communication in the 5.8GHz band may be used for video transmission. This allows for both real-time performance and high image quality through high-speed communication, providing the operator with a high level of realism.

[0012] The in-vehicle camera 21 is a camera installed inside the unmanned mobile body 10 and captures the surrounding environment of the mobile body. The captured video is transmitted via the video transmitter 20 and displayed on the AR glasses 41 worn by the user. The orientation of the in-vehicle camera 21 will be described in detail later.

[0013] The in-vehicle signal transceiver 30 receives various control signals transmitted via wireless communication from the signal transceiver 50 mounted on the tablet terminal 60. The various control signals are used for drive control of a steering servo motor 31 and the like mounted on the unmanned moving body 10. Further, sensor data and operation information on the vehicle side can also be transmitted, enabling real-time control through two-way communication with the control side.

[0014] The steering servo motor 31 is an actuator for controlling the angle of the drive wheels of the unmanned moving body 10. When the user operates the steering controller 71, a signal corresponding to the operation angle is transmitted via the tablet terminal 60, and steering is performed. Thereby, driving control similar to that of a real vehicle becomes possible.

[0015] The ESC motor 32 is an electric motor provided as the driving force of the unmanned moving body 10 and incorporates an ESC (Electronic Speed Controller) for speed control. A throttle signal corresponding to the depression amount of an accelerator pedal 72c and a brake pedal 72b, which will be described later, is converted and transmitted by the tablet terminal 60, and the acceleration and deceleration of the unmanned moving body 10 are controlled through the ESC motor 32.

[0016] The camera servo motor 33 is a drive mechanism that can change the orientation of the in-vehicle camera 21 and is driven in conjunction with the movement (left and right rotation) of the user's head acquired from the head gyro sensor 73. Thereby, it becomes possible to adjust the camera viewpoint in the sense of the operator moving their line of sight, improving the operational intuitiveness during operation.

[0017] The video receiver 40 receives real-time video signals wirelessly transmitted from the video transmitter 20 and transmits the video to the AR glasses 41 via an output interface such as HDMI (registered trademark, hereinafter abbreviated). By achieving high-precision reception performance, video transmission with less delay and noise can be realized even during remote operation, ensuring the visibility of the operator.

[0018] The AR glasses 41 are a head-mounted display that allows the operator to transparently view the real world while superimposing the captured video footage received from the unmanned mobile device 10 within the operator's field of view. Because the operator can perceive the real world and the captured video footage together, it provides a sense of immersion and visual awareness of the surrounding environment during operation. Furthermore, by working in conjunction with the head gyro sensor 73 described later, it becomes possible to control the field of view of the captured video footage according to the operator's gaze.

[0019] The signal transceiver 50 is attached to the tablet terminal 60 and wirelessly transmits control signals output from the onboard control device 70 to the vehicle-mounted signal transceiver 30. For example, it transmits multiple control signals in real time, such as steering, acceleration / deceleration, and camera control, using 2.4GHz band communication. It also supports bidirectional communication and can receive information from the vehicle and transmit it to the tablet terminal 60. For example, the signal transceiver 50 is implemented as a stick-type wireless communication module with a USB (Universal Serial Bus) interface. As a result, it has a structure that can be directly attached to the tablet terminal, making it versatile and easy to attach to various terminal devices, and offering excellent expandability and portability.

[0020] The tablet terminal 60 is an information processing device equipped with relay and control functions that receives various operation inputs from the onboard control system 70, converts them into an appropriate data format, and transmits them to the unmanned mobile vehicle 10 via the signal transceiver 50. By installing a dedicated application, the tablet terminal 60 allows users to input settings such as throttle, steering, and camera operation on the screen. Furthermore, the tablet terminal 60 has a touch panel that supports touch operation, allowing users to intuitively and quickly input necessary operation information and settings with their fingertips.

[0021] The onboard control system 70 is a device that the user actually boards and operates, and is equipped with a steering controller 71, pedal controllers 72, a head gyro sensor 73, and the like. The user can operate the unmanned mobile vehicle 10 in a driving posture similar to that of a real vehicle, and each operation is captured by a tablet terminal 60 and transmitted as a control signal to the remote unmanned mobile vehicle 10.

[0022] The steering controller 71 is a circular input device that receives the user's steering wheel input, and the rotation angle information is transmitted to the tablet terminal 60. Furthermore, auxiliary operations such as turning on the headlights can be performed using the operation knob 71a located on the steering wheel, enabling complex operations. Further details about the steering controller 71 will be described later.

[0023] The pedal controller 72 is a foot pedal type control input device that includes an accelerator pedal 72c, a brake pedal 72b, and a clutch pedal 72a. The amount of each pedal pressed is transmitted as a control signal to the steering controller 71, and output from the steering controller 71 to the tablet terminal 60. These signals are converted by the tablet terminal 60 and used to control the ESC motor 32 and other components of the unmanned mobile vehicle 10.

[0024] The clutch pedal 72a is an operating mechanism provided to simulate the operation of a manual vehicle, and can provide a sense of operation that is conscious of virtual gear changes. Signals based on clutch operation are converted by the tablet terminal 60 and transmitted to the unmanned mobile body 10 via the signal transceiver 50, and are reflected in the driving operation and control algorithm. Note that the clutch pedal 72a is not required to be used.

[0025] The brake pedal 72b is an input device for controlling the deceleration and stopping of the unmanned mobile body 10, and generates a brake signal according to the amount of pressure applied to the pedal. The generated signal is converted by the tablet terminal 60 and transmitted to the unmanned mobile body 10 via the signal transceiver 50, and brake control is performed by the ESC motor 32.

[0026] The accelerator pedal 72c is an input device for accelerating the unmanned mobile vehicle 10, and generates a throttle signal corresponding to the amount it is pressed. The generated signal is converted by the tablet terminal 60 and transmitted to the unmanned mobile vehicle 10 via the signal transceiver 50, and acceleration control is performed by the ESC motor 32.

[0027] The head gyro sensor 73 is an inertial sensor that, for example, detects the orientation of the operator's head (yaw, pitch, roll) in three axes, and is used as input for controlling the camera's viewpoint. The data detected by the head gyro sensor 73 is converted by the tablet terminal 60 and transmitted to the unmanned mobile body 10 via the signal transceiver 50, and is reflected in the driving of the camera servo motor 33. The log data detected by the head gyro sensor 73 can be periodically refreshed, for example, 10 times per second (at 100ms intervals), to improve the accuracy of viewpoint tracking.

[0028] The headgear 74 is a device worn on the user's head, with a head gyro sensor 73 attached to its top. This allows the head gyro sensor 73 to detect the orientation of the user's head. The headgear 74 may also be headphones capable of outputting predetermined sounds.

[0029] The passenger compartment 75 has a seat structure for the driver to sit and operate, and is arranged in combination with control systems such as a steering controller 71 and pedal controllers 72. Furthermore, in order to enhance the sense of realism, the arrangement balance may be designed to naturally correspond with visual information and physical movements during operation. This allows the driver to obtain a driving sensation closer to that of a real vehicle.

[0030] The adapter 80 is an interface conversion device that mediates the connection between the tablet terminal 60 and each device (AR glasses 41, signal transceiver 50, etc.), and supports different connection standards such as USB and HDMI. For example, the adapter 80 may be configured as a hub (multiport device) with multiple ports, making it possible to connect multiple peripheral devices to a single tablet terminal 60 simultaneously. This ensures a flexible connection configuration and signal consistency for the entire system.

[0031] (1.2. Details of each function) As shown in Figure 3, the steering controller 71 is a circular steering input device used by the user when operating the unmanned mobile vehicle 10. It has a general steering wheel shape but incorporates multiple operation buttons and dials. A rotating axis for turning is provided in the center, and the steering angle input is transmitted in real time to the steering servo motor 31 of the unmanned mobile vehicle 10. As an example, the steering controller 71 has an operation knob 71a, an upper button 71b, a paddle shifter 71c, an operation key 71d, a directional pad 71e, and a lower button 71f.

[0032] The control knob 71a is a rotary dial located on the upper right of the steering controller 71, and is responsible for switching the lighting functions. For example, each time the control knob 71a is rotated to the right, the fog lamps, low beams, and high beams are illuminated in stages, and when rotated to the left, they are turned off.

[0033] The upper buttons 71b are a pair of push-buttons located below the control knob 71a, and they have the function of setting the steering response. Pressing the right button increases the response level in stages, and pressing the left button decreases it. This makes it possible to optimize the steering angle response according to the driver's feel.

[0034] The Paddle Shift 71c is a lever-type switch located on the left and right sides behind the steering wheel, used to adjust throttle response. For example, pulling the right paddle increases throttle response by 10%, while pulling the left paddle decreases it by 10%. This highly functional physical switch allows for immediate adjustment of acceleration characteristics while driving.

[0035] The 71d control keys are a group of buttons arranged in a four-way configuration, controlling individual vehicle functions. For example, the triangular button turns the hazard lights on and off, the round button controls the flashing (high beam) operation, the X button controls the center correction of the head tracking function (a function that controls the direction of the on-board camera of the unmanned mobile vehicle in conjunction with head movements, as described later), and the square button controls a separate auxiliary light system.

[0036] The 71e directional pad is a four-way switch located on the left side of the steering wheel, primarily used to control the turn signals. Pressing the left or right key will activate either the left or right turn signal, respectively.

[0037] The lower buttons 71f are a pair of buttons located on the left and right sides of the lower part of the steering controller 71, and are used to individually illuminate the turn signal for the corresponding direction. Their intuitive press operation allows for immediate direction indication, thus helping to prevent accidental operation.

[0038] Furthermore, the settings for each component 71a to 71f provided in the steering controller 71 are arbitrary, and it is not necessary to install all of them. In addition, the function assignments and operation of each component can be freely changed through software settings, and the configuration is such that it can be flexibly adjusted later and optimized for each user. In this way, only the necessary components can be selectively equipped according to the operator's purpose of use, operability requirements, system configuration, or type of mobile object.

[0039] As shown in Figure 4A, the user wears a headgear 74 on their head, and a head gyro sensor 73 is attached to the top of this headgear 74 to detect head movement. The head gyro sensor 73 detects the rotational movement of the user's head, i.e., the yaw angle, pitch angle, and roll angle, in real time and transmits information about these angles to the unmanned mobile body 10 via a signal transceiver 50 attached to a tablet terminal 60.

[0040] As shown in Figure 4B, when the user's head rotates from side to side, the camera servo motor 33 mounted on the unmanned mobile vehicle 10 operates based on the signal from the head gyro sensor 73, controlling the orientation of the onboard camera 21 according to the direction of rotation. This changes the shooting viewpoint in conjunction with the movement of the operator's head, realizing an immersive experience and intuitive field of view control that is close to actual driving. This improves visibility and operational accuracy during remote control.

[0041] Referring to Figure 5, the display screen 61 shown on the tablet terminal 60 will be described. The tablet terminal 60 is an information processing device for managing the settings and status when remotely controlling the unmanned mobile unit 10, and provides the user with the setting of information necessary for control (control settings) and real-time status confirmation through the display screen 61. A dedicated application (program) is installed on the tablet terminal 60 and processes information related to the control of the unmanned mobile unit 10, such as inputting control settings, visualizing the operating status, and sending signals.

[0042] The display screen 61 is a user interface for operation mounted on the tablet terminal 60, and includes three areas: a control display area 62, a status display area 63, and a settings display area 64. Each area is equipped with buttons, graphs, input fields, etc., ensuring both visibility and ease of operation. Inputting control information to the display screen 61 is primarily intended to be done before operating the unmanned mobile unit 10, but settings can also be changed during operation.

[0043] The control display area 62 is an area for setting basic parameters related to the driving control of the unmanned mobile body 10, and allows for setting the throttle and steering direction, selecting the feedback mode, adjusting the sensor orientation, and even starting or stopping applications. For example, the control display area 62 includes a start / stop button 62a, a direction setting area 62b, a steering force feedback setting area 62c, a sensor setting area 62d, and an acceleration display area 62e.

[0044] The start / stop button 62a is an operation button for controlling the activation and deactivation of the application for operating the unmanned mobile unit 10. It activates the entire system when operation begins and safely and quickly stops it when operation ends. To prevent accidental operation, it is provided as a highly visible and easy-to-operate circular icon.

[0045] The direction setting field 62b is a UI element for setting the rotation direction for the throttle and steering, and can be set according to the structure of the unmanned mobile vehicle 10 and the specifications of the servo motor used. In particular, the rotation direction may differ depending on the manufacturer of remote-controlled cars, so enabling this setting allows the unmanned mobile vehicle control system 100 to be applied to a wide variety of actual machines.

[0046] The steering force feedback setting section 62c is where you adjust the force feedback (FFB) that applies a return force to steering inputs. By adjusting the feedback calculation method (e.g., angle-based, acceleration-based), output limit, various coefficients, etc., you can pursue a more realistic steering feel.

[0047] The sensor setting area 62d is a setting area for defining the installation direction of the acceleration sensor mounted on the unmanned mobile body 10 in software. Defining the XYZ axes relative to the physical mounting angle of the acceleration sensor enables proper direction recognition and contributes to improving the accuracy of acceleration detection. As for the acceleration sensor, a strain gauge sensor may be fixed to the actual device by soldering, or a detachable acceleration sensor may be used.

[0048] The acceleration display area 62e is an area that displays acceleration information detected by the acceleration sensor mounted on the unmanned mobile body 10 in real time. By visualizing vector components such as the vertical and horizontal axes numerically or graphically, the operator can intuitively grasp behaviors such as acceleration, deceleration, and impacts.

[0049] The status display area 63 is an area that displays the driving status of the unmanned mobile unit 10 and the operating status of various systems in real time, and is configured so that the operator can immediately grasp the current behavior of the mobile unit and changes in the surrounding environment. Specifically, visibility is ensured by displaying information on the driving system, sensor system, and lighting system separately. As an example, the status display area 63 has an operation status display section 63a, a sensor status display section 63b, and a light status indicator 63c.

[0050] The operation status display area 63a is an area that displays the current values ​​of each control system in response to the operator's input, and the brake, throttle, steering angle, and camera direction are displayed as percentages or numerical values. This allows the operator to quantitatively understand the current operation output of the unmanned mobile device 10 and immediately confirm whether the operation is being reflected as intended.

[0051] Sensor status display area 63b is an area that displays the X, Y, and Z axis values ​​set for the acceleration sensor mounted on the unmanned mobile vehicle 10 as numerical values. This is useful for analyzing driving behavior and correcting steering to understand acceleration changes and tilt conditions during travel.

[0052] The light status indicator 63c is an indicator (signal) that visually displays the operating status of the lights, such as the headlights and brake lights, mounted on the unmanned mobile unit 10. Icons or color coding are used so that the on / off status can be seen at a glance, contributing to understanding the status of the unmanned mobile unit 10.

[0053] The setting display area 64 is an area for adjusting the response characteristics of the unmanned mobile body 10 to the operator's input, and the operational response of each mechanism such as the brakes, throttle, steering, and camera can be optimized to the user's preference. For example, the setting display area 64 has a brake response setting area 64a, a throttle response setting area 64b, a steering response setting area 64c, and a camera response setting area 64d.

[0054] The brake response setting section 64a is a setting item for adjusting the deceleration and response rise when the operator presses the brake pedal 72b, and multiple modes can be set, from smooth deceleration to sudden braking. This enables fine-tuned brake control according to the operating environment and the characteristics of the actual vehicle.

[0055] The throttle response setting section 64b is the area for setting the response characteristics related to throttle output during acceleration. For example, "Linear Mode" allows for acceleration proportional to the input, while "Easing Mode" allows for a smooth response with gradually increasing output. In this way, adjustments can be made according to the operator's feel.

[0056] The steering response setting section 64c adjusts the steering angle response to steering wheel input, allowing for center correction (so-called trim adjustment) and dead zone (play) settings. This corrects minute deviations in steering wheel operation, improving straight-line stability and steering angle accuracy.

[0057] The camera response setting area 64d is where you set the responsiveness and resolution to operations that adjust the orientation of the onboard camera 21 (i.e., the rotation of the driver's head). For example, you can set the unit amount (resolution) of the head rotation angle in response to the operation as the camera step, and by setting a larger step value in environments where fine changes in the viewpoint are necessary, precise control of the view becomes possible.

[0058] (1.3. Hardware configuration of information processing equipment) Referring to Figure 6, the hardware configuration of the information processing device used as a tablet terminal 60 will be described. As an example, the information processing device is implemented using the computer 90 shown in Figure 6. The computer 90 may include a CPU 91, ROM 92, RAM 93, storage 94, input interface 95, output interface 96, and communication interface 97.

[0059] The CPU 91 functions as a control unit that executes processing. Specifically, the CPU 91 uses the RAM 93 as work memory and executes programs as applications stored in at least one of the ROM 92 or storage 94. During program execution, the CPU 91 controls each component via the system bus 98 and performs various processes.

[0060] ROM92 functions as part of the memory unit and stores programs that control the operation of computer 90. ROM92 contains the programs necessary for computer 90 to perform each of the processes described above. RAM93 functions as a memory area where the programs stored in ROM92 are deployed.

[0061] Storage 94 functions as part of the memory unit and stores data necessary for program execution and data obtained through program execution. Storage 94 includes one or more selected from Hard Disk Drives (HDDs) and Solid State Drives (SSDs).

[0062] The input interface (I / F) 95 can connect the computer 90 and the input device 95a. The input interface 95 is, for example, a serial bus interface such as USB. The CPU 91 can read various data from the input device 95a via the input interface 95.

[0063] The output interface (I / F) 96 can connect the computer 90 to the output device 96a. The output interface 96 is a video output interface such as a Digital Visual Interface (DVI) or a High-Definition Multimedia Interface (HDMI). The CPU 91 can send data to the output device 96a via the output interface 96 and cause the output device 96a to output data.

[0064] Input device 95a is an example of input means and includes a touch panel and a microphone (voice input). Output device 96a is an example of output means and includes one or more selected from a display, projector, printer, and speaker.

[0065] The communication interface (I / F) 97 allows the computer 90 to connect with an external server 97a located outside the computer 90. The communication interface 97 is, for example, a network card such as a LAN card. The CPU 91 can read various data from the external server 97a via the communication interface 97.

[0066] Furthermore, each process performed by the tablet terminal 60 in this disclosure may be implemented by a single computer 90, or by the cooperation of multiple computers 90.

[0067] The processing of the various data described above may be recorded as a program that can be executed by a computer on a magnetic disk (flexible disk and hard disk, etc.), an optical disk (CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, DVD±RW, etc.), a semiconductor memory, or another non-transitory computer-readable storage medium.

[0068] For example, information recorded on a recording medium can be read by a computer (or embedded system). The recording format (storage format) of the recording medium is arbitrary. For example, a computer reads a program from the recording medium and, based on this program, causes the processor to execute the instructions written in the program. In a computer, program acquisition (or reading) may be performed via a network.

[0069] (1.4.Summary) As described above, the unmanned mobile vehicle control system 100 in this disclosure comprises an unmanned mobile vehicle 10 equipped with an on-board camera 21, a ride-on control device 70 for a user to ride and control the unmanned mobile vehicle 10, a video output device that outputs video captured by the on-board camera 21, and a tablet terminal 60 that receives input from the user of information necessary for controlling the unmanned mobile vehicle 10. With this configuration, it is possible to easily set the information necessary for control for a wide variety of actual unmanned mobile vehicles, and a control system with excellent versatility and expandability can be provided.

[0070] Furthermore, the in-vehicle camera 21 is installed inside the unmanned mobile vehicle 10. This allows for the output of video from inside the vehicle, providing the user with a sense of realism and immersion that closely resembles the feeling of driving a real car.

[0071] Furthermore, the video output device is an AR glasses 41 that can be worn on the user's head. This allows the user to see through to the real world while superimposing images from the vehicle's onboard camera within their field of view, thus enabling smooth operation of the passenger compartment 75.

[0072] Furthermore, the unmanned mobile vehicle control system 100 is further equipped with a headgear 74 that is worn on the user's head. The headgear 74 includes a head gyro sensor 73 for detecting the orientation of the user's head, and the unmanned mobile vehicle 10 includes a camera servo motor 33 that can change the orientation of the onboard camera 21. The camera servo motor 33 changes the orientation of the onboard camera 21 in accordance with the detected orientation of the user's head. This allows the camera's field of view to be changed in conjunction with the operator's gaze movement, enabling intuitive view control and improving realism and operational accuracy.

[0073] Furthermore, the tablet terminal 60 performs conversion processing for information necessary for control, enabling wireless transmission from the onboard control device 70 to the unmanned mobile vehicle 10. This allows for the integrated conversion and transmission of various operation information obtained by the onboard control device 70, and enables wireless transmission of control signals to different unmanned mobile vehicle 10 units.

[0074] Furthermore, the unmanned mobile vehicle 10 includes at least one of the headlights and brake lights, and the onboard control device 70 accepts operation of at least one of the headlights and brake lights. This makes it possible to remotely control the display functions such as the lighting and indicators of the unmanned mobile vehicle 10, thereby providing operability similar to that of a real vehicle.

[0075] <2. Second Embodiment> The following describes a second embodiment of this disclosure, focusing on the differences from the first embodiment. As shown in Figure 7, in the second embodiment, the unmanned mobile vehicle control system 100 further includes an AI PC (Personal Computer) 81 as an information processing device, and the driving control module 83 of the AI ​​PC 81 enables driving control of the unmanned mobile vehicle 10. Specifically, the AI ​​PC 81 generates control commands in the driving control module 83 based on information obtained by the image analysis module 82, and these commands are transmitted from the signal transceiver 50 to the unmanned mobile vehicle 10 via the tablet terminal 60, thereby realizing autonomous driving control as an AI car.

[0076] In the second embodiment, a GNSS (Global Navigation Satellite System) 34 and a distance sensor 35 are added to the unmanned mobile vehicle 10. The GNSS receives signals from multiple satellites to acquire highly accurate positional information of the unmanned mobile vehicle 10, such as its latitude, longitude, and altitude. The acquired positional information is transmitted to the AI ​​PC 81 via the tablet terminal 60 and used as basic data for recording the driving trajectory and driving control. This improves the accuracy of automatic driving control.

[0077] The distance sensor 35 is a sensor for measuring the distance between the unmanned mobile body 10 and the surrounding environment, and detects the proximity to obstacles in front of or to the side in real time. Distance sensors generally use ranging methods such as infrared, ultrasonic, or laser, and are utilized as input information for obstacle avoidance and route selection algorithms in the AI ​​PC 81. This improves autonomous driving performance in a variety of environments.

[0078] The AI ​​PC 81 includes an image analysis module 82 and a driving control module 83. It uses video from the onboard camera 21 to recognize road shapes and obstacles, and determines steering angle and throttle output based on this. These processes are based on machine learning models, and control accuracy continuously improves as driving data is accumulated.

[0079] The image analysis module 82 receives video transmitted from the in-vehicle camera 21 as input and performs tasks such as object recognition, direction of travel determination, and line detection. A machine learning-based object detection algorithm (e.g., CNN) is used for video processing, enabling autonomous determination of road structure and surrounding dynamic objects (such as other vehicles). The analysis results are output to the driving control module 83 and reflected in automatic steering and acceleration / deceleration commands.

[0080] The driving control module 83 comprehensively analyzes various information obtained from the image analysis module 82, GNSS 34, and distance sensor 35, and generates control commands to control the direction and speed of the unmanned mobile body 10. The generated control commands are transmitted from the signal transceiver 50 via the tablet terminal 60 and reflected in the driving of each servo motor on the mobile body. This enables autonomous driving without human intervention.

[0081] Thus, in the second embodiment, the AI ​​PC 81 is configured as an external device rather than being mounted on the unmanned mobile vehicle 10. This simplifies the configuration of the unmanned mobile vehicle 10 itself, reducing the overall weight while enabling high-speed communication. As a result, it becomes possible to improve the driving performance and steering responsiveness of the unmanned mobile vehicle 10 while reducing battery load and improving acceleration and deceleration performance.

[0082] In addition to the AI-driven autonomous driving described above, the system may also be equipped with auxiliary functions to support human operation. For example, the AI ​​PC 81 can predict the optimal driving line based on driving data and environmental information it analyzes, and this driving line can be superimposed as an auxiliary line within the field of view of the AR glasses 41, allowing the driver to visually obtain reference information for the direction of travel and steering angle. Furthermore, by transferring and displaying real-time information such as the current location, direction of travel, sensor information, and road surface conditions stored on the tablet terminal 60 to the AR glasses 41, the driver can recognize the situation and make decisions without taking their eyes off the screen while driving.

[0083] Furthermore, if the unmanned mobile vehicle 10 gets stuck due to rough roads or obstacles while autonomous driving is being performed by AI, a function may be provided for the operator to manually assist in driving. For example, an emergency switching operation may be provided on the tablet terminal 60, allowing the automatic driving control to be temporarily deactivated by pressing a specific button, and the steering and throttle to be switched to manual control, enabling the vehicle to escape the situation based on human judgment. By introducing such a configuration in which humans and AI complement each other, driving reliability in all environments is improved, and more flexible and safe operation is realized.

[0084] <3. Other Embodiments> The unmanned mobile vehicle control system 100 according to this embodiment has been described above, but the application of the technical ideas of this disclosure is not limited to the above embodiment. For example, in the unmanned mobile vehicle control system 100 shown in Figure 8, an actuator 76 is newly added to the onboard control device 70. The actuator 76 drives the onboard unit 75 in the forward / backward / left / right or up / down direction based on detection information from a gyro sensor and an acceleration sensor installed on the unmanned mobile vehicle 10. As a result, the behavior of the unmanned mobile vehicle 10 during acceleration, deceleration, and turning is transmitted to the operator as physical movement, providing physical feedback similar to driving a real vehicle. This dramatically enhances the sense of presence and immersion, as well as improving operating accuracy and situational awareness.

[0085] Furthermore, while this disclosure exemplifies a small four-wheeled vehicle as the unmanned mobile body 10, it is not limited to this. For example, this disclosure can be applied to various types of unmanned mobile bodies, such as tank-type caterpillar-driven vehicles, flying vehicles such as quadcopters, water vehicles such as boats, underwater vehicles such as ROVs (Remotely Operated Vehicles / remotely operated underwater drones), and even autonomous robots, unmanned aircraft for disaster relief, motorcycles, and three-wheeled vehicles, regardless of the size of the mobile body. The configuration and control method can be changed depending on the mobile medium and application.

[0086] Specifically, when the aircraft is used as an unmanned mobile vehicle 10, the head gyro sensor 73 attached to the pilot's head may detect not only left-right head movements but also up-down head movements. By driving the camera servo motor 33 to control the orientation of the onboard camera 21 in the up-down direction based on the detection result of this up-down rotational movement (pitch angle), the pilot of the aircraft can intuitively control the visibility of flight altitude and overhead obstacles. This enables three-dimensional spatial awareness for the aircraft, contributing to improved visibility and operability.

[0087] Furthermore, for example, the operation of various lights such as headlights and brake lights mounted on the unmanned mobile vehicle 10 may be performed not only by control via the tablet terminal 60 or the onboard control device 70, but also using a dedicated application that can be installed on the operator's smartphone. This application has functions such as turning the lights on and off and selecting flashing patterns, and can be intuitively controlled through touch operation on the smartphone. This makes it possible to flexibly operate the lights in situations other than operation, such as during maintenance or display of the mobile vehicle, improving convenience and operability.

[0088] Furthermore, the unmanned mobile vehicle control system 100 may be configured to detect the occurrence of oversteer and understeer related to steering operations during driving and to transmit the determination result to the operator as acoustic data. For example, by outputting different types of warning sounds from headphones worn by the operator according to the determination result, abnormal vehicle behavior can be intuitively recognized without relying on visual cues. Such determinations can be made using data such as steering angle, yaw rate, vehicle speed, and lateral acceleration, based on well-known technologies. As an example, the expected yaw rate may be calculated for the steering angle input by the operator, and this may be compared with the actual yaw rate. If the actual yaw rate is greater than the expected yaw rate, it may be determined to be oversteer, and if it is smaller, it may be determined to be understeer. By providing the operator with acoustic feedback corresponding to the state obtained in this way, the reliability of the driving assistance can be increased.

[0089] Furthermore, while the above embodiment uses a tablet terminal 60 as an information processing device for user control, management of set information, and conversion processing of control signals, it is not limited to this. For example, other information processing devices with the necessary processing power and communication functions, such as laptop computers, desktop computers, or smartphones, may be used, and similar control and operation are possible if a dedicated application is installed. This makes it possible to realize a flexible system configuration according to the user's usage environment and portability needs.

[0090] Furthermore, although the above embodiment describes a cockpit-type control system in which the pilot is seated in the cockpit and operates the device, the invention is not limited to this example, and the technical concept of this disclosure can also be applied to control methods using a handheld transmitter (so-called radio control).

[0091] For example, the control of the headlights on an unmanned mobile vehicle can be operated by installing a dedicated application on a mobile device such as a smartphone, separate from the control transmitter. As an example, lighting devices such as headlights and brake lights can be individually turned on and off via the app's touch UI, allowing for flexible control even when not in operation.

[0092] Furthermore, even with remote control-type operation, a head tracking function can be applied that uses a gyro sensor or a head-mounted display such as AR glasses worn on the user's head to control the direction of the vehicle's onboard camera in conjunction with head movements. This allows the operator to move the camera viewpoint in accordance with their gaze while still holding the remote control, enabling a more immersive control experience.

[0093] Furthermore, the onboard control system in this disclosure is not limited to a dedicated cockpit-type system, but may also be configured to utilize the driver's seat of an existing vehicle as the control interface. In other words, by utilizing the driver's seat of an actual vehicle as an onboard control system and installing a tablet terminal 60, AR glasses 41, and various input devices inside, it becomes possible to remotely control other unmanned mobile vehicles while maintaining the comfort and operability of the actual vehicle.

[0094] Specifically, the scenario envisions two vehicles, A and B, where vehicle A is stationary, while an operator in its driver's seat controls vehicle B, which is located nearby or remotely. By acquiring video and operation information from vehicle B via a tablet terminal 60 or AR glasses 41 mounted in the driver's seat of vehicle A, and transmitting operation input to vehicle B via wireless communication, it becomes possible to safely and reliably control another unmanned mobile object from a real vehicle. This configuration provides a practical approach that can flexibly handle situations such as leaving the scene or switching between controlling multiple vehicles.

[0095] Furthermore, as mentioned above, the steering controller 71 is equipped with a force feedback function to enhance the realism of the steering feel. The force feedback function is a function that artificially applies forces such as reaction forces and vibrations to the steering wheel. Conventional technology related to force feedback functions is known to involve attaching a strain gauge sensor to the steering wiper to detect the deformation and stress of the steering wiper, quantitatively understanding the steering state, and then controlling the feedback value. However, with this method, especially when used in small unmanned mobile vehicles, the detected values ​​from the strain gauge sensor become unstable, making accurate feedback impossible and resulting in an unpleasant steering feel for the operator.

[0096] In contrast, the unmanned mobile vehicle control system 100 may employ, for example, a control method that gradually increases the force feedback force (FFB force) F based on the steering angle θ included in the control signal. For example, as the steering angle θ increases from zero, the FFB force F increases linearly with an inclination a (FFB COEF in the steering force feedback setting section 62c in Figure 5) (F=aθ) and reaches a maximum value. In this way, the steering controller 71 is controlled so that the reaction force increases as the steering angle increases. The maximum value of the FFB force and the inclination a can be arbitrarily changed by the operator using buttons on the tablet terminal 60 or the steering controller 71, making it easy to adjust the steering feel and customize it to the operator's preference.

[0097] Another example of a control method is to use a control method that increases gradually according to the steering angle θ, in addition to a control method that controls the strength of the feedback based on the speed v of the unmanned mobile body 10 obtained from an acceleration sensor (or speed sensor), in order to provide a natural return feeling to steering input even when the vehicle is stationary. Specifically, if the speed is above a predetermined threshold, the FFB force increases linearly with respect to the steering angle θ as described above (controlled by the inclination a). On the other hand, if the speed of the unmanned mobile body 10 is below a predetermined threshold, the FFB force is kept at a constant value (for example, a reference resistance μ corresponding to road surface friction) to suppress excessive reaction force. This enables light steering operation during low-speed driving and when stopped, and realizes a natural steering feel in which the weight increases according to the steering angle during high-speed driving. The predetermined threshold for speed, like the maximum value and inclination a, can be freely switched by operating a button on the tablet terminal 60 or steering controller 71, allowing adjustment of the response characteristics according to the operator's preference. If the speed is above a predetermined threshold, F = avθ may be used with speed v as the coefficient.

[0098] As yet another example of a control method, a method that more closely resembles the driving intervals of an actual vehicle may be adopted, which controls the strength of the feedback based on the acceleration obtained from an acceleration sensor. Specifically, as shown in Figure 9, the restoring force as feedback is determined from the relationship between the acceleration acting on the vehicle's center of gravity and the steering angle θ1 and the tire turning angle θ2 (θ2 = f(θ1). For example, θ2 = b × θ1, where b is a constant). μ + M A × |sin(θ² + θ³)| It is defined as follows: Here, μ is the reference resistance corresponding to road surface friction. force A and θ3 are the values ​​detected by the acceleration sensor. M is the mass of the vehicle. Therefore, by controlling the strength of the feedback based on the acceleration detected in the vehicle in this way, a force feedback function that is closer to the actual driving sensation can be realized. Note that if A is below a certain level, M A × |sin(θ² + θ³)| may be treated as a constant value.

[0099] Furthermore, the unmanned mobile vehicle control system 100 may also include a function to display the steering characteristics of the unmanned mobile vehicle 10 in real time on the tablet terminal 60 in order to understand the steering characteristics of the unmanned mobile vehicle 10. For example, the system may classify the characteristics as understeer (U), neutral steer (N), or oversteer (O), and display an identification symbol (U, N, or O) and a numerical index corresponding to the driving state on the tablet terminal 60, allowing the operator to visually understand the changes in the vehicle's behavior. In addition, the system may also display the route after driving in different colors according to the driving characteristics. This encourages the operator to be conscious of driving techniques that correspond to the steering characteristics when controlling the unmanned mobile vehicle 10. Furthermore, it enables vehicle design or vehicle settings to be made according to the steering characteristics.

[0100] The steering characteristics may be determined, for example, based on the tire slip angle. Specifically, the average value of the slip angle is calculated based on the velocity vector components obtained from acceleration sensors or yaw rate sensors attached to the front and rear wheels, and the difference between the front and rear is calculated to quantitatively identify the vehicle's steering characteristics (understeer, neutral steer, or oversteer). This process may be calculated using either the four-wheel sensor method or the center of gravity sensor method described below.

[0101] In the four-wheel sensor system, the difference in slip angle between the front and rear wheels is calculated based on the speed of each wheel obtained from acceleration sensors placed on each wheel, and the steering angle. This method has advantages in that it requires fewer input values ​​and less setup effort, and it is highly accurate because it is based on actual measurements of all four wheels.

[0102] On the other hand, the center of gravity sensor method acquires the behavior of the entire vehicle (acceleration and angular velocity) from acceleration sensors and yaw rate sensors mounted in the center of the vehicle, and calculates the difference in slip angle between the front and rear wheels by calculating the speed of each of the four wheels based on the distance between the center of gravity and the wheels. This method has the advantage of reducing the number of sensors, and is effective in reducing the weight and cost of small mobile devices.

[0103] Information regarding these steering characteristics is displayed in the status display area 63 of the tablet terminal 60, allowing the operator to immediately grasp any changes in behavior while driving. Furthermore, the system may be configured to automatically correct the force feedback characteristics according to the determined steering characteristics. For example, the steering reaction force may be increased when there is an understeer tendency, and the steering movement may be smoothed when there is an oversteer tendency, thus achieving a balance between driving feel and safety. As a result, the operator can more easily maintain the intended line accurately, enabling remote control with higher precision and a driving feel closer to that of a real vehicle.

[0104] Furthermore, the unmanned mobile vehicle control system 100 may perform smoothing processing for the head tracking function. In the head tracking function, the camera servo motor 33 controls the orientation of the on-board camera 21 according to the movement of the operator's head acquired by the head gyro sensor 73. However, because digital servo motors are highly responsive, they may react too sensitively, causing unpleasant vibrations or discomfort for the operator. To prevent this, smoothing processing is performed as shown below so that the movement of the camera servo motor 33 becomes smoother in response to the movement of the operator's head.

[0105] As an example of smoothing processing, the output signal from the head gyro sensor 73 may be filtered to remove high-frequency components. Specifically, by applying a low-pass filter or the like to remove high-frequency components caused by minute head movements or sensor noise, unnecessary over-responses from the servo motor can be avoided, resulting in smooth control operation.

[0106] Another example of smoothing is setting a coarser "camera step" to control the camera's panning motion in gradual steps. This is equivalent to intentionally setting a coarser resolution. Additionally, a "camera dead zone" may be set so that the camera only operates when the operator's head exceeds a certain angle. These processes prevent the camera from reacting to every slight movement of the head, maintaining smooth and stable viewpoint movement. The "camera step" and "camera dead zone" are displayed in the settings display area 64 on the tablet terminal 60, allowing the operator to change the settings as needed.

[0107] Another example of smoothing is setting the target angle of the camera servo motor 33 to the current angle θ of the operator's head. T (tT) (where T is the time it takes for a signal to be transmitted from the head gyro sensor 73 to the camera servo motor 33) and the angle θ from Δt seconds ago. TA smoothing process may be introduced that uses a value between (tT - Δt). Furthermore, the smoothing process may only be used when certain conditions are met, such as the pilot's head movement being in the deceleration direction. This allows for the realization of natural, analog-like viewpoint movement while taking advantage of the high speed of digital servo motors, and reduces overshoot (for example, the angle of the head θ). T The camera servo angle θ is relative to the magnitude of the change. C This also prevents situations where the magnitude of the change exceeds a certain limit. The conditions for using smoothing (e.g., the value of Δt), and / or the specific correction formula used in smoothing, should be set appropriately according to the precision, load, and purpose of the calculation.

[0108] While several embodiments of this disclosure have been illustrated above, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. Furthermore, the embodiments described above can be implemented in combination with each other. [Explanation of symbols]

[0109] 10: Unmanned mobile unit, 20: Video transmitter, 21: Onboard camera, 30: Signal transceiver, 31: Steering servo motor, 32: ESC motor, 33: Camera servo motor, 34: GNSS, 35: Distance sensor, 40: Video receiver, 41: AR glasses, 50: Signal transceiver, 60: Tablet terminal, 61: Display screen, 62: Control display area, 62a: Stop button, 62b: Direction setting area, 62c: Steering force feedback setting area, 62d: Sensor setting area, 62e: Acceleration display area, 63: Status display area, 63a: Operation status display area, 63b: Sensor status display area, 63c: Light status indicator, 64: Setting display area, 64a: Brake response setting area, 64b: Throttle response setting field, 64c: Steering response setting field, 64d: Camera response setting field, 70: Onboard control system, 71: Steering controller, 72: Pedal controller, 73: Head gyro sensor, 74: Headgear, 75: Seat unit, 76: Actuator, 80: Adapter, 81: PC for AI, 82: Image analysis module, 83: Driving control module, 90: Computer, 91: CPU, 92: ROM, 93: RAM, 94: Storage, 95: Input interface, 95a: Input device, 96: Output interface, 96a: Output device, 97: Communication interface, 97a: External server, 98: System bus, 100: Unmanned mobile vehicle control system

Claims

1. An unmanned mobile vehicle equipped with a camera, A ride-on control device for a user to board and operate the unmanned mobile vehicle, The system includes a video output device that outputs video captured by the aforementioned camera, The aforementioned onboard control device includes a steering controller for steering the unmanned mobile body, The unmanned mobile body includes an acceleration sensor for detecting acceleration, An unmanned mobile vehicle control system that controls force feedback in the steering controller based on the detected value detected by the acceleration sensor and the steering angle of the tire calculated from the steering angle during steering of the steering controller.

2. The restoring force of the aforementioned force feedback is μ + MA × |sin(θ2 + θ3)| The unmanned mobile vehicle control system according to claim 1, wherein the calculation is performed by θ2 being the steering angle of the tire, μ being the reference resistance force corresponding to road surface friction, M being the mass of the vehicle, A being the magnitude of the acceleration detected by the acceleration sensor, and θ3 being the angle between the acceleration and the forward direction of the unmanned mobile vehicle.

3. The restoring force of the aforementioned force feedback is μ + MA × |sin(θ2 + θ3)| The formula is calculated as follows: θ2 is the steering angle of the tire, μ is the reference resistance force corresponding to road surface friction, M is the mass of the vehicle, A is the magnitude of the acceleration detected by the acceleration sensor, and θ3 is the angle between the acceleration and the forward direction of the unmanned mobile body. The unmanned mobile vehicle control system according to claim 1, wherein if A is less than or equal to a certain value, MA × |sin(θ² + θ³)| is controlled to be a constant value.

4. The unmanned mobile vehicle control system according to claim 1, wherein the imaging device is installed inside the unmanned mobile vehicle.

5. The unmanned mobile vehicle control system according to claim 1, wherein the video output device is an augmented reality (AR) glasses that can be worn on the user's head.

6. The system further comprises a headgear that is worn on the user's head, The headgear includes a sensor for detecting the orientation of the user's head. The unmanned mobile unit includes a servo motor capable of changing the orientation of the imaging device, The unmanned mobile vehicle control system according to claim 1, wherein the servo motor is controlled so that the movement of the imaging device becomes smooth in response to the detected movement of the user's head.

7. The aforementioned unmanned mobile body includes at least one of a headlight and a brake light, The unmanned mobile vehicle control system according to claim 1, wherein the onboard control device receives operation of at least one of the headlights and the brake lights.

8. The unmanned mobile vehicle control system according to claim 1, wherein the ride-on control device drives a boarding section for the user to board based on the detected acceleration.

9. The unmanned mobile vehicle control system according to claim 1, further comprising an information processing device that displays steering characteristics in the steering of the unmanned mobile vehicle by the steering controller.

Images