MOBILE DEVICE AND REMOTE CONTROL METHOD

Abstract

Mobile device (100) intended to be carried by a user (2) of a vehicle (1) and capable of moving the vehicle (1) by remote control through the transmission of an operating signal corresponding to operation by the user (2), wherein the mobile device (100) has: a long image section (102) which is connected to an image section (101) which takes an image of the vehicle (1) and obtains a taken image of the image section (101), an image display section (103) that displays the captured image, a distance information acquisition section (107) that detects the vehicle (1) in the recorded image and obtains distance information about a distance to the vehicle (1) based on the recorded image, a determining section (109) which, based on the distance information, determines whether the distance to the vehicle (1) is within a specified permissible distance, an operating content capture section (111) which is connected to an operating section (112) which is provided so that it can be operated by the user (2) while checking a display of the captured image, and captures an operating content of the operating section (112), and an operating signal transmission section (110) which transmits the operating signal corresponding to the operating content to the vehicle (1) when it is determined that the distance to the vehicle (1) is within the permissible distance, wherein Even after the operating signal transmission section (110) has transmitted the operating signal to the vehicle (1), if the vehicle (1) is no longer detected in the recorded image, or if it is determined that the distance to the vehicle (1) is outside the permissible distance, the operating signal transmission section (110) stops transmitting the operating signal corresponding to the operating content until it is determined that the distance to the vehicle (1) is within the permissible distance. The image display section (103) displays, in addition to the captured image, the distance information about the vehicle (1) that is captured in the captured image and information on whether the distance information is within the permissible distance.

Description

Cross-reference to related registrations

[0001] This application is based on Japanese patent application No. 2017-179716, filed on September 20, 2017; the disclosure content thereof is included herein by reference. Technical field

[0002] The present invention relates to a mobile terminal and a remote control method for remotely controlling a vehicle by a user of the vehicle. background

[0003] A technique has been proposed to park a vehicle in or out of a parking space from outside the vehicle using a remote control device. Given the widespread use of mobile devices such as smartphones, a technique has also been proposed that allows the vehicle to be controlled remotely by pressing a forward / reverse button or a steering wheel button displayed on a touchscreen of a mobile device.

[0004] However, if a vehicle is remotely controlled in this way, it can be assumed that the user of the vehicle is thereby obligated to operate the vehicle safely in the same manner as in the case of normal driving. Consequently, it is necessary for the user of the vehicle to have a sufficient understanding of the vehicle's condition and the situation surrounding it, and for this reason (even with remote control), it is advisable to operate the vehicle in a state where the user is in close proximity to it.

[0005] Consequently, a technique for confirming the presence of the user near the vehicle by detecting an electronic key worn by the user was proposed (JP 2016 - 97 927 A). In the proposed technique, the vehicle searches for the nearby electronic key and, upon finding it, authenticates whether the key is a valid one. If the electronic key is valid, the vehicle determines that the user is nearby and accepts remote control from outside.

[0006] However, since the proposed technology uses an electronic key for authentication, it is necessary for the vehicle user to carry the electronic key in addition to the mobile device, even though the vehicle is being remotely controlled using the mobile device.

[0007] It goes without saying that such a problem would not occur if the vehicle user could confirm the presence of the mobile device by scanning the device instead of the electronic key. However, since the radio wave intensity of a mobile device fluctuates considerably due to the influence of a human body when approaching the vehicle, it is difficult to accurately determine whether the mobile device is near the vehicle.

[0008] As a result, when attempting to remotely control the vehicle, the user must carry the electronic key in addition to the mobile device for remote control.

[0009] DE 10 2012 016 868 A1 discloses a method and a system for operating a vehicle while monitoring its movement using a camera system and a mobile control unit. Further such systems are also known from DE 10 2015 223 471 A1 and WO 2015 / 068 032 A1.

[0010] The present invention was made in view of the above circumstances and consequently it is an object of the present invention to provide a mobile terminal and a remote control method that are capable of moving a vehicle by remote control using a mobile terminal, even if the user of the vehicle does not carry an electronic key.

[0011] This problem is solved by a mobile device with the features according to claim 1 and a remote control method with the features according to claim 7. Advantageous embodiments are the subject of the dependent claims.

[0012] A mobile device according to one aspect of the present invention, which is to be worn by a user of a vehicle, is capable of moving the vehicle by remote control through the transmission of an operating signal corresponding to operation by the user of the vehicle, and comprises an image acquisition section connected to an image acquisition section which captures an image of the vehicle and obtains a captured image from the image acquisition section, a distance information acquisition section which obtains distance information about a distance to the vehicle based on the captured image, a determination section which determines, based on the distance information, whether the distance to the vehicle is within a predetermined permissible distance, and an operating signal transmission section which transmits the operating signal to the vehicle when it is determined that the distance to the vehicle is within the permissible distance.

[0013] A remote control method according to another aspect of the present invention serves to move a vehicle by remote control using a mobile device carried by a user of the vehicle, and comprises obtaining a captured image of the vehicle from an image section attached to the mobile device, obtaining distance information about a distance to the vehicle based on the captured image, determining whether a distance to the vehicle is within a predetermined permissible distance based on the distance information, and transmitting an operating signal corresponding to operation by the user to the vehicle when it is determined that the distance to the vehicle is within the permissible distance.

[0014] If the distance between the mobile device and the vehicle exceeds the permissible distance, the operating signal will not be transmitted. Therefore, the fact that the vehicle receives the operating signal from the mobile device simply means that the mobile device is within the permissible range. Furthermore, since the permissible distance between the mobile device and the vehicle is determined based on the captured image, it is less likely to be affected by interference or similar factors compared to a scenario based on radio wave intensity or other similar indicators. Consequently, it can be determined with a high degree of reliability whether the mobile device is within the permissible distance of the vehicle.Even if, as a result, the vehicle user does not carry the electronic key, if the user carries a mobile device for remote control of the vehicle, a situation in which the vehicle is controlled from a distance that is further than the permissible distance can be avoided. Brief description of the characters

[0015] The above and other objects, features, and advantages of the present invention are clarified by the following detailed description, which is made with reference to the accompanying figures. These show: Fig. 1 a diagram representing a situation in which a vehicle is controlled by remote control using a mobile device, according to the present embodiment, Fig. 2. A diagram showing a rough internal structure of the mobile device. Fig. 3. A diagram showing an example of a screen display of the mobile device when the vehicle is remotely controlled. Fig. 4. A diagram illustrating an example of a situation in which remote control is prevented. Fig. 5 a flowchart of a start correctness determination phase of a vehicle remote control processing, Fig. 6. A flowchart of an operational execution phase of vehicle remote control processing, Fig. 7 a diagram representing a rough internal structure of a mobile device according to another embodiment, Fig. 8. A flowchart of a start correctness determination phase in the case of the internal structure, which is in Fig. 7 is shown, and Fig. 9 a diagram illustrating a screen display example according to yet another embodiment. Description of embodiments

[0016] The following describes embodiments to clarify the contents of the present invention.

[0017] Fig. Figure 1 shows a state in which a vehicle 1, viewed from above, is being moved towards a parking space 3 among several divided parking spaces in order to park the vehicle 1. A user 2 of the vehicle 1 is present outside the vehicle and remotely controls the vehicle 1 using a mobile device 100. The vehicle 1 is moved to a position indicated by a virtual line 1 and is parked in the parking space 3 by remote control from user 2, by reversing while steering to the left. Although the vehicle 1 can be remotely controlled in this way using the mobile device 100, the situation around the vehicle 1 cannot be adequately confirmed if user 2 is remotely controlling the vehicle 1 away from the vehicle 1.Consequently, in the case of the mobile device 100 of the present invention, if the mobile device 100 is near the vehicle 1, the mobile device 100 can operate the vehicle 1, but cannot operate the vehicle 1 if the mobile device 100 is far away from the vehicle 1.

[0018] Fig. Figure 2 shows a rough internal structure of the mobile terminal 100 according to the present embodiment. As shown, the mobile terminal 100 has an image section 101, an image storage section 102, an image display section 103, a display device 104, a section 105 for determining the appropriate vehicle, a vehicle identification information storage section 106, a distance information storage section 107, a vehicle dimension information storage section 108, a determination section 109, an operating signal transmission section 110, an operating content acquisition section 111, an operating section 112, a receiving section 113, an authentication signal transmission section 114, and an antenna 115.

[0019] It should be noted that each "section" of the mobile device 100 is merely a classification to simplify its internal structure, focusing on a function required when the vehicle 1 is remotely controlled, and does not indicate that the mobile device 100 is physically divided. Consequently, each "section" can be implemented as a computer program executed by a CPU, as an electronic circuit incorporating LSI, or as a combination of the computer program and the electronic circuit.

[0020] The imaging section 100, for example, has a CCD or CMOS camera located on the back of the mobile device 100 and continuously captures images during remote control of the vehicle 1. The image acquisition section 102 is connected to the imaging section 101 and obtains an image captured by the imaging section 101 at a predetermined interval. The captured images are then sequentially output to the image display section 103 and, if necessary, to section 105 for determining the appropriate vehicle and to the distance information acquisition section 107, for example, if a data request is received from these sections.

[0021] The image display section 103 outputs the captured images, received by the image long section 102, sequentially to the display device 104, which is provided, for example, on a front surface of the mobile device 100. As a result, the last captured image is always displayed on the display device 104. The display device 104 is provided with a thin panel, such as an LCD or organic EL, and is provided over a wide area, almost across the entire front surface. The display device 104 is a touch panel of a suitable type, such as a pressure-sensitive or electrostatic type, and various inputs and operating instructions can be entered by touching the displayed icons.During remote control, the user 2 of the vehicle 1 performs an operation while checking the vehicle 1, which is displayed on the display device 104, and adjusts the orientation of the mobile device 100 as needed so that the vehicle 1 can be captured by the image section 101.

[0022] Section 105, used to determine the suitable vehicle, analyzes the captured image received by image processing section 102, extracts a vehicle image from the captured image, extracts vehicle identification information from the vehicle image, and compares this extracted vehicle identification information with identification information for vehicle 1 that is pre-stored in vehicle identification information storage section 106. This comparison determines whether the vehicle captured by mobile device 100 is the vehicle 1 intended to be remotely controlled by mobile device 100—that is, whether it is a suitable vehicle. The vehicle identification result is output to determination section 109. In this example, the vehicle identification information includes, for instance, the license plate number of vehicle 1, the vehicle's shape, its color, and similar details.In principle, a vehicle's suitability is determined by matching the license plate number. However, instead of the license plate number, the vehicle's shape and color can also be used for this purpose. Depending on the positional relationship between user 2 of vehicle 1 and vehicle 1 itself, it is effective to determine suitability based on the vehicle's shape and color, even if the license plate is not visible in the captured image or if the license plate is not clearly legible. To more reliably determine suitability, matching the vehicle's shape and color can be a further condition, in addition to the license plate number. For example, the vehicle identification information storage section 106 stores this information.Information that is entered by user 2 of vehicle 1, who operates the mobile device 100, or transmitted from another device to the mobile device prior to remote control.

[0023] The distance information acquisition section 107 analyzes the captured image received from the image acquisition section 102, referring to the dimensional information of the vehicle 1 (including its overall length, lateral width, and other dimensions) pre-stored in the vehicle dimension information storage section 108. It then obtains distance information about the distance between the mobile device 100 and the vehicle 1. This distance information is output at the determination section 109. Since the size of the vehicle in the image depends on the image acquisition distance, the distance to the vehicle 1 can be obtained by comparing the vehicle dimensions in the image with a pre-stored vehicle dimension. The distance information is not limited to the distance obtained in this way and can include information relating to a size relationship with a predefined reference value. For example,The reference value for the vehicle dimensions is predefined, and the vehicle dimensions obtained from the image are compared to this reference value. The value is stored as large if the vehicle dimensions are equal to or greater than the reference value, and as small if they are smaller. In other words, distance information can be obtained based on the size of vehicle 1 in the captured image. The reference value for the vehicle dimensions is determined, for example, to be the vehicle dimensions in the image, provided the distance from the mobile device 100 to vehicle 1 is a safe distance, even if remote control is being used.

[0024] The vehicle dimension information storage section 108 stores, for example, dimension information relating to vehicle 1, which is entered by user 2 of vehicle 1, who is operating the mobile device 100, or which is transmitted from another device to the mobile device 100 prior to remote control. The permissible distance is a distance that can be determined to be safe, even if remote control is carried out, and is set, for example, at several meters to several tens of meters.

[0025] For example, a general-purpose dimension of approximately 5 m can be used as the reference value for the vehicle dimension instead of the actual dimension of the vehicle 1 to be operated, and the input of the vehicle dimension can be omitted. If such a general-purpose dimension is used as a reference when measuring the distance between the mobile device 100 and the vehicle 1, an error corresponding to the difference in the actual vehicle dimension may occur, and consequently, the permissible distance can be set taking this error into account.

[0026] The determination section 109 determines, based on the vehicle determination result received from section 105 for determining the suitable vehicle, and the distance information obtained from the distance information acquisition section 107, whether a remote control signal should be transmitted to vehicle 1. If the vehicle determination result is the determination of a suitable vehicle, the distance information is used to determine whether the distance from the mobile device 100 to vehicle 1 is within the permissible range. If the distance is within the permissible range, a transmission authorization signal is issued to the operating signal transmission section 110.In cases where the distance information is derived from the vehicle's dimensions in the image, the determination is positive if the distance is equal to or less than the permissible distance. In cases where the distance information is a size relative to the vehicle's dimensions, the determination is positive if the size relative is "large," meaning equal to or greater than the reference value. However, if the vehicle is not the appropriate vehicle, or if the distance is determined to be outside the permissible range, remote control should not be permitted, and consequently, a transmission permission instruction will either not be issued or a transmission prevention instruction will be issued.In contrast to the determination above, when using section 109, the determination of whether the distance is within the permissible distance can be carried out first, and if it is determined that the distance is within the permissible distance, the determination of whether the vehicle is a suitable vehicle can be carried out.

[0027] The operating content determination section 111 detects an operating content with which the operating section 112 is to be operated, using operating buttons or the like, generates an operating signal according to the operating content, and outputs the operating signal to the operating signal transmission section 110. In addition to the operating buttons shown in the figures, the operating section 112 has the display device 104 or the like, which serves as a touch panel, and the operating content includes operation that is carried out using the touch panel. The operating signal transmission section 110 outputs the operating signal received from the operating content detection section 111 if the transmission authorization instruction has been received from the determination section 109. The output operating signal is transmitted by the antenna 115 to the vehicle 1.If no transmission authorization instruction is present, the operating signal will not be transmitted.

[0028] The receiving section 113 receives a signal transmitted by vehicle 1 via antenna 115. After receiving the authentication request signal from vehicle 1, the receiving section 113 outputs the authentication request signal to the authentication signal transmission section 114. After receiving the authentication request signal, the authentication signal transmission section 114 transmits a pre-stored authentication signal to vehicle 1. Vehicle 1 transmits the authentication request signal in a predefined cycle and searches for the mobile device 100, which is present in the surrounding area.When user 2 of vehicle 1 comes close to vehicle 1 while holding mobile device 100, mobile device 100 receives the authentication request signal when mobile device 100 enters an area where the radio wave from vehicle 1 arrives, with authentication processing with vehicle 1 being carried out automatically by a so-called passive method.

[0029] Fig. Figure 3 shows an implementation scenario for the remote control of vehicle 1 using the mobile device 100, along with a display example of the display device 104. The vehicle 1 to be operated is positioned in front of the mobile device 100, and the vehicle 1 is operated in such a way that it is to be parked between the other two vehicles located behind it. The mobile device 100 includes the display device 104 and a plurality of operating buttons, e.g., four operating buttons 120a, 120b, 120c, and 120d, as components of the operating section 112. The operating section 112 is, as described above, arranged in the same plane as the display device 104 and can be operated while the display of the display device 104 is being checked. Fig. Figure 3 shows a situation in which a remote control application is activated and operated by the mobile device 100, and an operating screen of the application is displayed on the display device 104. The operating screen displays the words "left," "forward," "backward," and "right" in sequence, starting from the leftmost position above the respective buttons 120. These words represent the functions of applications associated with the respective operating buttons 120a to 120d. Vehicle 1 can be steered to the left by operating button 120a, forward by operating button 120b, backward by operating button 120c, and right by operating button 120d. Each operation is performed, for example, while each operating button 120 is pressed.

[0030] A distance indicator section 122 is located in an upper section of the operating screen. The distance indicator section 122 features "near" and "far" text displayed at its left and right ends, respectively; a wide bar indicating the permissible distance, extending from the position where the "near" text is displayed towards the "far" text to a midpoint; a dashed line displayed between the permissible distance bar and "far"; and an "out of range" text displayed below the dashed line. A black triangle marker (hereinafter referred to as a black triangle marker) is displayed on the permissible distance bar. The distance indicator section 122 displays the current distance from the mobile device to the vehicle 1 and indicates whether the current distance is within a permissible range.The black triangle marking represents the current distance, and the bar for the permissible distance represents a range of distances within which remote control is allowed. The dashed line represents an area exceeding the permissible distance. When the black triangle marking is positioned closer to "near," the distance to vehicle 1 is closer; conversely, when the black triangle marking is positioned closer to "far," the distance to vehicle 1 is farther. If the black triangle marking is present on the bar for the permissible distance, user 2 of vehicle 1 can visually determine that the distance to vehicle 1 is within the permissible range and remote control can be performed.If the black triangle marking is present on the dashed line, user 2 can visually recognize that the distance to the vehicle exceeds the permissible distance and remote control cannot be performed.

[0031] In the lower right section of the operating screen, an end-of-operation button 124 is displayed. User 2 of the vehicle taps the end-of-operation button 124 to transmit a remote control end instruction to vehicle N1 to end remote control.

[0032] Vehicle 1 is displayed near the center of the operating screen. Remote control can be performed when Vehicle 1 is displayed on the operating screen. User 2 of the vehicle presses and operates the operating buttons 120a to 120d while capturing an image of Vehicle 1 with the mobile device 100, thereby transmitting corresponding operating instructions—forward, reverse, turn left, and turn right—to the vehicle to move it to a target parking position. Since remote control, as described above, is performed while both units, the operating screen and the actual Vehicle 1, are viewed when Vehicle 1 is displayed on the operating screen and is within the permissible distance, User 2 of the vehicle 1 can safely perform remote control while monitoring the surroundings of Vehicle 1.“Forward”, “backward”, “left” and “right”, which are displayed on the operating screen, are, for example, the same as the operating directions in normal operation of the vehicle, but “left” and “right” can, for example, be set to be opposite to normal operation according to the left and right directions on the screen.

[0033] Furthermore, the operating buttons 120, which correspond to opposite operating directions ("forward," "rear," or "left" and "right"), are mutually exclusive. Even if these buttons 120 are pressed simultaneously, the operating signals from both buttons (forward and reverse) and the operating signals from both buttons (steer left and turn right) are not transmitted to vehicle 1. When the two exclusive operating buttons 120 are pressed simultaneously, an operating signal corresponding to one previously pressed button may be transmitted to vehicle 1, and the operation of the subsequently pressed button may be ignored. Alternatively, signal transmission may be stopped, and a warning indicating that simultaneous operation is not possible may be displayed on the operating screen.

[0034] On the other hand, a state in which one of the buttons, "front" and "back," and one of the buttons, "left" and "right," are pressed simultaneously is permitted. When pressed simultaneously, the vehicle moves forward or backward while changing the direction of its wheels. In other words, while "front" or "back" and "left" or "right" are pressed, vehicle 1 moves forward and backward while changing its turning radius. Conversely, if either "left" or "right" is pressed alone, a so-called "stationary steering" is performed, in which the direction of the wheels is changed while vehicle 1 is stationary.

[0035] An automatic parking button (123) is provided on the lower left side of the operating screen of the display device (104). The automatic parking button (123) is displayed and selectable when the vehicle (1) is equipped with a self-driving function. When the user (2) of the vehicle (1) presses the automatic parking button (123), the vehicle (1) takes an image of the surrounding area using an onboard camera (not shown) and detects the parking space (3) by analyzing the image. The vehicle (1) then automatically parks itself in the parking space (3) by automatically performing steering, braking, and acceleration operations, while detecting any obstacles around the vehicle (1) using sonar or radar (not shown).

[0036] Even if vehicle 1 has the automatic parking function, user 2 of vehicle 1 is still responsible for ensuring that vehicle 1 is parked safely. Even after pressing button 123 to start automatic parking, user 2 of vehicle 1 can confirm whether vehicle 1 has been automatically parked and the situation around vehicle 1, as shown on the mobile device's display 104. The distance to vehicle 1 can also be easily maintained using the distance indicator 122. If any difficulty or danger is detected, vehicle 1 can be stopped, and automatic parking can be ended by pressing the end-of-operation button 124.This allows user 1 of vehicle 1 to park vehicle 1 more safely than if they were driving while sitting in a driver's seat of vehicle 1.

[0037] While user 2 presses button 123 for automatic parking, vehicle 1 alternatively moves towards the inside of parking space 3 using the automatic parking function. However, if user 2 removes their finger from button 123 for automatic parking, the automatic parking function of vehicle 1 can be interrupted and vehicle 1 can be stopped immediately. This ensures safety even when vehicle 1 is parked automatically.

[0038] In the mobile device 100 of the present invention, the operating buttons 120 are as shown in Fig. Figure 3 shows that the operating buttons 120 are implemented by the hardware of the mobile device 100. However, some or all of the operating buttons 120 can be displayed on the display device 104 of the mobile device 100, meaning they are implemented by software. If a mobile device has, for example, three or fewer operating buttons 120 attached to it, and is used for remote control, it is effective to implement the operating buttons 120 in software.

[0039] Fig. Figure 4 shows an example of the display on the display device 104 in a situation where remote control is prevented. In this display example, the vehicle 1 does not appear completely on the display screen of the display device 104, and part of the vehicle 1 extends to the left. If, in the present embodiment, for example, the entire vehicle 1 appears on the display screen of the display device 104, it is determined that an image of the vehicle 1, which is to be remotely controlled, is being captured, and in this case, it is determined whether the vehicle is suitable and whether the distance is within the permissible range. Since the size of the vehicle in the captured image can be precisely determined by determining whether the vehicle 1 is captured in this manner, the determination of whether the vehicle is suitable and whether the vehicle is within the permissible range is better ensured.Instead of determining the vehicle's identity, it can be determined, as described above, that the vehicle 1 is in a state resembling a captured image if, for example, a constant or a significant portion of the vehicle 1 is visible within the display screen. If the license plate is clearly legible, it can be determined, based on the elements described on the license plate, whether the vehicle is a suitable vehicle, even if the entire vehicle 1 is not necessarily depicted.

[0040] The process for remotely parking the vehicle while the engine is running, using the mobile device 100, is described. Prior to the following processing, the vehicle 1 performs authentication processing of the mobile device 100. This authentication processing is carried out, for example, as described above, by the mobile device 100, which receives the authentication request signal, repeatedly transmitted by the vehicle 1 towards its surroundings at a predetermined time interval, via the receiving section 113 and transmits the authentication signal from the authentication signal transmission section 114 to the vehicle 1.

[0041] Fig. Figure 5 shows a flowchart of a determination process for the appropriate remote control, in which the mobile terminal 100 analyzes an image of the vehicle 1, records the vehicle identification information and the distance information and determines whether the remote control can be carried out.

[0042] As shown in the figure, during remote vehicle control processing, the remote control function of vehicle 1 is activated first (S100). The remote control function of vehicle 1 is always paused or stopped to prevent unnecessary energy consumption or unwanted remote control. Consequently, if remote control is to be carried out, an activation signal for the remote control function is first transmitted from the mobile device 100 to vehicle 1.

[0043] Next, it is determined whether the response signal from vehicle 1 was received (S101). After receiving the activation signal for the remote control function, vehicle 1 activates the remote control function and transmits a response signal to mobile device 100. If the response signal was not received (NO in S101), it is assumed that the activation signal for the remote control function did not reach the vehicle, and consequently, processing returns to S100, and the activation signal for the remote control function is transmitted back to vehicle 1.

[0044] When the response signal from vehicle 1 is received (YES in S101) during the start operation (S100) and the determination (S101) are repeated, an instruction prompting user 2 of vehicle 1 to take a picture of vehicle 1 is displayed on the operating screen (S102). In the present embodiment, as described above, the determination of whether the vehicle is an operational target, the determination of whether the distance is suitable for remote control, and the like are performed based on the captured vehicle image. For this reason, when starting remote control, it is necessary to capture the image of vehicle 1 with the mobile device 100, and consequently, user 2 of vehicle 1 aligns the camera of the imaging section 101 with vehicle 1 according to the instruction to start the image capture.At the time of image capture, the image section 101 can be automatically activated when it is determined that the response signal has been received (S101) or can be activated by user 2 of vehicle 1 by pressing an image capture start button provided on the operating screen.

[0045] Next, the captured image is obtained by the image processing unit 102 (S103). This processing is carried out, for example, immediately after an instruction to capture the image is displayed on the operating screen, or by user 2 of vehicle 1, who presses the image capture start button displayed on the screen. After obtaining the captured image from the image processing unit 101, the image processing unit 102 outputs the captured image sequentially to unit 105 for determining the appropriate vehicle.

[0046] Next, in section 105, a vehicle extraction process is performed on the received captured image (S104) to determine the appropriate vehicle and whether the vehicle is present in the captured image (S105). Since remote control, as described above, is permitted when the image of vehicle 1 is captured by the mobile device 100, it is first determined whether the image of the vehicle is being captured. If user 2 of vehicle 1 has not yet started capturing the image of vehicle 1 and the vehicle is not present in the captured image, the vehicle is not extracted and the determination is negative (NO in S105), so the processing returns to S102 and an instruction to initiate image capture is displayed again on the operating screen.In this case, user 2 of vehicle 1 either restarts the image capture process or starts an image capture by pointing the camera at vehicle 1. Whether the image of the vehicle is captured is determined, as described above, based on whether the image of the entire vehicle is captured. However, it can also be determined, for example, that a state is permitted in which part of the vehicle image is not captured. At this time, section 105, for determining the suitable vehicle, determines, as described above, whether the vehicle is a suitable vehicle based on the vehicle's license plate number. For this reason, the determination (S105) of whether the vehicle is present is positive if the vehicle with the clearly visible license plate is present.If the license plate is not clearly visible, even if the vehicle is visible, a negative determination is made for this reason, and the processing reverts to S102.

[0047] If the vehicle is successfully extracted from the captured image while the above operations are repeated (YES in S105), the determination is positive, so the vehicle identification information is extracted from the image by analyzing the image of the vehicle in the captured image (S106). Since the periphery of the captured image is also being captured at this time, in addition to vehicle 1, the vehicle image is extracted from the captured image first at the time of this image analysis. Then, for example, the license plate number, the shape, and the color of the vehicle are extracted, and the extracted information is collated with information stored in the vehicle identification information memory section 106 to determine whether the vehicle in the captured image is a suitable vehicle, that is, whether the captured vehicle is vehicle 1, which is to be remotely controlled (S107).

[0048] If the pre-stored information does not match the license plate number obtained from the image, or the vehicle's shape or color, the system determines that the vehicle is not suitable for remote control by the mobile device 100 (NO in S107). A message indicating that the vehicle is not suitable, such as "this vehicle is not a vehicle that can be operated by this device," is then displayed on the operating screen (S111). The vehicle's remote control function is terminated, and the remote control application of the mobile device 100 is closed. In other words, a series of processes is terminated without any remote control operation taking place.After reviewing the display, user 2 of vehicle 1 can understand that they have mistakenly attempted to operate a vehicle different from the one intended for operation, or a vehicle that is not their own. Since this determines whether remote control activation is appropriate, a vehicle other than the one pre-registered in mobile device 100 can be prevented from being mistakenly remotely controlled.

[0049] If, on the other hand, the pre-stored information matches the license plate number obtained from the image, or matches the vehicle's shape and color, instead of or in addition to the license plate number, it is determined that the vehicle is a vehicle intended to be operated by the mobile device 100, i.e., a suitable vehicle (YES in S107), and the distance information acquisition section 107 analyzes the vehicle image and captures the distance information (S108). In this analysis, the vehicle image is used as is to determine whether the vehicle is a suitable vehicle or not. The capture result is displayed, for example, by updating the position of the black triangle marker on the operating screen or by text such as "Operating area" and "Outside the operating area".

[0050] If the acquired distance information determines that the distance from mobile device 100 to vehicle 1 exceeds the permissible distance (NO in S109), an instruction to approach the vehicle is displayed on the operating screen (S110), and the processing returns to S102. An instruction to capture an image of the vehicle is then displayed to user 2 of vehicle 1 (S102), and the subsequent steps (S102 to S109) are repeated. Whether the vehicle is suitable and whether the distance to the vehicle is equal to or less than the permissible distance is determined using the same vehicle image. Consequently, if user 2 of vehicle 1 approaches the vehicle and the image capture is performed again, the determination of whether the vehicle is suitable is repeated.

[0051] If it is determined that the distance to vehicle 1 is equal to or less than the permissible distance (YES in S109), while the image acquisition, extraction, and determination steps (S102 to S109) are repeated in this manner, the processing proceeds to a flowchart that is shown in Fig. Figure 6 shows the procedure for remote control, as the condition for allowing remote control of vehicle 1 is met. At this time, the operating screen displays, for example, a message indicating that remote control is enabled, or prompts user 2 of vehicle 1 to start operation for user 2 of vehicle 1, and user 2 activates icons, buttons, or the like displayed on the operating screen, or activates operating buttons 120a to 120d to start remote control.

[0052] Fig. 6 shows one to Fig. 5. The following flowchart is a flowchart for executing remote control after remote control has been authorized. When remote control is authorized, user 2 of vehicle 1 determines a direction of travel and a steering direction to move vehicle 1 from its current position towards parking space 3 and activates operating section 112 according to the desired content. In other words, user 2 of vehicle 1 activates the icons, buttons, or similar on the operating screen, or activates operating buttons 120a to 120d. As a result, the operating content is captured by operating content capture section 111 (S112).

[0053] The detected operating content is then determined (S113). If the operating content is not the end of the remote control (NO in S113), because the operation intended to move vehicle 1 is being carried out, the operating content detection section 111 generates an operating signal according to the operating content and outputs the operating signal to the operating signal transmission section 110, which then transmits an operating signal corresponding to the operating content detected by the operating signal transmission section 110 to vehicle 1 (S114). For example, if the operating button 120a is pressed, an operating signal to steer left is transmitted to vehicle 1, and the wheels are steered to the left in a stationary manner.When operating buttons 120a and 120b are pressed simultaneously, the operating signals for steering left and moving forward are transmitted to vehicle 1 at the same time, and vehicle 1 is steered to the left while moving forward. When operating button 120c is pressed, an operating signal for reversing is transmitted to vehicle 1, and vehicle 1 moves backward. When button 123 is tapped for automatic parking, vehicle 1 moves backward straight ahead toward parking space 3. With regard to operating modes, other modes of operation besides those shown, which are similar to these modes, an operating signal corresponding to the pressed operating button 120, or the like, is transmitted, and vehicle 1 moves, or the like.

[0054] When the operating signal is transmitted as described above and vehicle 1 moves, the distance to the mobile device 100 changes, and consequently, it is confirmed that the distance is maintained within the permissible range for remote control. This confirmation processing is a procedure necessary to ensure safety and is performed, for example, by analyzing a captured image and acquiring distance information in the same way as when remote control is started. In other words, after the operating signal has been transmitted, the image processing section 102 retrieves the image captured by the imaging section 101 at that time (S115), extracts the image of the vehicle from the captured image, and further analyzes the image of the vehicle to acquire the distance information (S116). The acquisition result is then, for example,This is indicated by updating the position of the black triangle marker on the operating screen or by a text message such as "within the operating range". Next, the captured distance information is used to determine whether the distance to vehicle 1 is equal to or less than the permissible distance (S117). If the distance is less than the permissible distance (YES in S117), the processing returns to S112, the content of the operation in operating section 112 is recaptured, and the following steps are repeated. However, if it is determined that the current distance is not within the permissible distance (NO in S117), for example, because vehicle 1 is moving away from user 2, an instruction to approach the vehicle is displayed on the operating screen (S118), and the processing then proceeds to recapture the captured image (S115), and the following steps are repeated.

[0055] Steps S112 to S118 are repeated as described above, with remote control being carried out while maintaining the distance between the mobile device 100 and the vehicle 1, based on the sequentially acquired images, such that it is equal to or less than the permissible distance. Therefore, remote control can be carried out safely while the environmental conditions of the vehicle 1 are verified.

[0056] When the vehicle image is analyzed (S116) after transmitting an operating signal (S114), if no vehicle is present in the image, this fact is displayed on the display device 104 and the remote control is automatically terminated, for example, or the processing returns to "Instruct the operator to take the vehicle image" (S102 in). Fig. 5) and it is determined again whether the vehicle is a suitable vehicle. If the remote control is automatically terminated, user 2 of vehicle 1 repeats the remote control start process, if necessary.

[0057] If it is determined that the operating content is the end instruction of the remote control during processing to repeat the above steps (YES in S113), the end instruction signal of the remote control function is transmitted from the operating signal transmission section 110 to vehicle 1 (S119). For example, if user 2 of vehicle 1 can move vehicle 1 into the designated parking position, user 2 performs an operation, such as tapping the operating end button 124 provided on the operating screen. As a result, the end of the remote control is detected by the operating content detection section 111, and the processing described above is executed. Vehicle 1, having received the end instruction signal, terminates the remote control function. After issuing an instruction to terminate the remote control function to vehicle 1, the mobile device 100 terminates the remote control application. A response signal of the end instruction can be, for example, a ping.The signal is sent back from vehicle 1 to the mobile device, and the mobile device can terminate the remote control application after the response signal is acknowledged.

[0058] Although each of the above steps was described in the case where vehicle 1 is parked by remote control, even if vehicle 1 is moved from the parking position, the same remote control can be performed without any differences compared to the case where the vehicle is parked, except that the direction of movement and the like of the vehicle are different.

[0059] The mobile device 100 according to the present embodiment obtains, for example, distance information about the distance to the vehicle 1 based on the captured image. If the distance between the mobile device and the vehicle is within the permissible range, remote control is enabled and the operating signal is transmitted from the mobile device 100 to the vehicle 1. Therefore, if the user 2 of the vehicle 1 is carrying the mobile device 100, it can be determined whether the user 2 is near the vehicle 1. Consequently, even if the user 2 is not carrying the electronic key, the vehicle 1 can be moved remotely using the mobile device 100 if the user 2 is present around the vehicle 1.

[0060] The mobile terminal 100 according to the present embodiment has the display device 104 for displaying the captured image and the operating button 120, which is arranged below the display device 104 and configures the operating section 112 on the same surface. The operating data is captured in the operating data acquisition section 111, and the operating signal is transmitted from the operating signal transmission section 110 to the vehicle 1. Since remote control can be carried out while the vehicle 1 being operated can be visually confirmed by the display device 104, operation can therefore be carried out more safely.

[0061] The mobile device 100 according to the present embodiment additionally stores the license plate number in the vehicle identification information storage section 106 to identify the vehicle to be operated, extracts the vehicle identification information from the vehicle image in the recorded image, determined by section 105 to determine whether the vehicle is a suitable vehicle that can be operated, and if the distance from the mobile device 100 to the vehicle 1 is within the permissible distance and it is determined that the vehicle is a suitable vehicle, the remote control signal is transmitted to the vehicle 1.This makes it possible to prevent remote control of vehicle 1 without incorrectly capturing the image of the vehicle being operated, and to determine, based on the captured image, whether the distance to the vehicle is within the permissible range and whether the distance to the target vehicle 1 is appropriate. Since vehicle 1 can only be operated by the mobile device 100, which is authenticated with the vehicle, the other vehicles cannot be remotely controlled. Furthermore, since user 2 of vehicle 1 can perform remote control while the target vehicle is confirmed by the display device 104 and visually, user 2 can instantly notice another vehicle displayed on the display device 104 and change the image capture target.Although the safety of the remote control is sufficiently ensured, even if the determination of whether the vehicle is a suitable vehicle is not carried out, if the determination is carried out beforehand, a higher level of safety can consequently be obtained.

[0062] Fig. Figure 7 shows a diagram that represents another example of the internal structure of the mobile device 100, and corresponds to Fig. 2. Sections, which are the same as in Fig. If there are two, the same reference symbol is assigned and a description of the common parts is omitted.

[0063] In the present embodiment, an automatic focusing section 126 is provided, and the imaging section 101 is provided with a lens drive mechanism for changing the focal length. The automatic focusing section 126 is provided with an automatic focusing function of a well-known passive method, such as a phase-difference method, an image plane phase-difference method, or a contrast method, and acquires an image from the imaging section 101, analyzes the acquired image to calculate a focal length adjustment, and outputs a focus position change signal to the imaging section 101. The lens drive mechanism is operated such that the imaging section 101 changes a focus position by extending or retracting a lens according to a focus position change signal and focuses the lens on the vehicle 1.When focusing on vehicle 1 occurs in this manner, the automatic focusing section 126 detects the actual lens expansion quantity in the imaging section 101 and outputs an expansion quantity signal to the distance information Erlang section 107. Alternatively, instead of the expansion quantity signal, the focus position change signal output by the automatic focusing section 126 to the imaging section 101 at the time of focusing can be output to the distance information Erlang section 107. The distance information Erlang section 107 calculates the focal length at the time of focusing, that is, the distance from the mobile device 100 to vehicle 1, based on the received expansion quantity signal or the focus position change signal, and outputs the distance information signal to the determining section 109. As in the case described in . Fig. As shown in Figure 2, the determining section 109 determines whether the vehicle is within the permissible distance, together with the determination of whether the vehicle is a suitable vehicle. In the present embodiment, the method for obtaining the distance information differs from that in the embodiment described above, and it is not necessary to pre-enter the vehicle dimension information in the vehicle dimension information storage section 108.

[0064] Fig. Figure 8 is a diagram showing a processing flow that corresponds to the internal structure. The processing flow corresponds to Fig. 5 and a section that differs from Fig. The process, which differs in part from section 5, is essentially described below. When a vehicle image capture is instructed to user 2 of vehicle 1 (S102), when the image section 101 of the mobile device 100 is pointed towards vehicle 1 and image capture is initiated, the autofocus section 126 processes the image data sequentially sent by the image section 101, changes the focus position of the image data, and instructs the image section to focus on the vehicle (S120). Next, the image long section 102 obtains a captured image (S103) and performs the extraction of the vehicle from the captured image (S104) and the subsequent steps.

[0065] If, based on the analysis result of the vehicle image, it is determined that the vehicle is a suitable vehicle (YES in S107), the distance information acquisition section 107 obtains the distance information from the mobile device 100 to the vehicle 1 based on a lens expansion quantity signal or a focus position change signal sent by section 126 for autofocus, that is, the focus position information when the captured image, which serves as a source of the vehicle image, is obtained (S121). In the processing flow described above, the vehicle image is not used for distance information as described above; instead, the distance information obtained based on the focus position information output by section 126 for autofocus at the time of focusing is used.

[0066] In the present embodiment, remote control processing is also carried out according to the same procedure as described in Fig. 6 is executed. At that time, instead of capturing the distance information (S116), the distance from the mobile device to the vehicle 1 is calculated based on the information obtained from section 126 for automatic focusing, in the same way as the “calculation of the distance from the focal length information to the vehicle” (S121) shown in Fig. 8 is shown, obtained.

[0067] Fig.Figure 9 shows a display example of another embodiment of the mobile device 100. In this display example, a distance display section 128 is provided in an upper section of an operating screen instead of the distance display section 122. In the distance display section 128, a dashed line is displayed between a wide bar for the permissible distance and "near" on the left side of the wide bar for the permissible distance. The dashed line represents an area outside a permissible range of remote control, as in the case of the "far" side, and in this example, remote control is prevented if the distance between the mobile device 100 and the vehicle 1 is equal to or less than a predefined distance.

[0068] If, for example, user 2 gets too close to vehicle 1, user 2 could come into contact with vehicle 1 if an operating direction is faulty. For this reason, in the present embodiment, a lower limit value is also determined in addition to the upper limit value of the permissible distance in determination section 109, i.e., in the processing sequence. Furthermore, the determination of whether the distance to the vehicle is within the permissible distance is specified in sections 109 and 117. Specifically, the determination of whether the distance to the vehicle is within the permissible distance is defined as being within both the upper and lower limits.

[0069] A flowchart or a processing of the flowchart described in the present invention comprises several sections (or steps), and each section is expressed, for example, as S100. Furthermore, each section can be divided into several subsections, while the several sections can be combined into a single section. Additionally, each section configured in this way can be referred to as a circuit, a device, a module, or a means.

[0070] Each or a combination of the several sections can also be implemented as (i) a software section in combination with a hardware section (e.g., a computer), and as (ii) a hardware section (e.g., an integrated circuit, a wired logic circuit) with or without the functionality of the associated device. Furthermore, the hardware section can be configured within a microcomputer.

[0071] Although the present invention has been described according to the embodiments, it should be understood that the present invention is not limited to such embodiments or structures. The present invention encompasses various modifications and variants within the range of equivalence. Additionally, various combinations and forms, as well as other combinations and forms comprising more or fewer elements or only a single element, are also included within the scope and concept of the present invention.

Claims

[1] Mobile terminal (100) intended to be carried by a user (2) of a vehicle (1) and capable of moving the vehicle (1) by remote control by transmitting an operating signal corresponding to operation by the user (2) to the vehicle (1), wherein the mobile terminal (100) comprises: a long image section (102) which is connected to an image section (101) which takes an image of the vehicle (1) and obtains a taken image of the image section (101), an image display section (103) that displays the captured image, a distance information acquisition section (107) that detects the vehicle (1) in the recorded image and obtains distance information about a distance to the vehicle (1) based on the recorded image, a determining section (109) which, based on the distance information, determines whether the distance to the vehicle (1) is within a specified permissible distance, an operating content capture section (111) which is connected to an operating section (112) which is provided so that it can be operated by the user (2) while checking a display of the captured image, and captures an operating content of the operating section (112), and an operating signal transmission section (110) which transmits the operating signal corresponding to the operating content to the vehicle (1) when it is determined that the distance to the vehicle (1) is within the permissible distance, wherein Even after the operating signal transmission section (110) has transmitted the operating signal to the vehicle (1), if the vehicle (1) is no longer detected in the recorded image, or if it is determined that the distance to the vehicle (1) is outside the permissible distance, the operating signal transmission section (110) stops transmitting the operating signal corresponding to the operating content until it is determined that the distance to the vehicle (1) is within the permissible distance. The image display section (103) displays, in addition to the captured image, the distance information about the vehicle (1) that is captured in the captured image and information on whether the distance information is within the permissible distance. [2] Mobile terminal (100) according to claim 1, wherein the distance information acquisition section (107) acquires the distance information based on the size of the vehicle (1) in the recorded image. [3] Mobile terminal (100) according to claim 1, further comprising: a section (126) for automatic focusing which causes the image section (101) to focus on the vehicle (1) by analyzing the captured image, and changes a focus position of the image section (101), wherein The distance information acquisition section (107) obtains the distance information based on information about the focus position of the image section (101), which is modified by the section (126) for automatic focusing. [4] Mobile terminal (100) according to any one of claims 1 to 3, further comprising: a vehicle identification information storage section (106) which stores vehicle identification information for identifying a suitable vehicle to be remotely controlled by the user (2), and a section (105) for determining the suitable vehicle, which determines whether the vehicle (1) is the suitable vehicle by extracting the vehicle identification information from the image of the vehicle (1) in the recorded image, wherein The operating signal transmission section (110) transmits the operating signal when it is determined that the distance to the vehicle (1) is within the permissible distance, and when it is determined that the vehicle (1) is the suitable vehicle. [5] Mobile terminal (100) according to claim 2, which further comprises a vehicle dimension information storage section (108) which stores vehicle dimension information about a vehicle dimension of the suitable vehicle to be remotely controlled by the user (2), the distance information acquisition section (107) obtains the distance information based on the size of the vehicle (1) in the captured image and the vehicle dimension information. [6] Mobile terminal (100) according to any one of claims 1 to 5, further comprising: a receiving section (113) that receives a predefined authentication request signal transmitted by the vehicle (1), and an authentication signal transmission section (114) which transmits a predefined authentication signal, which is stored in advance, after the authentication request signal is received. [7] Remote control method for moving a vehicle (1) by remote control using a mobile terminal (100) carried by a user (2) of the vehicle (1), wherein the remote control method comprises: Obtaining a captured image of the vehicle (1) from an image section (101) attached to the mobile device and displaying the captured image (S103), Detection of the vehicle (1) in the recorded image and obtaining distance information about a distance to the vehicle (1) (S108), Determine whether a distance to the vehicle (1) is within a specified permissible distance, based on the distance information (S109), Displays, in addition to the captured image, the distance information about the vehicle (1) captured in the captured image, and information on whether the distance information is within the permissible distance, Capturing operational content that is entered into an operational section (112) which is provided so that it can be operated by the user (2) while checking a display of the captured image (S112), and Transmitting an operating signal corresponding to the operating content to the vehicle (1) when it is determined that the distance to the vehicle (1) is within the permissible distance (S114), wherein Even after the operating signal has been transmitted to the vehicle (1), if the vehicle (1) is no longer captured in the recorded image, or if it is determined that the distance to the vehicle (1) is outside the permissible distance, the transmission of the operating signal corresponding to the operating content will be stopped until it is determined that the distance to the vehicle (1) is within the permissible distance.

Images