Mobile support system

Abstract

To provide technique enabling accurate detection of positional deviation of an infrastructure camera.SOLUTION: The present disclosure relates to a mobile object assisting system assisting movement of a mobile object in a predetermined area in which one or more markers are arranged. The mobile object assisting system acquires an image captured by a camera installed in the predetermined area. The camera includes a first camera and a second camera that are configured to capture a same marker from different positions. The mobile object assisting system compares a first image acquired this time by the first camera with a first image acquired previous time to detect a change in position of the same marker. The mobile object assisting system compares a second image acquired this time by the second camera with a second image acquired previous time to detect a change in position of the same marker. When change in the position of the same marker is detected from the first image and the change in position of the same marker is not detected from the second image, the mobile object assisting system determines that the first camera is positionally deviated. The camera is an infrastructure camera.SELECTED DRAWING: Figure 6

Description

Technical Field

[0005] ,

[0001] The present disclosure relates to a mobile body support system.

Background Art

[0002] Patent Document 1 discloses an image processing apparatus for accurately determining the presence of a moving object area where a moving object exists. According to this technology, a moving object area is detected from each of a plurality of camera images and converted into a moving object area in a specified plane coordinate system. Then, the overlap of each moving object area is detected, and the presence of the moving object area in the real space is determined based on the detection result of the overlap. [[ID=***翻译内容较多,请耐心等待。以下是完整翻译内容***]]

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Techniques for estimating the position of a moving object using an image captured by an infrastructure camera are known. The result of the position estimation can be used, for example, for controlling the moving object. However, at this time, if the position of the camera is shifted, an error occurs in the position estimation, and there is a possibility that the control of the moving object cannot be performed correctly. Therefore, when the camera is misaligned, it is required to accurately detect the misalignment before or during control. One object of the present disclosure is to provide a technique capable of accurately detecting the misalignment of an infrastructure camera.

Means for Solving the Problems

[0005] One aspect of this disclosure relates to a mobile support system for assisting the movement of a mobile object in a predetermined area where one or more markers are placed. The mobile support system comprises one or more processors. The one or more processors acquire images captured by cameras installed in the predetermined area. The cameras include a first camera and a second camera that capture the same marker from different positions. The one or more processors compare a first image acquired from the first camera with a first image acquired previously to detect a change in the position of the same marker, and compare a second image acquired from the second camera with a second image acquired previously to detect a change in the position of the same marker. If a change in the position of the same marker is detected from the first image, and no change in the position of the same marker is detected from the second image, the processors determine that the first camera has shifted position. The cameras are infrastructure cameras. [Effects of the Invention]

[0006] According to this disclosure, it is possible to accurately detect misalignment of infrastructure cameras. [Brief explanation of the drawing]

[0007] [Figure 1] This is a diagram illustrating the outline of the first embodiment. [Figure 2] This figure shows an example configuration of a mobile support system according to the first embodiment. [Figure 3] This flowchart shows an example of the processing flow according to the first embodiment. [Figure 4] This is a diagram to explain the amount of change in position. [Figure 5] This flowchart shows an example of a processing flow according to the second embodiment. [Figure 6] This is a diagram illustrating the outline of the third embodiment. [Modes for carrying out the invention]

[0008] Embodiments of this disclosure will be described with reference to the attached drawings.

[0009] 1. Overview We consider a control device that controls a moving object in a predetermined area where markers are placed. An example of this predetermined area is a parking lot. The parking lot may also be an AVP parking lot. Examples of the moving object include vehicles and robots. As an example, the following explanation will consider the case where the moving object is a vehicle.

[0010] Figure 1 shows an example of a designated area AR. Markers M are placed in the designated area AR. Markers M are markers that have information about their location within the designated area AR. Camera 20 is an infrastructure camera installed on the infrastructure and captures images of the designated area AR, including markers M. In Figure 1, the area captured by camera 20 is shown by a dashed line. The control device acquires the image 21 captured by camera 20 and estimates the position of the vehicle 200 based at least on the image 21. Furthermore, the control device 10 can control the movement of the vehicle 200 within the designated area AR based on the estimated position of the vehicle 200.

[0011] When using position estimation results obtained from images 21 from the infrastructure camera 20 in this manner, if the camera 20 is misaligned, errors may occur in the position estimation, potentially preventing the vehicle 200 from being controlled correctly. Therefore, it is necessary to accurately detect any misalignment of the camera 20 before or during vehicle control.

[0012] Therefore, the first embodiment provides a technology that can accurately detect the misalignment of the camera 20. Markers M are used to detect the misalignment. In the first embodiment, the camera 20 captures an image 21 that includes multiple markers M. When the control device 10 acquires a new image 21 from the camera 20, it compares it with an image 21 acquired before the new image 21. The former is referred to as the "currently acquired image 21P" or simply "image 21P," and the latter as the "previously acquired image 21R" or simply "image 21R." The control device 10 then determines whether the camera 20 has misaligned based on the number of markers M whose position has changed between image 21P and image 21R. Hereinafter, markers M whose position has changed between image 21P and image 21R are referred to as "position change markers." The control device 10 determines that the camera 20 has misaligned only when there are a predetermined number or more position change markers, and does not determine that the camera 20 has misaligned when the number of position change markers is less than the predetermined number.

[0013] In the first embodiment, the camera 20 captures an image 21 containing multiple markers M for the following reasons: Even if the position of one marker M changes between image 21P and image 21R, it cannot be definitively concluded that this is due to a displacement of the camera 20. For example, the actual position of the marker M may have shifted due to vibration or shock within a predetermined area AR, or the marker M may have become unrecognizable due to dirt or smudging. Therefore, it is necessary to distinguish whether the presence of a position-changed marker is due to a displacement of the camera 20 or to an event that occurred to the marker M.

[0014] In the first embodiment, by imaging the range including a plurality of markers M with the camera 20, it becomes easy to distinguish whether the cause of the displacement of the marker M in the image 21 is the displacement of the camera 20 or an event that has occurred to the marker M. When the positions of many markers M in the image 21 have changed, it is highly likely that it is due to the displacement of the camera 20. Conversely, when there are few markers M whose positions have changed in the image 21, it is highly likely that it is due to an event that has occurred to the marker M. Therefore, by imaging a plurality of markers M with the camera 20 and determining that the displacement of the camera 20 has occurred only when the number of markers with position changes is equal to or more than a predetermined number, the accuracy of detecting the displacement of the camera 20 can be improved.

[0015] 2. Configuration example FIG. 2 is a block diagram showing a configuration example of the mobile body support system 100 according to the first embodiment. The mobile body support system 100 includes a control device 10, a camera 20, and a communication device 30.

[0016] The camera 20 is an infrastructure camera installed in the infrastructure. The camera 20 images a predetermined area AR so as to include a plurality of markers M. The image 21 obtained by imaging is acquired by the control device 10 and temporarily stored in a storage device 120 described later.

[0017] The communication device 30 communicates with the outside of the mobile body support system 100. The connection destination of the communication device 30 includes at least the vehicle 200.

[0018] The control device 10 supports the movement of the vehicle 200 in a predetermined area AR. The support for the movement of the vehicle 200 performed by the control device 10 includes the position estimation of the vehicle 200 and the control of the movement of the vehicle 200. The support for the movement of the vehicle 200 may further include transmitting the estimated position of the vehicle 200 to the vehicle 200. In addition, the control device 10 determines the displacement of the camera 20 when or before performing the position estimation of the vehicle 200.

[0019] The control device 10 estimates the position of the vehicle 200 based at least on the image 21. The position estimation of the vehicle 200 based on the image 21 can be performed by converting the position of the vehicle 200 in the image 21 into the position of the vehicle 200 in the real space. Further, the control device 10 may perform position estimation using the sensor information of the vehicle 200 in addition to the image 21. Examples of the sensor information include an image captured by an in-vehicle camera, acceleration information of the vehicle 200, steering angle information, and the like. The sensor information is acquired from sensors mounted on the vehicle 200 such as an in-vehicle camera, a wheel speed sensor, a G sensor, a pinion angle sensor, a yaw rate sensor, and the like. The control device 10 can acquire the sensor information by communicating with the vehicle 200 through the communication device 30.

[0020] The control device 10 includes one or more processors 110 (hereinafter simply referred to as the processor 110) and one or more storage devices 120 (hereinafter simply referred to as the storage device 120). The processor 110 executes various processes. For example, the processor 110 includes a CPU (Central Processing Unit). The storage device 120 stores various information and various programs. Examples of the storage device 120 include a volatile memory, a non-volatile memory, an HDD (Hard Disk Drive), an SSD (Solid State Drive), and the like. The processor 110 can execute various processes including determination of the misalignment of the camera 20 by executing the program stored in the storage device 120. Further, the various information stored in the storage device 120 includes image information regarding the image 21.

[0021] 3. Example of processing flow FIG. 3 is a flowchart showing an example of a process performed by the mobile body support system 100 (processor 110) to detect the misalignment of the camera 20. A series of processes are performed before or during the mobile body support system 100 supports the vehicle 200. Further, a series of processes are realized by the processor 110 executing the program stored in the storage device 120.

[0022] In step S101, the mobile support system 100 acquires the newly captured image 21P from the camera 20. The acquired image 21P is temporarily stored in the storage device 120.

[0023] In step 102, the mobile support system 100 detects position change markers by comparing the image 21P acquired this time with the image 21R acquired previously from the camera 20. Image 21R is the most recent image among the images 21 that were acquired from the camera 20 and stored in the storage device 120 before image 21P. The position change markers are the markers M in image 21R whose position has changed in image 21P.

[0024] In step S103, the mobile support system 100 determines whether the number of position change markers detected in step S103 is greater than or equal to a predetermined number. If the number of position change markers is greater than or equal to the predetermined number (step S103; Yes), the process proceeds to step S104. On the other hand, if there are fewer than the predetermined number of position change markers, or if there are no position change markers at all (step S103; No), the process proceeds to step S105. The predetermined number is a number used as a criterion for determining the positional displacement of the camera 20. The predetermined number is set in advance so that it is 2 or more and less than or equal to the number of markers M included in the image 21. For example, the predetermined number may be set to be the same as the number of markers M included in the image 21. In this case, the positional displacement of the camera 20 will only be detected when the position of all markers M included in the image 21 has changed.

[0025] In step S104, the mobile support system 100 determines that a positional shift has occurred in the camera 20.

[0026] In step S105, the mobile support system 100 determines that no positional displacement has occurred in the camera 20.

[0027] If the series of processes is completed and a result is obtained indicating that no positional displacement of the camera 20 has occurred, the mobile support system 100 will provide support for the movement of the vehicle 200 in the same manner as under normal circumstances. If a result is obtained indicating that a positional displacement of the camera 20 has occurred, the mobile support system 100 may perform the following control actions on the vehicle 200 and the camera 20.

[0028] The first issue is that it reduces the reliability of the vehicle 200's position estimation based on image 21. In other words, the mobile support system 100 prioritizes results obtained by other methods over detection results based on image 21 when estimating the vehicle 200's position. For example, if there are other infrastructure cameras that image a predetermined area AR, it is assumed that the system will prioritize using images obtained from those other infrastructure cameras for the range that those cameras can image to estimate the vehicle 200's position. Alternatively, it is assumed that sensor information obtained from the vehicle 200 will be used for estimating the vehicle 200's position with priority over image 21.

[0029] The second option is to stop controlling the vehicle 200. For example, if the camera 20 is significantly misaligned, making it difficult to estimate the vehicle's position from information other than the image 21, then stopping control of the vehicle 200 is conceivable.

[0030] The third step is to calibrate the camera 20. Calibration involves correcting the correspondence between position coordinates in image 21 and position coordinates in real space, and is performed using markers M. The mobile support system 100 corrects the position coordinates in image 21 based on the position of each marker M in image 21 and the position information that each marker M possesses. By performing this calibration, the system can continue to estimate the position of the vehicle 200 using image 21 and provide support for its movement.

[0031] In addition, there are cases where the calibration of the camera 20 is deemed to have failed. Calibration failure can occur, for example, when the positional displacement of the camera 20 exceeds a correctable amount, or when positional information cannot be obtained from at least a portion of the marker M that should be included in the image 21. Inability to obtain positional information from the marker M can occur, for example, when the positional displacement of the camera 20 is large enough that the marker M is outside the camera 20's field of view, or when the marker M becomes unreadable due to dirt. Therefore, the mobile support system 100 may first perform calibration of the camera 20 and stop controlling the vehicle 200 when it is determined that the calibration has failed. Alternatively, when it is determined that the calibration has failed, the system may notify the administrator of a designated area AR and prompt them to correct the position of the camera 20 or clean the camera 20's lens and marker M.

[0032] 4. Second Embodiment In the second embodiment, the configuration example and the point that a positional shift of the camera 20 is determined only when the number of positional change markers is greater than or equal to a predetermined number are the same as in the first embodiment. In the second embodiment, the "amount of positional change" is also acquired.

[0033] Figure 4 is a diagram illustrating the amount of position change. The top image is image 21P, acquired from camera 20 this time, and the bottom image is image 21R, acquired from camera 20 last time. Images 21P and 21R contain five markers M1, M2, M3, M4, and M5. Of these, three markers, M3, M4, and M5, are position change markers. In image 21P, the positions of markers M3, M4, and M5 before the position change, i.e., in image 21R, are shown by dotted lines. In addition, the displacement distances D3, D4, and D5 of each position change marker M3, M4, and M5 as seen in image 21P are shown by double-headed arrows.

[0034] The amount of position change is the sum of the distances traveled by the position change markers included in image 21. For example, the amount of position change may be the average value or the maximum value of the distances traveled by all position change markers included in image 21.

[0035] In the second embodiment, when the mobile support system 100 determines that a positional displacement has occurred in the camera 20, it changes the control to the vehicle 200 and the camera 20 according to the magnitude of the positional change. Figure 5 is a flowchart of an example of the process in the second embodiment. The process up to step S104 is the same as the flowchart in Figure 3 and is therefore omitted.

[0036] After it is determined in step S104 that there is a positional shift of the camera 20, the process proceeds to step S106, where the mobile support system 100 determines whether the amount of positional change is less than or equal to a predetermined amount. If the amount of positional change is less than or equal to the predetermined amount (step S106; Yes), the process proceeds to step S107. If the amount of positional change is greater than the predetermined amount (step S106; No), the process proceeds to step S108. The predetermined amount is a preset amount.

[0037] In step S107, the mobile support system 100 reduces the reliability of the position estimation based on the image 21.

[0038] In step S108, the mobile support system 100 stops controlling the vehicle 200.

[0039] In the example shown in Figure 5, when the amount of position change is small, control of the vehicle 200 continues, only reducing the reliability of the position estimation based on information other than image 21. On the other hand, when the amount of position change is large, control of the vehicle 200 is stopped.

[0040] Thus, in the second embodiment, the control is changed according to the magnitude of the position change. This allows for more appropriate control of the vehicle 200 according to the situation. Note that the control selected according to the magnitude of the position change is not limited to the control illustrated in Figure 5. For example, the mobile support system 100 may perform calibration of the camera 20 when the position change is greater than a predetermined amount.

[0041] As described above, in the first and second embodiments, the camera 20 captures an image 21 that includes a plurality of markers M. Then, the camera 20's positional displacement is detected based on the number of positional change markers. In this way, the accuracy of detecting the camera's positional displacement can be improved.

[0042] 5. Third Embodiment A third embodiment will now be described. The configuration example of the mobile support system 100 is the same as that shown in Figure 2. However, it differs from the first and second embodiments in that the camera 20 includes multiple cameras. It also differs from the first and second embodiments in that the markers M included in the image 21 captured by each camera 20 do not necessarily have to be multiple.

[0043] Figure 6 shows an example of multiple cameras included in camera 20, namely a first infrastructure camera (first camera) 20-1 and a second infrastructure camera (second camera) 20-2. The first camera 20-1 and the second camera 20-2 are installed in different locations and capture the first image 21-1 and the second image 21-2 so as to include the same marker M0. Although Figure 6 shows an example where there is one marker M0, there may be multiple markers M0.

[0044] In the third embodiment, the detection of the displacement of the first camera 20-1 is performed as follows. First, the mobile support system 100 acquires a new first image 21-1P captured by the first camera 20-1. Then, it compares the newly acquired first image 21-1P with the previously acquired first image 21-1R to detect a change in the position of the marker M0 within the first image 21-1. The previously acquired first image 21-1R is the most recent image of the first image 21-1 that was acquired from the first camera 20-1 before the first image 21-1P and stored in the storage device 120.

[0045] Furthermore, the mobile support system 100 acquires a new second image 21-2P captured by the second camera 20-2. It then compares the newly acquired second image 21-2P with the previously acquired second image 21-2R to detect changes in the position of marker M0 within the second image 21-2. The previously acquired second image 21-2R is the most recent image of the second image 21-2 that was acquired by the second camera 20-2 before the second image 21-2P and stored in the storage device 120.

[0046] The mobile support system 100 determines that the first camera 20-1 has shifted position if a change in the position of marker M0 is detected from the first image 21-1, and no change in the position of marker M0 is detected from the second image 21-2. The mobile support system 100 does not determine that the first camera 20-1 has shifted position if a change in the position of marker M0 is detected from either the first image 21-1 or the second image 21-2.

[0047] In the third embodiment, instead of multiple markers M, multiple cameras 20 are used to detect the displacement of the cameras 20. As described in the first embodiment, if there is a marker M whose position changes within the image 21, the cause of the change in the position of the marker M within the image 21 could be either a displacement of the cameras 20 or an event that occurred with the marker M. If a marker M0 whose position changes within the first image 21-1 does not change within the second image 21-2, then the cause of the change in the position of the marker M0 within the first image 21-1 is likely to be a displacement of the first camera 20-1. Conversely, if the position of the same marker M0 changes in all of the multiple images 21 captured by the multiple cameras 20, then the cause of the change in the position of the marker M0 is likely to be an event that occurred with the marker M0. Therefore, by using multiple cameras 20 in this way to detect the displacement of the cameras 20, the accuracy of the displacement detection can be improved.

[0048] In the third embodiment as well, the control by the mobile support system 100 may be changed according to the amount of position change. In this case, the amount of position change can be set as the distance traveled by marker M0 within the first image 21-1. Furthermore, if there are multiple markers M0 and multiple markers M0 have changed positions within the first image 21-1, the amount of position change may be set as the average value of the respective distances traveled, or as the maximum value. [Explanation of Symbols]

[0049] 10 Control device 20 cameras 21 images 30 Communication equipment 100 Mobile Support Systems 110 processors 120 Storage device 200 vehicles M Marker

Claims

[Claim 1] A mobile support system that assists the movement of a mobile object in a predetermined area where one or more markers are placed, Equipped with one or more processors, The one or more processors described above are: Images captured by infrastructure cameras installed in the predetermined area are acquired. The infrastructure camera includes a first infrastructure camera and a second infrastructure camera that image the same marker from different positions. The first image acquired this time from the first infrastructure camera is compared with the first image acquired last time to detect changes in the position of the same marker. By comparing the second image acquired this time from the second infrastructure camera with the second image acquired last time, a change in the position of the same marker is detected. If a change in the position of the same marker is detected in the first image, and no change in the position of the same marker is detected in the second image, then it is determined that the first infrastructure camera has shifted position. If it is determined that the first infrastructure camera has shifted position, and the amount of change in the position of the same marker in the first image is greater than a predetermined amount, the control of the moving object is stopped. Mobile support system.

Images