Mobile object assisting system

Abstract

To provide technique enabling accurate detection of positional deviation of an infrastructure camera.SOLUTION: The present disclosure relates to a mobile object assisting system assisting movement of a mobile object in a predetermined area in which one or more markers are arranged. The mobile object assisting system acquires an image captured by a camera installed in the predetermined area. The camera includes a first camera and a second camera that are configured to capture a same marker from different positions. The mobile object assisting system compares a first image acquired this time by the first camera with a first image acquired previous time to detect a change in position of the same marker. The mobile object assisting system compares a second image acquired this time by the second camera with a second image acquired previous time to detect a change in position of the same marker. When change in the position of the same marker is detected from the first image and the change in position of the same marker is not detected from the second image, the mobile object assisting system determines that the first camera is positionally deviated. The camera is an infrastructure camera.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present disclosure relates to a mobile body support system.

Background Art

[0002] Patent Document 1 discloses an image processing apparatus for accurately determining the presence of a moving object region where a moving object exists. According to this technique, a moving object region is detected from each of a plurality of camera images and converted into a moving object region in a specified plane coordinate system. Then, the overlap of each moving object region is detected, and the presence of the moving object region in the real space is determined based on the detection result of the overlap.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Techniques for estimating the position of a moving object using an image captured by an infrastructure camera are known. The result of the position estimation can be used, for example, for controlling the moving object. However, at this time, if the position of the camera is shifted, an error may occur in the position estimation, and the control of the moving object may not be performed correctly. Therefore, when the camera is misaligned, it is required to accurately detect the misalignment before or during control. One object of the present disclosure is to provide a technique capable of accurately detecting the misalignment of an infrastructure camera.

Means for Solving the Problems

[0005] One aspect of the present disclosure relates to a mobile body support system that supports the movement of a mobile body in a predetermined area where one or more markers are arranged. The mobile body support system includes one or more processors. The one or more processors acquire an image captured by a camera installed in the predetermined area. The camera includes a first camera and a second camera that image the same marker from different positions. The one or more processors compare the first image acquired this time from the first camera with the first image acquired last time to detect a change in the position of the same marker, and compare the second image acquired this time from the second camera with the second image acquired last time to detect a change in the position of the same marker. If a change in the position of the same marker is detected from the first image and no change in the position of the same marker is detected from the second image, it is determined that the first camera is misaligned. The camera is an infrastructure camera.

Effects of the Invention

[0006] According to the present disclosure, misalignment of the infrastructure camera can be accurately detected.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Modes for Carrying Out the Invention

[0008] Embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0009] 1. Summary Consider a control device that controls a moving object in a predetermined area where markers are arranged. As an example of the predetermined area, a parking lot is exemplified. The parking lot may be an AVP parking lot. Examples of the moving object include a vehicle and a robot. As an example, in the following description, the case where the moving object is a vehicle will be considered.

[0010] FIG. 1 shows an example of a predetermined area AR. Markers M are arranged in the predetermined area AR. The marker M is a landmark having information about the position within the predetermined area AR. The camera 20 is an infrastructure camera installed in the infrastructure, and images the image within the predetermined area AR so as to include the marker M. In FIG. 1, the range imaged by the camera 20 is indicated by a dashed line. The control device acquires the image 21 imaged by the camera 20 and estimates the position of the vehicle 200 based at least on the image 21. Further, the control device 10 can control the movement of the vehicle 200 within the predetermined area AR based on the estimated position of the vehicle 200.

[0011] When using the result of position estimation using the image 21 obtained from the infrastructure camera 20 in this way, if the position of the camera 20 is displaced, an error may occur in the position estimation and the control of the vehicle 200 may not be performed correctly. Therefore, it is required to accurately detect the displacement of the camera 20 before or during the control of the vehicle 200.

[0012] Therefore, the first embodiment provides a technique capable of accurately detecting the displacement of the camera 20. A marker M is used for detecting the displacement. In the first embodiment, the camera 20 captures an image 21 so as to include a plurality of markers M. When the control device 10 acquires a new image 21 from the camera 20, it compares the image 21 acquired earlier than the new image 21 with the new image 21. The former is referred to as "the image 21P acquired this time" or simply "the image 21P", and the latter is referred to as "the image 21R acquired last time" or simply "the image 21R". Then, the control device 10 determines whether the camera 20 is displaced based on the number of markers M whose positions have changed between the image 21P and the image 21R. Hereinafter, a marker M whose position has changed between the image 21P and the image 21R is referred to as a "position-changing marker". The control device 10 determines that the camera 20 is displaced only when there are a predetermined number or more of position-changing markers, and does not determine that the camera 20 is displaced when the number of position-changing markers is less than the predetermined number.

[0013] In the first embodiment, the reason why the camera 20 captures an image 21 including a plurality of markers M is as follows. Even if the position of one marker M has changed between the image 21P and the image 21R, it cannot be determined that the cause is the displacement of the camera 20. For example, it is also conceivable that the actual position of the marker M has shifted due to vibration or impact within a predetermined area AR, or that the marker M cannot be accurately recognized due to dirt or abrasion of the marker M. Therefore, it is necessary to distinguish whether the presence of the position-changing marker is due to the displacement of the camera 20 or an event that has occurred to the marker M.

[0014] In the first embodiment, by imaging a range including a plurality of markers M with the camera 20, it becomes easy to distinguish whether the cause of the positional deviation of the marker M in the image 21 is the positional deviation of the camera 20 or an event that has occurred to the marker M. When the positions of many markers M in the image 21 have changed, it is highly likely that it is due to the positional deviation of the camera 20. Conversely, when there are few markers M whose positions have changed in the image 21, it is highly likely that it is due to an event that has occurred to the marker M. Therefore, by imaging a plurality of markers M with the camera 20 and determining that the positional deviation of the camera 20 has occurred only when the number of markers with position changes is equal to or more than a predetermined number, the accuracy of detecting the positional deviation of the camera 20 can be improved.

[0015] 2. Configuration example FIG. 2 is a block diagram showing a configuration example of the mobile body support system 100 according to the first embodiment. The mobile body support system 100 includes a control device 10, a camera 20, and a communication device 30.

[0016] The camera 20 is an infrastructure camera installed in the infrastructure. The camera 20 images a predetermined area AR so as to include a plurality of markers M. The image 21 obtained by imaging is acquired by the control device 10 and temporarily stored in a storage device 120 described later.

[0017] The communication device 30 communicates with the outside of the mobile body support system 100. At least the vehicle 200 is included in the connection destination of the communication device 30.

[0018] The control device 10 supports the movement of the vehicle 200 in a predetermined area AR. The support for the movement of the vehicle 200 performed by the control device 10 includes the position estimation of the vehicle 200 and the control of the movement of the vehicle 200. The support for the movement of the vehicle 200 may also include transmitting the estimated position of the vehicle 200 to the vehicle 200. Further, when performing or before performing the position estimation of the vehicle 200, the control device 10 determines the positional deviation of the camera 20.

[0019] The control device 10 estimates the position of the vehicle 200 based on at least the image 21. The position estimation of the vehicle 200 based on the image 21 can be performed by converting the position of the vehicle 200 in the image 21 into the position of the vehicle 200 in the real space. Further, the control device 10 may perform position estimation using the sensor information of the vehicle 200 in addition to the image 21. Examples of the sensor information include an image captured by an in-vehicle camera, acceleration information of the vehicle 200, steering angle information, and the like. The sensor information is acquired from sensors mounted on the vehicle 200 such as an in-vehicle camera, a wheel speed sensor, a G-sensor, a pinion angle sensor, a yaw rate sensor, and the like. The control device 10 can acquire the sensor information by communicating with the vehicle 200 through the communication device 30.

[0020] The control device 10 includes one or more processors 110 (hereinafter simply referred to as the processor 110) and one or more storage devices 120 (hereinafter simply referred to as the storage device 120). The processor 110 executes various processes. For example, the processor 110 includes a CPU (Central Processing Unit). The storage device 120 stores various information and various programs. Examples of the storage device 120 include a volatile memory, a non-volatile memory, an HDD (Hard Disk Drive), an SSD (Solid State Drive), and the like. The processor 110 can execute various processes including determination of the misalignment of the camera 20 by executing the programs stored in the storage device 120. Further, the various information stored in the storage device 120 includes image information regarding the image 21.

[0021] 3. Example of processing flow FIG. 3 is a flowchart showing an example of a process performed by the mobile body support system 100 (processor 110) to detect the misalignment of the camera 20. A series of processes are performed before or during the mobile body support system 100 supports the vehicle 200. Further, a series of processes are realized by the processor 110 executing the programs stored in the storage device 120.

[0022] In step S101, the mobile body support system 100 acquires the image 21P newly captured by the camera 20. The acquired image 21P is temporarily stored in the storage device 120.

[0023] In step 102, the mobile body support system 100 compares the currently acquired image 21P with the previously acquired image 21R from the camera 20 to detect a position change marker. The image 21R is the newest image among the images 21 that were acquired from the camera 20 and stored in the storage device 120 before the image 21P. The position change marker is the marker M among the plurality of markers M included in the image 21R whose position has changed in the image 21P.

[0024] In step S103, the mobile body support system 100 determines whether the number of position change markers detected in step S103 is equal to or greater than a predetermined number. If the number of position change markers is equal to or greater than the predetermined number (step S103; Yes), the process proceeds to step S104. On the other hand, if there are less than the predetermined number of position change markers or if there are no position change markers (step S103; No), the process proceeds to step S105. The predetermined number is a reference number for determining the positional deviation of the camera 20. The predetermined number is set in advance to be 2 or more and not more than the number of markers M included in the image 21. For example, the same number as the number of markers M included in the image 21 may be set as the predetermined number. In this case, the positional deviation of the camera 20 will be detected only when the positions of all the markers M included in the image 21 have changed.

[0025] In step S104, the mobile body support system 100 determines that a positional deviation of the camera 20 has occurred.

[0026] In step S105, the mobile body support system 100 determines that no positional deviation of the camera 20 has occurred.

[0027] When it is determined that a series of processes has ended and there is no displacement in the position of camera 20, the mobile support system 100 supports the movement of vehicle 200 in the same way as in normal times. As controls that the mobile support system 100 performs on vehicle 200 and camera 20 when it is determined that displacement in the position of camera 20 has occurred, for example, the following are assumed.

[0028] The first is to reduce the reliability of the position estimation of vehicle 200 based on image 21. That is, the mobile support system 100 gives priority to the results obtained by other methods over the detection results based on image 21 and uses them for the position estimation of vehicle 200. For example, when there is another infrastructure camera that images a predetermined area AR, it is assumed that for the range that can be imaged by the other infrastructure camera, the images obtained from the other infrastructure camera are preferentially used for the position estimation of vehicle 200. Alternatively, it is assumed that the sensor information obtained from vehicle 200 is preferentially used for the position estimation of vehicle 200 over image 21.

[0029] The second is to stop the control of vehicle 200. For example, it is assumed that the control of vehicle 200 is stopped when the displacement of camera 20 is large and it is difficult to estimate the position of vehicle 200 from information other than image 21.

[0030] The third is to calibrate camera 20. Calibration is to correct the correspondence between the position coordinates in image 21 and the position coordinates in the real space, and it is performed using marker M. The mobile support system 100 corrects the position coordinates in image 21 based on the position of each marker M in image 21 and the position information that each marker M has. By performing calibration in this way, it is possible to continue to estimate the position of vehicle 200 using image 21 and support the movement.

[0031] Note that when calibrating the camera 20, the calibration may be determined to have failed. The calibration may fail, for example, when the misalignment of the camera 20 exceeds a size that can be corrected, or when position information cannot be obtained from at least a part of the marker M that should be included in the image 21. The inability to obtain position information using the marker M may occur, for example, when the misalignment of the camera 20 is large and the marker M is outside the viewing angle of the camera 20, or when the marker M becomes unreadable due to dirt. Therefore, the mobile body support system 100 may first calibrate the camera 20 and stop controlling the vehicle 200 when it is determined that the calibration has failed. Alternatively, when it is determined that the calibration has failed, the administrator of the predetermined area AR may be notified, and correction of the position of the camera 20, cleaning of the lens of the camera 20, the marker M, etc. may be prompted.

[0032] 4. Second Embodiment Also in the second embodiment, it is common with the first embodiment in that the configuration example and the misalignment of the camera 20 are determined only when the number of position change markers is equal to or more than a predetermined number. In the second embodiment, further, the "amount of position change" is obtained.

[0033] FIG. 4 is a diagram for explaining the amount of position change. The upper is the image 21P newly obtained from the camera 20, and the lower is the image 21R obtained from the camera 20 last time. The images 21P and 21R include five markers M1, M2, M3, M4, and M5. Among them, the position change markers are three markers M3, M4, and M5. In the image 21P, the positions of the markers M3, M4, and M5 before the position change, that is, in the image 21R, are indicated by dotted lines. Also, the moving distances D3, D4, and D5 of the respective position change markers M3, M4, and M5 when viewed within the image 21P are indicated by double-headed arrows.

[0034] The amount of position change is the amount obtained by comprehensively calculating the moving distances of the position change markers included in the image 21. For example, the amount of position change may be the average value or the maximum value of the moving distances of all the position change markers included in the image 21.

[0035] In the second embodiment, when it is determined that the misalignment of the camera 20 has occurred, the mobile body support system 100 changes the control of the vehicle 200 and the camera 20 according to the magnitude of the amount of position change. FIG. 5 is a flowchart showing an example of the processing in the second embodiment. The processing up to step S104 is omitted because it is the same as the flowchart in FIG. 3.

[0036] After it is determined in step S104 that there is misalignment of the camera 20, the process proceeds to step S106, and the mobile body support system 100 determines whether the amount of position change is equal to or less than a predetermined amount. If the amount of position change is equal to or less than the predetermined amount (step S106; Yes), the process proceeds to step S107. If the amount of position change is greater than the predetermined amount (step S106; No), the process proceeds to step S108. The predetermined amount is an amount set in advance.

[0037] In step S107, the mobile body support system 100 reduces the reliability of the position estimation based on the image 21.

[0038] In step S108, the mobile body support system 100 stops the control of the vehicle 200.

[0039] In the example of FIG. 5, when the amount of position change is small, only the reliability of the position estimation based on information other than the image 21 is reduced, and the control of the vehicle 200 continues. On the other hand, when the amount of position change is large, the control of the vehicle 200 is stopped.

[0040] In this way, in the second embodiment, the control is changed according to the magnitude of the amount of position change. As a result, the vehicle 200 can be controlled more appropriately according to the situation. Note that the control selected according to the magnitude of the amount of position change is not limited to the control illustrated in FIG. 5. For example, when the amount of position change is greater than a predetermined amount, the mobile support system 100 may perform calibration of the camera 20.

[0041] As described above, in the first and second embodiments, the camera 20 captures the image 21 so as to include a plurality of markers M. Then, based on the number of position change markers, the misalignment of the camera 20 is detected. In this way, the accuracy of detecting the misalignment of the camera can be improved.

[0042] 5. Third Embodiment The third embodiment will be described. The configuration example of the mobile support system 100 is common to that in FIG. 2. However, it is different from the first and second embodiments in that the camera 20 includes a plurality of cameras. Also, it is different from the first and second embodiments in that the markers M included in the images 21 captured by the respective cameras 20 may not be plural.

[0043] FIG. 6 shows, as an example of a plurality of cameras included in the camera 20, a first infrastructure camera (first camera) 20-1 and a second infrastructure camera (second camera) 20-2. The first camera 20-1 and the second camera 20-2 are installed at different positions and capture a first image 21-1 and a second image 21-2 so as to include the same marker M0. Note that FIG. 6 shows an example in which there is one marker M0, but there may be a plurality of markers M0.

[0044] In the third embodiment, the detection of the displacement of the first camera 20-1 is performed as follows. First, the mobile body support system 100 newly acquires the first image 21-1P captured by the first camera 20-1. Then, the first image 21-1P acquired this time is compared with the first image 21-1R acquired last time, and the change in the position of the marker M0 in the first image 21-1 is detected. The previously acquired first image 21-1R is the latest image among the first images 21-1 acquired from the first camera 20-1 before the first image 21-1P and stored in the storage device 120.

[0045] Also, the mobile body support system 100 newly acquires the second image 21-2P captured by the second camera 20-2. Then, the second image 21-2P acquired this time is compared with the second image 21-2R acquired last time, and the change in the position of the marker M0 in the second image 21-2 is detected. The previously acquired second image 21-2R is the latest image among the second images 21-2 acquired from the second camera 20-2 before the second image 21-2P and stored in the storage device 120.

[0046] Then, when a change in the position of the marker M0 is detected from the first image 21-1 and no change in the position of the marker M0 is detected from the second image 21-2, the mobile body support system 100 determines that the first camera 20-1 has been displaced. When a change in the position of the marker M0 is detected from either the first image 21-1 or the second image 21-2, the mobile body support system 100 does not determine that the first camera 20-1 has been displaced.

[0047] In the third embodiment, instead of a plurality of markers M, a plurality of cameras 20 are used to detect the misalignment of the cameras 20. As described in the first embodiment, when there is a marker M whose position changes in the image 21, the cause of the change in the position of the marker M in the image 21 can be either the misalignment of the camera 20 or an event that occurred to the marker M. If the marker M0 whose position changes in the first image 21-1 does not change its position in the second image 21-2, the cause of the change in the position of the marker M0 in the first image 21-1 is likely to be the misalignment of the first camera 20-1. Conversely, if the position of the same marker M0 changes in any of the plurality of images 21 captured by the plurality of cameras 20, the cause of the change in the position of the marker M0 is likely to be an event that occurred to the marker M0. Therefore, by detecting the misalignment of the cameras 20 using such a plurality of cameras 20, the accuracy of detecting the misalignment can be improved.

[0048] Note that also in the third embodiment, the control by the mobile body support system 100 may be changed according to the amount of position change. In this case, the amount of position change can be set as the moving distance of the marker M0 in the first image 21-1. Also, when there are a plurality of markers M0 and there are a plurality of markers M0 whose positions change in the first image 21-1, the amount of position change may be set as the average value of the respective moving distances, or may be set as the maximum value.

Explanation of Reference Numerals

[0049] 10 Control device 20 Camera 21 Image 30 Communication device 100 Mobile body support system 110 Processor 120 Storage device 200 Vehicle M Marker

Claims

【Claim 1】 A mobile body support system for supporting the movement of a mobile body in a predetermined area where markers of 1 or more are arranged, comprising one or more processors, wherein the one or more processors, acquire an image captured by an infrastructure camera installed in the predetermined area, the infrastructure camera includes a first infrastructure camera and a second infrastructure camera that image the same marker from different positions, compare the first image acquired this time from the first infrastructure camera with the first image acquired last time, and detect a change in the position of the same marker, compare the second image acquired this time from the second infrastructure camera with the second image acquired last time, and detect a change in the position of the same marker, when a change in the position of the same marker is detected from the first image and no change in the position of the same marker is detected from the second image, it is determined that the first infrastructure camera has shifted in position Mobile body support system.

Images