Mobile support system and mobile support method

Abstract

To provide a technique for improving recognition accuracy for a marker recognized by a moving body, regarding the moving body that recognizes a marker arranged in a predetermined area.SOLUTION: A moving body support system supports a moving body that recognizes a marker arranged in a predetermined area. The moving body support system includes one or more processors. The one or more processors are configured to execute processes of: estimating brightness at a position of the marker in the predetermined area; calculating a luminance correction value of an image including the marker according to the brightness at the position of the marker; acquiring a first image including a target marker around the moving body by using a camera; generating a second image by correcting luminance of the first image by using the luminance correction value for the target marker; and recognizing the target marker based on the second image.SELECTED DRAWING: Figure 12

Description

[Technical field]

[0001] The present disclosure relates to a technology for supporting a moving object that operates by recognizing markers arranged in a predetermined area. [Background technology]

[0002] Patent Document 1 relates to a parking assistance technology that identifies the relative positional relationship between a vehicle and a target parking position by recognizing a mark, and calculates a parking trajectory for guiding the vehicle to the target parking position. The mark is placed at the target parking position, and is recognized from an image captured by a video camera mounted on the vehicle. In the parking assistance technology of Patent Document 1, a photometric area is set on the image, the brightness value of the image is adjusted based on the brightness value of the photometric area, and then a process of recognizing the mark from the image is performed. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2010-215029 A Summary of the Invention [Problem to be solved by the invention]

[0004] Consider a mobile object that recognizes a marker installed in a specific area. The mobile object recognizes the marker by acquiring an image using a camera. The brightness at the marker's position changes due to environmental changes such as the weather, the time of day, and the presence or absence of street lights. The brightness at the marker's position also changes when a shadow is cast on the marker. If the brightness at the marker's position changes, it may become difficult for the mobile object to recognize the marker. If the mobile object cannot recognize the marker with high accuracy, for example, the accuracy of the mobile object's operation based on the marker recognition result decreases.

[0005] An object of the present disclosure is to provide a technology that can improve the accuracy of marker recognition by a moving object. [Means for solving the problem]

[0006] The first aspect relates to a mobile object support system that supports a mobile object in recognizing markers placed in a predetermined area. The mobile assistance system includes one or more processors. The one or more processors A brightness estimation process for estimating brightness at a position of a marker within a predetermined area without using an image obtained by a camera mounted on a moving object; A process of calculating a brightness correction value of an image including a marker according to the brightness at the position of the marker; A process of acquiring a first image including target markers around the moving object using a camera mounted on the moving object; generating a second image by correcting the luminance of the first image using a luminance correction value for the target marker; Recognizing a target marker based on the second image; The apparatus is configured to execute the following steps:

[0007] The second aspect relates to a mobile object assistance method for assisting a mobile object in recognizing markers arranged in a predetermined area. The mobility support method is as follows: Estimating brightness at a position of a marker within a predetermined area without using an image obtained by a camera mounted on a moving object; Calculating a brightness correction value of an image including the marker according to the brightness at the position of the marker; acquiring a first image including target markers around the moving object using a camera mounted on the moving object; generating a second image by correcting a luminance of the first image using a luminance correction value for the marker of interest; Recognizing a target marker based on the second image; Includes. Effect of the Invention

[0008] According to the present disclosure, a brightness correction value for correcting an acquired image is calculated according to the brightness at the position of a marker. By correcting the image with the brightness correction value, the accuracy of marker recognition by a moving object is improved. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a conceptual diagram for explaining an overview of automatic valet parking. [Diagram 2] FIG. 1 is a conceptual diagram for explaining a problem. [Diagram 3] FIG. 1 is a conceptual diagram for explaining a problem. [Figure 4] FIG. 1 is a conceptual diagram for explaining a problem. [Diagram 5] FIG. 1 is a conceptual diagram for explaining a problem. [Figure 6] 1 is a conceptual diagram for explaining an example of a flow of processing by the mobile object support system according to the present embodiment. [Figure 7] 1 is a conceptual diagram for explaining an example of a flow of processing by the mobile object support system according to the present embodiment. [Figure 8] 1 is a conceptual diagram for explaining an example of a flow of processing by the mobile object support system according to the present embodiment. [Figure 9] 1 is a conceptual diagram for explaining an example of a flow of processing by the mobile object support system according to the present embodiment. [Figure 10] FIG. 1 is a block diagram showing an example of the configuration of a vehicle. [Figure 11] FIG. 2 is a block diagram showing an example of the configuration of a management device; [Figure 12] FIG. 2 is a block diagram for explaining an example of a mobile object support process according to the present embodiment. [Figure 13] 5 is a flowchart showing a first example of a moving object support process according to the present embodiment. [Figure 14] FIG. 4 is a block diagram for explaining an example of a brightness estimation process according to the present embodiment. [Figure 15] 10A to 10C are conceptual diagrams for explaining the effect of the second shadow position estimation process according to the present embodiment. [Figure 16] 11 is a table for explaining an example of a luminance correction value according to the embodiment; [Figure 17] 13 is a flowchart showing a second example of a moving object support process according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0011] 1. Overview The present disclosure relates to a mobile object support system that supports a mobile object that recognizes a marker arranged in a predetermined area. Supporting a mobile object means providing general support related to the mobile object, including monitoring the mobile object, controlling the operation of the mobile object, and managing information related to the mobile object. Examples of the predetermined area include a parking lot and an area where a shuttle bus operates. Examples of the mobile object include a vehicle and a robot. The vehicle may be an autonomous vehicle. As an example, the following description considers a case where the mobile object is a vehicle. When generalizing, the term "vehicle" in the following description should be read as "mobile object".

[0012] FIG. 1 is a conceptual diagram for explaining "Automated Valet Parking (AVP)" as an example of a case where a vehicle 1 recognizes a marker M placed in a predetermined area AR. In this example, the predetermined area AR is a parking lot. An entrance area is an area where the vehicle 1 starts and ends automated valet parking, and is included in the predetermined area AR. The parking lot may be indoors or outdoors. Multiple markers M are placed in the parking lot.

[0013] Vehicle 1 is an AVP vehicle that supports automatic valet parking in a parking lot, and can automatically drive at least in the parking lot. Vehicle 1 is equipped with a recognition sensor for recognizing the surrounding situation. The recognition sensor includes a camera. Vehicle 1 automatically drives in the parking lot while recognizing the surrounding situation using the recognition sensor.

[0014] The vehicle 1 uses a camera to obtain an image showing the situation around the vehicle 1, and recognizes the marker M based on the obtained image. By recognizing the marker M, the vehicle 1 can identify the parking lot, recognize the initial position when entering the parking lot, correct the target route, detect the target parking position, estimate the vehicle's own position, and the like. For example, the vehicle 1 performs self-localization, which estimates the vehicle's own position with high accuracy, by combining the recognition result of the marker M using the camera with the position information of the marker M in the parking lot. Alternatively, the vehicle 1 may recognize the parking lot based on the recognition result of the marker M, and confirm that the vehicle has entered the correct parking lot. Alternatively, the vehicle 1 may recognize the entrance area based on the recognition result of the marker M.

[0015] The target route PT is a movement route for the vehicle 1 to move to the target parking frame. The target parking frame is a parking frame assigned to the vehicle 1. The target route PT may be a movement route from the entrance area to the target parking frame, or may be a movement route from the current position of the vehicle 1 to the target parking frame. The vehicle 1 performs automatic driving so as to follow the target route PT based on the position of the vehicle 1 estimated by self-location estimation and the target route PT. This enables the vehicle 1 to automatically move from the entrance area to the target parking frame.

[0016] The management device 2 manages the automatic valet parking in the parking lot. The management device 2 may be a server. The management device 2 can communicate with each vehicle (vehicle 1, parked vehicle 3) in the parking lot. For example, the management device 2 may issue an entry instruction or an exit instruction to the vehicle 1. The management device 2 may know the scheduled exit time of each vehicle (vehicle 1, parked vehicle 3) in the parking lot. When there is an AVP vehicle scheduled to enter the parking lot, the management device 2 may know the scheduled entry time of the vehicle scheduled to enter. The management device 2 may provide the vehicle 1 with position information of the marker M in the parking lot. The management device 2 may assign a target parking frame to the vehicle 1. The management device 2 may generate a target route PT and provide information of the target route PT to the vehicle 1. The management device 2 may know the position of each vehicle (vehicle 1, parked vehicle 3) in the parking lot. The management device 2 may remotely operate each vehicle (vehicle 1, parked vehicle 3) in the parking lot.

[0017] In order for the vehicle 1 to operate accurately, it is important for the vehicle 1 to accurately recognize the marker M. However, if the brightness at the position of the marker M changes, the vehicle 1 may not be able to correctly recognize the marker M. For example, if the position of the marker M becomes bright, the brightness of the acquired image increases, causing the image to be blown out, and the vehicle 1 may not be able to recognize the marker M. Alternatively, if the position of the marker M becomes dark, the brightness of the acquired image decreases, causing the image to become black overall, and the vehicle 1 may not be able to recognize the marker M. In this way, the accuracy of recognition of the marker M may decrease due to the brightness of the image changing due to the change in brightness at the position of the marker M.

[0018] 2 to 5 conceptually show the state of a parking lot in the morning, midday, evening, and night, respectively. The brightness at the position of marker M changes depending on the weather and the time of day. When it becomes daytime and the sun rises, the brightness at the position of marker M becomes brighter than in the morning or evening. Conversely, when it becomes night and the parking lot becomes dark, the brightness at the position of marker M becomes darker. Alternatively, although not shown, the brightness at the position of marker M may also become darker when it is raining or cloudy. In this way, the brightness at the position of marker M changes as the surrounding environment changes.

[0019] The brightness at the position of the marker M also changes depending on the shadow cast at the position of the marker M. For example, in Figures 2, 4, and 5, shadows cast by parked vehicles 3 and walls in the parking lot are cast on some of the markers M, making the brightness at the position of the marker M dark. The shadow cast on the marker M changes depending on the position of the light source, such as the sun or a street lamp, and the positions of obstacles, such as the parked vehicles 3 and walls.

[0020] In this way, the accuracy of marker recognition by the vehicle 1 may decrease due to changes in the surrounding environment or the position of a shadow. The mobile object support system according to this embodiment makes it possible to improve the accuracy of marker recognition by the vehicle 1 even in a situation where the brightness at the position of the marker M may change.

[0021] 2. Brightness estimation and luminance correction In the following description, a "camera image" refers to an image of the surroundings of the vehicle 1 obtained by a camera mounted on the vehicle 1. The mobile support system according to this embodiment improves the accuracy of marker recognition by correcting the brightness of the camera image. Specifically, the mobile support system acquires brightness information about the brightness at the position of the marker M. Preferably, the mobile support system acquires the brightness information at the position of the marker M without using a camera image acquired by a camera mounted on the vehicle 1. The mobile support system calculates a "brightness correction value" for correcting the brightness of the camera image based on the brightness information. The brightness correction value is set to darken a camera image that is too bright, or brighten a camera image that is too dark. Then, the brightness of the camera image is corrected by the brightness correction value, and the marker M is recognized based on the corrected image.

[0022] 6 to 9 are diagrams showing an example of a process flow related to the brightness correction value. In FIG. 6, first, a request for vehicle 1 to enter the parking lot is transmitted from a user terminal of a user of the parking lot (automatic valet parking) to the management device 2. At this time, the user terminal may transmit the scheduled entry time of vehicle 1 together with the entry request. If entry of vehicle 1 is possible, the management device 2 permits the entry request of vehicle 1 and notifies the user terminal that entry of vehicle 1 has been permitted. When the management device 2 permits the entry request of vehicle 1, it acquires brightness information. The brightness information is acquired by the management device 2 estimating using brightness estimation information. The brightness estimation information is information used to estimate illuminance, the position of a shadow, and the like in the parking lot. The brightness estimation information may include information taking into account the scheduled entry time of vehicle 1. A specific example of brightness estimation information and an acquisition method thereof will be described later.

[0023] The management device 2 calculates a brightness correction value for the marker M based on the acquired brightness information. The management device 2 transmits the calculated brightness correction value to the vehicle 1. The vehicle 1 performs marker recognition using the brightness correction value. Specifically, the vehicle 1 uses a camera to acquire an image that is assumed to include the marker M around the vehicle 1. The vehicle 1 corrects the brightness of the acquired image using the brightness correction value, and recognizes the marker M based on the corrected image. In this way, by performing marker recognition based on an image corrected using the brightness correction value, the influence of a change in brightness at the position of the marker M can be reduced, and the accuracy of the marker recognition can be improved. The marker M that is the target recognized by the vehicle 1 may also be called a "target marker."

[0024] 6, the acquisition of brightness information and the calculation of brightness correction values ​​are performed by the management device 2. In other words, the vehicle 1 does not need to acquire brightness information and calculate brightness correction values. Therefore, the processing load on the vehicle 1 is significantly reduced.

[0025] FIG. 7 is a diagram showing another example of a process flow related to the brightness correction value. The calculation of the brightness correction value may be performed by the vehicle 1 as shown in FIG. 7. In FIG. 7, the management device 2 transmits acquired brightness information to the vehicle 1, and the vehicle 1 calculates the brightness correction value of the marker M based on the transmitted brightness information. The vehicle 1 acquires an image that is assumed to include the marker M using a camera, and corrects the brightness of the acquired image using the brightness correction value. The vehicle 1 can improve the accuracy of marker recognition by recognizing the marker M based on the corrected image.

[0026] 7, the brightness information is acquired by the management device 2. That is, the vehicle 1 does not need to acquire the brightness information, at least. Therefore, the processing load on the vehicle 1 is reduced.

[0027] FIG. 8 is a diagram showing yet another example of a process flow related to the brightness correction value. The brightness information may be acquired by the vehicle 1 as shown in FIG. 8. At this time, a part or all of the brightness estimation information may be transmitted from the management device 2 to the vehicle 1. For example, information on the position of the parked vehicle 3 in the parking lot at the scheduled entry time of the vehicle 1 is transmitted from the management device 2 to the vehicle 1 as the brightness estimation information. The vehicle 1 estimates the brightness at the position of the marker M using the brightness estimation information and acquires the brightness information. The vehicle 1 calculates the brightness correction value of the marker M based on the brightness information. The vehicle 1 acquires an image that is assumed to include the marker M using a camera, and corrects the brightness of the acquired image using the brightness correction value. The vehicle 1 can improve the accuracy of marker recognition by recognizing the marker M based on the corrected image.

[0028] FIG. 9 is a diagram showing yet another example of a process flow related to the calculation of a brightness correction value. In the example of FIG. 9, the management device 2 acquires brightness information, calculates a brightness correction value, and recognizes a marker. The vehicle 1 acquires an image that is assumed to include a marker using a camera, and transmits the acquired image to the management device 2. The management device 2 corrects the brightness of the image transmitted from the vehicle 1 using the brightness correction value, and recognizes the marker M based on the corrected image. By recognizing the marker M, the management device 2 acquires, for example, position information of the vehicle 1, and transmits the acquired position information to the vehicle 1. In FIG. 9 as well, the management device 2 recognizes the marker M based on the corrected image, thereby improving the accuracy of marker recognition.

[0029] As described later, the brightness at the position of the marker M is estimated by using brightness estimation information. The brightness estimation information is information used to estimate the illuminance and the position of a shadow in a parking lot. Typically, the brightness estimation information does not include a camera image obtained by a camera mounted on the vehicle 1. In that case, the mobile support system can estimate the brightness at the position of the marker M without using the camera image. As a comparative example, consider estimating the brightness around the vehicle 1 based on the camera image. In the comparative example, the camera image needs to be analyzed for each frame of the camera image, which increases the processing load. On the other hand, according to the present embodiment, since it is not necessary to estimate the brightness for each frame of the image, the processing load is reduced.

[0030] The brightness estimation process without using a camera image may be performed in advance before the vehicle 1 enters the warehouse. Furthermore, the brightness correction value may also be calculated in advance before the vehicle 1 enters the warehouse. By performing the necessary processes in advance before the vehicle 1 enters the warehouse, the processing load after the entry can be reduced, and the vehicle 1 can be operated smoothly. Furthermore, by performing the necessary processes in advance, the effect of processing delays can be suppressed. For example, a situation in which the marker recognition accuracy is not as expected due to processing delays can be prevented.

[0031] The brightness estimation information may include the scheduled entry time of the vehicle 1. If the scheduled entry time of the vehicle 1 is known, it is possible to estimate in advance the brightness at the position of the marker M at the scheduled entry time and calculate the brightness correction value in advance. The scheduled entry time is information unique to automatic valet parking in a parking lot. It can be said that performing necessary processing in advance based on the scheduled entry time is a feature unique to automatic valet parking in a parking lot.

[0032] 2, 4, etc., shadows cast by parked vehicles 3 may fall on some of the markers M, causing the brightness at the positions of the markers M to become dark. Therefore, the brightness estimation information may include information on the parking positions of the parked vehicles 3 in the parking lot. In the case of automatic valet parking, the management device 2 knows the parking positions of the parked vehicles 3 in the parking lot. It can be said that estimating the brightness at the positions of the markers M based on the parking positions of the parked vehicles 3 in the parking lot is also a feature unique to automatic valet parking in a parking lot.

[0033] 3. Example of vehicle 1 configuration 10 is a block diagram showing an example of the configuration of the vehicle 1. The vehicle 1 includes a vehicle state sensor 11, a recognition sensor 12, a communication device 13, a driving device 14, and a control device 15.

[0034] The vehicle state sensor 11 detects the state of the vehicle 1. Examples of the vehicle state sensor 11 include a vehicle speed sensor (wheel speed sensor), a steering angle sensor, a yaw rate sensor, a lateral acceleration sensor, and the like.

[0035] The recognition sensor 12 recognizes the surrounding situation of the vehicle 1. The recognition sensor 12 includes a camera. Other examples of the recognition sensor 12 include a LIDAR (Laser Imaging Detection and Ranging), a radar, an illuminance sensor, and the like.

[0036] The communication device 13 communicates with the outside of the vehicle 1. For example, the communication device 13 communicates with the management device 2.

[0037] The traveling device 14 includes a steering device, a drive device, and a braking device. The steering device steers the wheels of the vehicle 1. For example, the steering device includes an electric power steering (EPS) device. The drive device is a power source that generates a driving force. Examples of the drive device include an engine, an electric motor, and an in-wheel motor. The braking device generates a braking force.

[0038] The control device 15 controls the vehicle 1. Specifically, the control device 15 includes one or more processors 16 (hereinafter simply referred to as processor 16) and one or more storage devices 17 (hereinafter simply referred to as storage devices 17). The processor 16 executes various processes. The storage device 17 stores various information. Examples of the storage device 17 include a volatile memory, a non-volatile memory, a hard disk drive (HDD), and a solid state drive (SSD). The processor 16 executes a control program, which is a computer program, to realize various processes by the control device 15. The control program is stored in the storage device 17 or recorded in a computer-readable recording medium.

[0039] The processor 16 acquires various information. The acquired various information is stored in the storage device 17. The various information includes map information 710, vehicle position information 720, brightness estimation information 730, and brightness information 740.

[0040] The map information 710 is map information about a predetermined area AR. The map information 710 includes position information of the markers M, position information of the parking spaces, position information of structures, position information of lights, position information of entrance areas, and the like. The map information 710 may be provided to the vehicle 1 by a manager of the parking lot or the like. Alternatively, the map information 710 may be transmitted from the management device 2 to the vehicle 1 via the communication device 13.

[0041] The vehicle position information 720 includes the position information of the vehicle 1. The vehicle position information 720 includes the position information of the vehicle 1 calculated from the vehicle state information obtained by the vehicle state sensor 11. Specifically, the processor 16 calculates the amount of movement of the vehicle 1 based on the vehicle speed and steering angle of the vehicle 1 obtained by the vehicle speed sensor and steering angle sensor, and thereby calculates the position information of the vehicle 1. The vehicle position information 720 includes the position information of the vehicle 1 calculated in this manner.

[0042] Furthermore, the processor 16 corrects the position information of the vehicle 1 by comparing the installation position of the marker M indicated in the map information 710 with the recognized position of the marker M by the camera. In this way, the processor 16 performs self-location estimation to estimate the position of the vehicle 1 with high accuracy. By repeating the calculation of the position information based on the vehicle state information and the correction based on the marker recognition, the processor 16 can continuously obtain highly accurate position information of the vehicle 1. The vehicle position information 720 includes highly accurate position information of the vehicle 1 obtained by self-location estimation.

[0043] The vehicle position information 720 may also include information on the target route PT. The target route PT is calculated from the current position of the vehicle 1 or the position of the entrance area, and the position of the target parking frame. The target route PT may be calculated in advance from the position of the entrance area and the position of the target parking frame before the vehicle 1 enters the parking lot. Alternatively, the target route PT may be calculated from the current position of the vehicle 1 and the position of the target parking frame after the vehicle 1 enters the parking lot. The target route PT may be calculated by the management device 2 and provided to the vehicle 1, or may be calculated by the processor 16.

[0044] The brightness estimation information 730 is information used to estimate the brightness at the position of the marker M. An example of the brightness estimation information 730 will be described later.

[0045] The brightness information 740 is information that indicates the brightness at the position of the marker M. A method for acquiring the brightness information 740 will be described later.

[0046] 4. Example of configuration of management device 2 11 is a block diagram showing a configuration example of the management device 2. The management device 2 includes a communication device 23, one or more processors 26 (hereinafter simply referred to as the processor 26), and one or more storage devices 27 (hereinafter simply referred to as the storage device 27).

[0047] The communication device 23 communicates with the vehicle 1 via a communication network. The communication device 23 may also communicate with the parked vehicle 3. The communication device 23 may also communicate with infrastructure sensors. The infrastructure sensors are sensors installed in a predetermined area AR, and include infrastructure cameras, infrastructure illuminance sensors, and the like.

[0048] The processor 26 executes various processes. The storage device 27 stores various information. Examples of the storage device 27 include a volatile memory, a non-volatile memory, a HDD, and an SSD. The processor 26 executes a control program, which is a computer program, to realize various processes by the management device 2. The control program is stored in the storage device 27, or recorded in a computer-readable recording medium.

[0049] The map information 710 is provided to the management device 2 by a manager of the parking lot or the like, and is stored in the storage device 27. The processor 26 may communicate with the vehicle 1 via the communication device 23 and transmit the map information 710 to the vehicle 1.

[0050] The vehicle position information 720 includes the position information of the vehicle 1, information about the target route PT, and the like.

[0051] The position information of the vehicle 1 may be acquired by the processor 26 communicating with the vehicle 1 via the communication device 23. Alternatively, the position information of the vehicle 1 may be acquired by an infrastructure camera installed in a predetermined area AR.

[0052] The target route PT may be acquired by the processor 26 calculating it from the current position of the vehicle 1 or the position of the entrance area, and the position of the target parking frame. Alternatively, the processor 26 may communicate with the vehicle 1 to acquire the target route PT calculated by the processor 16 of the vehicle 1.

[0053] An example of the brightness estimation information 730 and a method for acquiring the brightness information 740 will be described later.

[0054] 5.Mobile Support Processing An example of a mobile object support process performed by the mobile object support system according to this embodiment will be described in detail below.

[0055] 12 is a block diagram showing an example of a functional configuration of a mobile object support system according to this embodiment. The mobile object support system includes, as functional blocks, a brightness estimation unit 110, a brightness correction value calculation unit 120, a vehicle position acquisition unit 130, a first image acquisition unit 140, a second image generation unit 150, and a marker recognition unit 160. These functional blocks may be realized by the processor 16 executing a control program which is a computer program, or may be realized by the processor 26 executing a control program which is a computer program. Alternatively, each functional block may be realized by distributed processing of the processor 16 and the processor 26.

[0056] 13 is a flowchart showing a first example of the mobile object support processing according to the present embodiment. The first example of the mobile object support processing will be described with reference to FIGS.

[0057] 5-1. Brightness estimation process (step S110) In step S110, the brightness estimation unit 110 performs a brightness estimation process to estimate the brightness at the position of the marker M. The brightness estimation process may be performed before the vehicle 1 enters the warehouse, or may be performed after the vehicle 1 enters the warehouse.

[0058] 5-1-1. Information for brightness estimation The brightness estimation information 730 is information that is referenced when estimating brightness. The brightness estimation unit 110 estimates the brightness at the position of the marker M based on the position information of the marker M and the brightness estimation information 730, and acquires brightness information 740.

[0059] 14 is a block diagram for explaining an example of brightness estimation processing by the brightness estimation unit 110. The brightness estimation unit 110 includes a shadow position estimation unit 111. The shadow position estimation unit 111 includes a first shadow position estimation unit 112 and a second shadow position estimation unit 113. Marker position information 711 is position information of the marker M in a predetermined area AR, and is obtained from map information 710. Brightness estimation information 730 includes illuminance information 731, light source position information 732, obstacle position information 735, and vehicle information 738.

[0060] The illuminance information 731 indicates at least one of the illuminance at the position of the marker M and the illuminance of the predetermined area AR. For example, the illuminance is estimated based on the date, the time period, weather information, sunshine information, etc. As another example, the illuminance may be detected by an illuminance sensor. The illuminance sensor may be an infrastructure illuminance sensor installed in the predetermined area AR, or an in-vehicle illuminance sensor mounted on the vehicle 1.

[0061] The light source position information 732 indicates the position of the light source. The light source position information 732 includes at least one of sun position information 733 and lighting position information 734. The sun position information 733 is information indicating the position of the sun, and is calculated based on the date and time of day. The lighting position information 734 is information about lighting installed in the specified area AR, and includes information about the installation positions of the lighting. The lighting includes street lights installed in the specified area AR. The information about the installation positions of the lighting is obtained from the map information 710.

[0062] The obstacle position information 735 indicates the position of an obstacle that may cast a shadow within the predetermined area AR. The obstacle position information 735 includes at least one of structure position information 736 and parked vehicle position information 737.

[0063] The structure position information 736 indicates the positions of structures installed within the predetermined area AR. Examples of structures include pillars and walls. The structure position information 736 is obtained from the map information 710.

[0064] The parked vehicle position information 737 indicates the position of the parked vehicle 3 within the predetermined area AR. The parked vehicle position information 737 can be acquired by the management device 2 communicating with the parked vehicle 3. Alternatively, the parked vehicle position information 737 may be acquired by the management device 2 communicating with an infrastructure camera. The management device 2 may transmit the acquired parked vehicle position information 737 to the vehicle 1.

[0065] The vehicle information 738 includes at least one of the current position or the future position of the vehicle 1. The current position of the vehicle 1 is obtained from the vehicle position information 720. The current position of the vehicle 1 may be the position information of the vehicle 1 calculated from the vehicle state information, may be the position information of the vehicle 1 obtained by highly accurate self-position estimation, or may be information obtained by an infrastructure camera. The future position of the vehicle 1 is obtained as the position of the vehicle 1 on the target route PT. The target route PT is obtained from the vehicle position information 720. The vehicle information 738 may further include vehicle size information indicating the size of the vehicle 1. The size of the vehicle 1 means at least one of the length, width, and height of the vehicle 1. The vehicle size information can be acquired in advance by the storage device 17 of the vehicle 1. The vehicle size information may be provided to the management device 2 and stored in the storage device 27.

[0066] The shadow position estimation unit 111 performs a shadow position estimation process to estimate the position of a shadow within a predetermined area AR. The shadow position includes a first shadow position estimated by a first shadow position estimation unit 112 and a second shadow position estimated by a second shadow position estimation unit 113.

[0067] The first shadow position estimation unit 112 estimates a first shadow position, which is the position of a shadow generated by a light source and an obstacle in the predetermined area AR. The position of the light source is obtained from the light source position information 732. The position of the obstacle in the predetermined area AR is obtained from the obstacle position information 735. The first shadow position estimation unit 112 performs a first shadow position estimation process to estimate the first shadow position based on the light source position information 732 and the obstacle position information 735.

[0068] The second shadow position estimation unit 113 estimates a second shadow position, which is the position of a shadow generated by the light source and the vehicle 1. The position of the light source is obtained from the light source position information 732. The position of the vehicle 1 is obtained from the vehicle information 738 as the current position or future position of the vehicle 1. When the brightness estimation process is performed before the vehicle 1 enters the warehouse, the position of the vehicle 1 acquired by the second shadow position estimation unit 113 is the future position of the vehicle 1. The second shadow position estimation unit 113 performs a second shadow position estimation process that estimates the second shadow position based on the light source position information 732 and the vehicle information 738. In the second shadow position estimation process, the second shadow position may be estimated using vehicle size information in addition to the position of the light source and the position of the vehicle 1. The vehicle size information is obtained from the vehicle information 738.

[0069] Fig. 15 shows an example in which a shadow is generated at the position of the marker M by a light source and the vehicle 1, causing a change in brightness at the position of the marker M. Since the shadow position estimation unit 111 includes the second shadow position estimation unit 113, it is possible to accurately estimate the brightness at the position of the marker M even in a case such as that shown in Fig. 15.

[0070] The brightness estimation process performed by the brightness estimation unit 110 includes estimating the brightness at the position of the marker M based on the illuminance information 731 and the marker position information 711. The brightness estimation process may further include estimating the brightness at the position of the marker M based on the position of the shadow obtained by the shadow position estimation process and the marker position information 711. The brightness information 740 acquired by the brightness estimation process may be acquired at once for all the markers M in the parking lot, or may be acquired only for some of the markers M. When the brightness information is acquired only for some of the markers M, it may be acquired only for the markers M located near the future position of the vehicle 1, for example.

[0071] 5-1-2. Expected time of arrival The brightness estimation unit 110 may obtain information on the scheduled entry time of the vehicle 1, and perform brightness estimation processing using the brightness estimation information 730 at the scheduled entry time. The scheduled entry time is transmitted from a user terminal or the like to the management device 2 or the vehicle 1, and is obtained by the processor 16 or the processor 26.

[0072] The illuminance information 731 at the scheduled entry time is estimated based on the season, the position of the sun at the entry time, weather information at the entry time, sunshine information, and the like.

[0073] The light source position information 732 at the scheduled entry time includes at least one of the sun position information 733 at the scheduled entry time and the lighting position information 734. The sun position information 733 at the scheduled entry time is calculated based on the season and the scheduled entry time.

[0074] The obstacle position information 735 at the scheduled entry time includes at least one of structure position information 736 and parked vehicle position information 737 at the scheduled entry time.

[0075] The parked vehicle position information 737 at the scheduled entry time can be calculated by the management device 2 communicating with a user terminal or the like to acquire the scheduled departure time of the parked vehicle 3 and the scheduled entry time of the AVP vehicle scheduled to enter. The management device 2 may transmit the acquired parked vehicle position information 737 at the scheduled entry time to the vehicle 1.

[0076] The vehicle information 738 at the scheduled entry time is information about the future position of the vehicle 1. The future position of the vehicle 1 is obtained as the position of the vehicle 1 on the target route PT.

[0077] 5-2. Luminance correction value calculation process (step S120) In step S120, the brightness correction value calculation unit 120 calculates a brightness correction value. The brightness correction value is a value for correcting the brightness of an image including the marker M acquired by the camera, and is calculated for each marker M based on the brightness information 740. The brightness correction value is set to darken an image that is too bright, or to brighten an image that is too dark. In other words, the brightness correction value is set so that the marker M is more easily recognized. The brightness correction value may be a value for correcting the brightness of each pixel of the image, or may be a value for correcting the color according to the brightness of the image. The brightness correction value may be acquired for all the markers M in the parking lot at once, or may be acquired only for some of the markers M. The brightness correction value calculation process may be performed before the vehicle 1 enters the parking lot, or may be performed after the vehicle 1 enters the parking lot.

[0078] 5-3. Vehicle position acquisition process (step S130) In step S130, the vehicle position acquisition unit 130 acquires the position information of the vehicle 1. The position information of the vehicle 1 acquired by the vehicle position acquisition unit 130 is the position information of the vehicle 1 calculated from the vehicle state information, and is obtained from the vehicle position information 720. Alternatively, the position information of the vehicle 1 acquired by the vehicle position acquisition unit 130 may be obtained by an infrastructure camera. The processing from step S130 onwards is performed after the vehicle enters the warehouse.

[0079] 5-4. First image acquisition process (step S140) In step S140, the first image acquisition unit 140 acquires a first image that is assumed to include a target marker Mt, using a camera mounted on the vehicle 1. The target marker Mt is a marker that is located near the current position of the vehicle 1, among the markers M. The target marker Mt is determined by estimating the marker M that is located near the current position of the vehicle, based on the position information of the vehicle 1 acquired by the vehicle position acquisition unit 130 and the marker position information 711.

[0080] 5-5. Second image generation process (step S150) In step S150, the second image generating unit 150 corrects the first image using the luminance correction value for the target marker Mt to obtain a second image.

[0081] FIG. 16 is a table showing an example of the luminance correction value. The luminance correction value may be a coefficient determined for each category such as daytime, nighttime, presence or absence of shadow, etc., as in the table of FIG. 16. The second image is generated by correcting the luminance of each pixel of the first image, for example, using this coefficient. The coefficient is set to 1 when the brightness at the position of the marker M is a reference brightness, and the brighter it is, the smaller the value, and the darker it is, the larger the value. The second image generation unit 150 calculates the luminance of each pixel of the image acquired by the camera, and when the coefficient is smaller than 1, stronger correction is applied to pixels with high luminance. Conversely, when the coefficient is larger than 1, stronger correction is applied to pixels with low luminance. In this way, the luminance is corrected for each pixel, and the second image is generated.

[0082] 5-6. Marker recognition process (step S160) In step S160, the marker recognition unit 160 recognizes the target marker Mt based on the second image. When the marker recognition unit 160 acquires the recognition result of the target marker Mt, the processing of the current cycle ends.

[0083] Effects By the mobile object assistance process described above, a brightness correction value for correcting the brightness of an image including the marker M (target marker Mt) is calculated according to the brightness at the position of the marker M. The mobile object assistance system can improve the recognition accuracy of the target marker Mt by correcting the brightness of the first image using the brightness correction value. By improving the recognition accuracy of the target marker Mt, the accuracy of the operation of the vehicle 1 based on the recognition result of the target marker Mt is also improved.

[0084] In the first example, the brightness estimation process and the brightness correction value calculation process can be performed in advance before the vehicle 1 enters the warehouse. By calculating the brightness correction value in advance, the time from when the vehicle 1 acquires an image to when it recognizes a marker can be shortened, enabling the vehicle 1 to operate smoothly. In addition, since it is not necessary to calculate the brightness correction value every time the vehicle 1 moves, the processing load of the processor 16 of the vehicle 1 and the processor 26 of the management device 2 can be reduced. Even when the brightness correction value calculation process is performed after the vehicle 1 enters the warehouse, the brightness information and the brightness correction value are acquired using information such as weather and time, and therefore, they can be performed before the image is acquired by the camera. Compared to the case where the brightness of the image is checked every time an image is acquired by the camera, the processing load of the processor 16 of the vehicle 1 can be reduced.

[0085] 6. Second Example of Mobile Assistance Processing 17 is a flowchart showing a second example of the mobile object support processing according to the present embodiment. The second example of the mobile object support processing will be described with reference to FIG.

[0086] 6-1. Vehicle position acquisition process (step S210) In step S210, the vehicle position acquisition unit 130 acquires the position information of the vehicle 1. The position information of the vehicle 1 is obtained from the vehicle position information 720 as information on the current position of the vehicle 1. Alternatively, the position information of the vehicle 1 may be obtained by an infrastructure camera. In the second example, the processing from step 210 onwards is performed after the vehicle 1 is brought into the warehouse.

[0087] 6-2. Brightness estimation process (step S220) In step S220, the brightness estimation unit 110 estimates the brightness at the position of the marker M. The brightness estimation unit 110 estimates the brightness at the position of the marker M based on the position information of the marker M and the brightness estimation information 730, and acquires the brightness information 740.

[0088] Of the information included in the brightness estimation information 730, the vehicle information 738 is information about the current position of the vehicle 1. The current position of the vehicle 1 is obtained from the vehicle position information 720. Other information included in the brightness estimation information 730 is obtained by the brightness estimation unit in the same manner as in step 110.

[0089] 5-2. Luminance correction value calculation process (step S230) In step S230, the luminance correction value calculation unit 120 calculates a luminance correction value. The process in step S230 is similar to the process in step S120 in Fig. 13. From step S240 onwards, the same processes as from step S140 in Fig. 13 onwards are performed.

[0090] Effects As in the first example, the mobile object support system can improve the accuracy of marker recognition by correcting the brightness of the image using the brightness correction value. The improved accuracy of marker recognition also improves the accuracy of the operation of the vehicle 1.

[0091] In the second example, the brightness estimation process and the brightness correction value calculation process are performed after acquiring the current position of the vehicle 1. Since the brightness information 740 is estimated using the current position of the vehicle 1, it is possible to reduce errors in the brightness information 740. Also, in the second example, it is not necessary to use an image acquired by a camera to acquire the brightness information and the brightness correction value, and it is possible to reduce the processing load of the processor 16 of the vehicle 1 compared to the case where the brightness information and the brightness correction value are calculated using an image.

[0092] 6. Other embodiments The present disclosure can be applied to other than automatic valet parking of a vehicle 1 in a parking lot. For example, the present disclosure can be applied to automatic valet parking in which a vehicle without an autonomous driving function is towed by an autonomous driving robot. The present disclosure can also be applied to a case where markers M are placed in a city and mobility such as a vehicle or a robot recognizes the markers M and performs localization processing.

[0093] When generalizing, the word "vehicle" in the above description should be read as "mobile body." [Explanation of symbols]

[0094] 1 vehicle 2 Management device 3. Parked vehicles 11 Vehicle condition sensor 12 Recognition Sensor 13. Communications Equipment 14 Running gear 15 Control device 16 processors 17 Storage device 23 Communication Equipment 26 processors 27 Storage device 110 Brightness Estimation Unit 111 Shadow position estimation section 112 First shadow position estimation section 113 Second shadow position estimation section 120 Brightness correction value calculation unit 130 Vehicle position acquisition unit 140 First image acquisition unit 150 Second image generating unit 160 Marker Recognition Unit 710 Map Information 711 Marker position information 720 Vehicle location information 730 Estimation information 731 Illuminance information 732 Light source position information 733 Sun position information 734 Lighting position information 735 Obstacle location information 736 Structure location information 737 Parked vehicle location information 738 Vehicle Information 740 Brightness Information AR designated area M Marker Mt Target Marker PT Target Route

Claims

1. A mobile object assistance system that supports automatic valet parking in a parking lot and assists a mobile object in recognizing a marker placed in the parking lot, comprising: One or more processors; The one or more processors: A process of acquiring information on a scheduled entry time when the mobile object is scheduled to enter the parking lot; a brightness estimation process for estimating the brightness at the position of the marker in the parking lot at the scheduled entry time without using an image obtained by a camera mounted on the moving object; calculating a luminance correction value of an image including the marker according to the luminance at the position of the marker; acquiring a first image including markers of interest around the moving object using the camera; generating a second image by correcting the luminance of the first image using the luminance correction value for the marker of interest; A process of recognizing the target marker based on the second image; configured to run Mobility support system.

2. The mobile support system according to claim 1, The mobile device further includes a management device that communicates with the mobile device. The management device includes at least a part of the one or more processors and executes the brightness estimation process. Mobility support system.

3. The mobility support system according to claim 2, The management device further executes the process of calculating the luminance correction value. Mobility support system.

4. A mobility support system according to any one of claims 1 to 3, The brightness estimation process includes: A shadow position estimation process for estimating a position of a shadow in the parking lot; a process of estimating the brightness at the position of the marker based on marker position information indicating the position of the marker in the parking lot and the position of the shadow; Includes Mobility support system.

5. The mobility support system according to claim 4, The shadow position estimation process includes a first shadow position estimation process for estimating the position of the shadow generated by an obstacle in the parking lot. Mobility support system.

6. The mobility support system according to claim 5, The first shadow position estimation process includes: A process of acquiring light source position information indicating a position of a light source; A process of acquiring obstacle position information indicating a position of the obstacle; A process of estimating the position of the shadow generated by the light source and the obstacle based on the light source position information and the obstacle position information; Includes Mobility support system.

7. The mobility support system according to claim 6, The obstacles include at least one of other moving objects present in the parking lot and structures installed in the parking lot. Mobility support system.

8. A mobility support system according to any one of claims 4 to 7, The shadow position estimation process includes a second shadow position estimation process for estimating the position of the shadow generated by at least the moving object. Mobility support system.

9. The mobility support system according to claim 8, The second shadow position estimation process includes: A process of acquiring light source position information indicating a position of a light source; A process of acquiring moving object size information indicating a size of the moving object; A process of estimating the position of the shadow generated by the moving object based on the light source position information, the position of the moving object, and the moving object size information; Includes Mobility support system.

10. A mobility support system according to claim 6, 7 or 9, The light source position information is Solar position information indicating the position of the sun which varies depending on the time of day; Lighting position information indicating the installation positions of lighting in the parking lot; Contains at least one of Mobility support system.

11. A mobility support system according to any one of claims 1 to 10, The one or more processors further include: Using the camera mounted on the moving object, an image showing a situation around the moving object is acquired; obtaining marker position information indicating the position of the marker within the parking lot; The image that is assumed to include the target markers around the moving object is acquired as the first image based on the position of the moving object and the marker position information. It was configured as Mobility support system.

12. A mobility assistance system according to any one of claims 1 to 11, The one or more processors further include: obtaining marker position information indicating the position of the marker within the parking lot; A moving object position estimation process is executed to estimate the position of the moving object while correcting the position of the moving object based on the recognition result of the target marker based on the second image and the marker position information. It was configured as Mobility support system.

13. A mobile object support system that supports a mobile object that recognizes markers arranged in a predetermined area, One or more processors; The one or more processors: a brightness estimation process for estimating brightness at the position of the marker within the predetermined area without using an image obtained by a camera mounted on the moving object; calculating a luminance correction value of an image including the marker according to the luminance at the position of the marker; acquiring a first image including markers of interest around the moving object using the camera; generating a second image by correcting the luminance of the first image using the luminance correction value for the marker of interest; A process of recognizing the target marker based on the second image; configured to run The brightness estimation process includes: A shadow position estimation process for estimating a position of a shadow within the predetermined area; estimating the brightness at the position of the marker based on marker position information indicating the position of the marker within the predetermined area and the position of the shadow; Including, the shadow position estimation process includes a second shadow position estimation process that estimates the position of the shadow generated by at least the moving object, The second shadow position estimation process includes: A process of acquiring light source position information indicating a position of a light source; A process of acquiring moving object size information indicating a size of the moving object; A process of estimating the position of the shadow generated by the moving object based on the light source position information, the position of the moving object, and the moving object size information; Includes Mobility support system.

14. A method for assisting a mobile object in automatic valet parking in a parking lot, the method comprising the steps of: acquiring information on a scheduled entry time when the moving object is scheduled to enter the parking lot; estimating brightness at the position of the marker in the parking lot at the scheduled entry time without using an image obtained by a camera mounted on the moving object; calculating a brightness correction value of an image including the marker according to the brightness at the position of the marker; acquiring a first image including target markers around the moving object using the camera; generating a second image by correcting the luminance of the first image using the luminance correction value for the marker of interest; Recognizing the target marker based on the second image; Includes Mobile support method.

15. A method for supporting a moving object that supports a moving object in recognizing a marker arranged in a predetermined area, comprising: Obtaining light source position information indicating a position of a light source; acquiring moving object size information indicating a size of the moving object; estimating a position of a shadow generated by the moving object based on the light source position information, the position of the moving object, and the moving object size information; estimating brightness at the position of the marker based on marker position information indicating the position of the marker within the specified area and the position of the shadow generated by the moving object, without using an image obtained by a camera mounted on the moving object; calculating a brightness correction value of an image including the marker according to the brightness at the position of the marker; acquiring a first image including target markers around the moving object using the camera; generating a second image by correcting the luminance of the first image using the luminance correction value for the marker of interest; Recognizing the target marker based on the second image; Includes Mobile support method.

Images