Mobile body assistance system and mobile body assistance method

Abstract

The present disclosure provides a mobile body that recognizes a marker disposed in a prescribed area, and improves the accuracy of the recognition of the marker by the mobile body. The present disclosure relates to a mobile body assistance system that assists a mobile body that recognizes a marker disposed in a prescribed area. The mobile body assistance system includes one or more processors. The one or more processors are configured to execute the following processes: a process of estimating a luminance at a position of a marker within the prescribed area; a process of calculating a luminance correction value for an image containing the marker, based on the luminance at the position of the marker; a process of acquiring, using a camera, a first image containing an object marker around the mobile body; a process of generating a second image by correcting the luminance of the first image using the luminance correction value for the object marker; and a process of recognizing the object marker based on the second image.

Description

Technical Field

[0001] This disclosure relates to techniques for assisting a moving body in its actions by recognizing markers configured in a designated area. Background Technology

[0002] Patent Document 1 relates to a parking assistance technology that determines the relative positional relationship between a vehicle and a target parking location by identifying markers and calculates a parking trajectory to guide the vehicle to the target parking location. The markers are placed at the target parking location and identified through images captured by a camera mounted on the vehicle. In the parking assistance technology of Patent Document 1, a metering area is defined on the image, and the brightness value of the image is adjusted based on the brightness value of the metering area before processing is performed to identify the markers from the image. Existing technical documents Patent documents

[0003] Patent Document 1: Japanese Patent Application Publication No. 2010-215029 Summary of the Invention The technical problem that the invention aims to solve

[0004] Consider a moving body identifying markers positioned within a designated area. The moving body identifies the markers by acquiring images using a camera. The brightness at the marker's location varies with environmental factors such as weather, time of day, and the presence or absence of streetlights. Additionally, shadows cast on the markers can also cause variations in brightness at their location. If the brightness at the marker's location varies, the moving body may have difficulty identifying the markers. If the moving body cannot accurately identify the markers, for example, the accuracy of the moving body's movements based on the marker identification results will decrease.

[0005] The purpose of this disclosure is to provide a technique that can improve the accuracy of mobile bodies in recognizing markers. Means for solving technical problems

[0006] The first aspect relates to a mobile body assisting system that assists in identifying mobile bodies marked in a designated area. The mobile body assistance system has one or more processors. One or more processors are configured to perform the following processing: Brightness estimation processing estimates the brightness at the location of a marker within a specified area, without using images obtained by a camera mounted on a moving body. The process involves calculating the brightness correction value of the image containing the marker based on the brightness at the marker's location; Processing of acquiring a first image containing object markers around the moving object using a camera mounted on the moving object; The process of generating a second image by correcting the brightness of a first image using brightness correction values ​​for object markers; and Processing based on object labeling in the second image recognition.

[0007] The second aspect relates to a moving body assistance method for assisting in the identification of moving bodies marked in a specified area. Movement-assisted methods include: The brightness at the location of a marker within a defined area is estimated without using images obtained by a camera mounted on a moving body. Calculate the brightness correction value of the image containing the marker based on the brightness at the marker's location; Use a camera mounted on the moving object to obtain a first image containing markers of objects around the moving object; A second image is generated by correcting the brightness of the first image using brightness correction values ​​for object markers; and The object is labeled based on the second image. Invention Effects

[0008] According to this disclosure, a brightness correction value is calculated based on the brightness at the location of the marker to correct the acquired image. By correcting the image using the brightness correction value, the accuracy of marker recognition by moving objects is improved. Attached Figure Description

[0009] Figure 1 This is a conceptual diagram used to illustrate the outline of autonomous valet parking. Figure 2 It is a conceptual diagram used to illustrate technical issues. Figure 3 It is a conceptual diagram used to illustrate technical issues. Figure 4 It is a conceptual diagram used to illustrate technical issues. Figure 5 It is a conceptual diagram used to illustrate technical issues. Figure 6 This is a conceptual diagram illustrating an example of the processing flow of a mobile body assistance system based on this embodiment. Figure 7 This is a conceptual diagram illustrating an example of the processing flow of a mobile body assistance system based on this embodiment. Figure 8 This is a conceptual diagram illustrating an example of the processing flow of a mobile body assistance system based on this embodiment. Figure 9 This is a conceptual diagram illustrating an example of the processing flow of a mobile body assistance system based on this embodiment. Figure 10 This is a block diagram illustrating an example of vehicle configuration. Figure 11 This is a block diagram illustrating an example of the configuration of a management device. Figure 12 This is a block diagram used to illustrate an example of the mobile body auxiliary processing involved in this embodiment. Figure 13 This is a flowchart illustrating a first example of mobile body auxiliary processing according to this embodiment. Figure 14 This is a block diagram used to illustrate an example of the brightness estimation process involved in this embodiment. Figure 15 This is a conceptual diagram used to illustrate the effect of the second shadow position estimation process involved in this embodiment. Figure 16 This is a table used to illustrate examples of the brightness correction values ​​involved in this embodiment. Figure 17 This is a flowchart illustrating a second example of the mobile body auxiliary processing involved in this embodiment. Detailed Implementation

[0010] The embodiments of this disclosure will be described with reference to the accompanying drawings.

[0011] 1. Summary This disclosure relates to a mobile body assistance system that assists in identifying mobile bodies marked within a designated area. Mobile body assistance refers to assisting all aspects related to the mobile body, including monitoring the mobile body, controlling its movements, and managing information related to the mobile body. Examples of designated areas include parking lots and the operating area of ​​mobile buses. Examples of mobile bodies include vehicles and robots. Vehicles may be autonomous vehicles. As an example, the following description considers the case where the mobile body is a vehicle. In general, "vehicle" will be replaced with "mobile body" in the following description.

[0012] Figure 1 This is a conceptual diagram illustrating "Automated Valet Parking (AVP)" as an example of a vehicle 1 recognizing a marker M configured in a designated area AR. In this example, the designated area AR is a parking lot. The entry area is the area where vehicle 1 begins or ends autonomous valet parking, and is contained within the designated area AR. The parking lot can be indoors or outdoors. Multiple markers M are configured in the parking lot.

[0013] Vehicle 1 is an AVP vehicle corresponding to the autonomous valet parking service in the parking lot, capable of autonomous driving at least within the parking lot. Vehicle 1 is equipped with recognition sensors for recognizing the surrounding environment. The recognition sensors include cameras. Vehicle 1 autonomously drives within the parking lot while using the recognition sensors to recognize the surrounding environment.

[0014] Vehicle 1 uses a camera to acquire images showing its surroundings and identifies marker M based on the acquired images. By identifying marker M, Vehicle 1 can perform tasks such as parking lot identification, initial parking position determination, target path correction, target parking position detection, and self-position estimation. For example, Vehicle 1 can perform high-precision self-position estimation (Localization) by combining the identification results of marker M from the camera with the position information of marker M in the parking lot. Alternatively, Vehicle 1 can also identify the parking lot based on the identification results of marker M and confirm that it has entered the correct parking lot. Or, Vehicle 1 can also identify the parking area based on the identification results of marker M.

[0015] The target path PT is the movement path used for vehicle 1 to move towards the target parking frame. The target parking frame is the parking frame assigned to vehicle 1. The target path PT can be a movement path from the entry area to the target parking frame, or a movement path from vehicle 1's current position to the target parking frame. Based on its position estimated through its own position and the target path PT, vehicle 1 automatically moves by following the target path PT. Thus, vehicle 1 can automatically move from the entry area to the target parking frame.

[0016] Management device 2 manages autonomous valet parking in the parking lot. Management device 2 can be a server. Management device 2 can communicate with each vehicle (vehicle 1, parked vehicle 3) in the parking lot. For example, management device 2 can issue entry and exit instructions to vehicle 1. Management device 2 can also track the scheduled exit times of each vehicle (vehicle 1, parked vehicle 3) in the parking lot. Management device 2 can also track the scheduled entry times of vehicles that have scheduled to enter the parking lot. Management device 2 can also provide the location information of marker M in the parking lot to vehicle 1. Management device 2 can also assign target parking frames to vehicle 1. Management device 2 can also generate a target path PT and provide the target path PT information to vehicle 1. Management device 2 can also track the location of each vehicle (vehicle 1, parked vehicle 3) in the parking lot. Management device 2 can also remotely operate each vehicle (vehicle 1, parked vehicle 3) in the parking lot.

[0017] For vehicle 1 to perform its actions correctly, it is crucial that vehicle 1 accurately recognizes marker M. However, if the brightness at the location of marker M changes, vehicle 1 may sometimes fail to recognize marker M correctly. For example, if the location of marker M becomes brighter, the acquired image will be overexposed, and vehicle 1 may sometimes be unable to recognize marker M. Alternatively, if the location of marker M becomes darker, the acquired image will be darker, and the image will appear entirely black, thus vehicle 1 may sometimes be unable to recognize marker M. Therefore, because changes in the brightness at the location of marker M lead to changes in the overall image brightness, the accuracy of marker M recognition may decrease.

[0018] Figures 2 to 5 The diagram conceptually illustrates the conditions of a parking lot in the morning, during the day, at dusk, and at night. The brightness at marker M varies depending on the weather and time of day. During the day, as the sun rises, marker M becomes brighter than in the morning and evening. Conversely, at night, as the parking lot darkens, marker M becomes dimmer. Alternatively, although not illustrated, marker M may also dim on rainy or cloudy days. Thus, changes in the surrounding environment cause variations in the brightness of marker M.

[0019] Furthermore, the brightness at the location of marker M will also change due to the presence of shadows at that location. For example, in Figure 2 , Figure 4 and Figure 5 In the diagram, the shadow cast by the parked vehicle 3 or the walls within the parking lot covers a portion of marker M, causing the brightness at the location of marker M to decrease. The shadow covering marker M varies depending on the position of the light source, such as the sun or streetlights, and the position of obstacles such as the parked vehicle 3 or the walls.

[0020] As a result, due to changes in the surrounding environment and the position of shadows, the accuracy of vehicle 1 in recognizing the mark may sometimes decrease. The moving body assistance system according to this embodiment can improve the accuracy of vehicle 1 in recognizing the mark even when the brightness at the position of the mark M may change.

[0021] 2. Brightness estimation and brightness correction In the following description, "camera image" refers to an image of the surroundings of vehicle 1 obtained by a camera mounted on vehicle 1. The mobility assistance system according to this embodiment improves the accuracy of marker recognition by correcting the brightness of the camera image. Specifically, the mobility assistance system acquires brightness information related to the brightness at the location of marker M. Preferably, the mobility assistance system acquires the brightness information at the location of marker M without using the camera image obtained by the camera mounted on vehicle 1. Based on the brightness information, the mobility assistance system calculates a "brightness correction value" for correcting the brightness of the camera image. The brightness correction value is set to darken an overly bright camera image or brighten an overly dark camera image. Then, the brightness of the camera image is corrected according to the brightness correction value, and marker M is recognized based on the corrected image.

[0022] Figures 6 to 9 This is a diagram illustrating an example of the processing flow related to brightness correction values. Figure 6 First, the user terminal of the parking lot (autonomous valet parking) sends a request for vehicle 1 to enter the parking lot to the management device 2. At this time, the user terminal can also send the scheduled entry time of vehicle 1 along with the entry request. If vehicle 1 can enter the parking lot, the management device 2 approves the entry request and notifies the user terminal that vehicle 1's entry has been approved. After approving the entry request, the management device 2 obtains brightness information. Brightness information is obtained by the management device 2 using brightness estimation information. Brightness estimation information is used to estimate the illuminance, shadow positions, etc., within the parking lot. Brightness estimation information may also include information that takes into account the scheduled entry time of vehicle 1. Specific examples and methods for obtaining brightness estimation information will be described later.

[0023] The management device 2 calculates a brightness correction value for the marker M based on the acquired brightness information. The management device 2 then sends the calculated brightness correction value to the vehicle 1. The vehicle 1 uses the brightness correction value for marker recognition. Specifically, the vehicle 1 uses a camera to acquire an image assuming it includes the marker M surrounding the vehicle 1. The vehicle 1 corrects the brightness of the acquired image using the brightness correction value and recognizes the marker M based on the corrected image. In this way, by performing marker recognition based on the image corrected using the brightness correction value, the impact of brightness variations at the location of the marker M can be reduced, improving the accuracy of marker recognition. Furthermore, the marker M, which is sometimes identified by the vehicle 1, is also referred to as an "object marker."

[0024] exist Figure 6 In the example shown, the acquisition of brightness information and the calculation of brightness correction values ​​are performed by management device 2. In other words, vehicle 1 does not need to acquire brightness information or calculate brightness correction values. Therefore, the processing load on vehicle 1 can be significantly reduced.

[0025] Figure 7 This diagram illustrates another example of the processing flow related to brightness correction values. The calculation of brightness correction values ​​can also be done as follows: Figure 7 As shown, this is carried out by vehicle 1. Figure 7 In this process, the brightness information obtained by the management device 2 is sent to vehicle 1, and vehicle 1 calculates the brightness correction value of the marker M based on the sent brightness information. Vehicle 1 uses a camera to acquire an image assuming that the marker M is included, and corrects the brightness of the acquired image using the brightness correction value. By recognizing the marker M based on the corrected image, vehicle 1 can improve the accuracy of marker recognition.

[0026] exist Figure 7 In the example shown, the acquisition of brightness information is performed by management device 2. That is, vehicle 1 does not need to acquire brightness information. Therefore, the processing load on vehicle 1 can be reduced.

[0027] Figure 8 This diagram illustrates yet another example of the processing flow related to brightness correction values. Brightness information can also be obtained as follows... Figure 8 The process is performed by vehicle 1 as shown. Alternatively, some or all of the brightness estimation information can be sent from management device 2 to vehicle 1. For example, information related to the location of parked vehicle 3 in the parking lot at the scheduled entry time of vehicle 1 can be sent from management device 2 to vehicle 1 as brightness estimation information. Vehicle 1 uses the brightness estimation information to estimate the brightness at the location of marker M, obtaining brightness information. Vehicle 1 calculates a brightness correction value for marker M based on the brightness information. Vehicle 1 uses a camera to acquire an image assuming it contains marker M, and corrects the brightness of the acquired image using the brightness correction value. By recognizing marker M based on the corrected image, vehicle 1 can improve the accuracy of marker recognition.

[0028] Figure 9 This is another example of the processing flow related to the calculation of brightness correction values. In Figure 9 In the example, management device 2 acquires brightness information, calculates brightness correction values, and identifies the marker. Vehicle 1 uses a camera to acquire an image assuming it contains marker M, and sends the acquired image to management device 2. Management device 2 corrects the brightness of the image sent from vehicle 1 using the brightness correction values, and identifies marker M based on the corrected image. By identifying marker M, management device 2 acquires, for example, the location information of vehicle 1, and sends the acquired location information to vehicle 1. Figure 9 In addition, by having the management device 2 recognize the marker M based on the corrected image, the accuracy of marker recognition can also be improved.

[0029] As described below, the brightness at the location marked M is estimated using brightness estimation information. This brightness estimation information is used to estimate illuminance, the location of shadows, etc., within the parking lot. Typically, the brightness estimation information does not include camera images obtained by a camera mounted on vehicle 1. In this case, the mobility assistance system can estimate the brightness at the location marked M without using camera images. As a comparative example, consider estimating the brightness around vehicle 1 based on camera images. In the comparative example, it is necessary to parse each frame of the camera image, which increases the processing load. On the other hand, according to this embodiment, since it is not necessary to estimate the brightness for each frame of the image, the processing load is reduced.

[0030] Brightness estimation processing without using camera images can also be performed in advance before vehicle 1 enters the parking space. Furthermore, brightness correction values ​​can also be calculated in advance before vehicle 1 enters the parking space. By performing necessary processing in advance before vehicle 1 enters the parking space, the processing load after entry can be reduced, allowing vehicle 1 to operate smoothly. In addition, by performing necessary processing in advance, the effects of processing delays can be suppressed. For example, it is possible to prevent situations where the desired mark recognition accuracy cannot be obtained due to processing delays.

[0031] The brightness estimation information may also include the scheduled entry time of vehicle 1. If the scheduled entry time of vehicle 1 is known, the brightness at the location of marker M at the scheduled entry time can be estimated in advance, and the brightness correction value can be calculated in advance. In addition, the scheduled entry time is information unique to autonomous valet parking in parking lots. Performing necessary processing in advance based on the scheduled entry time can be considered a feature unique to autonomous valet parking in parking lots.

[0032] In addition, such as Figure 2 , Figure 4 As shown, sometimes the shadow cast by the parked vehicle 3 partially covers the marker M, causing the brightness at the location of marker M to darken. Therefore, the brightness estimation information can also include information about the parking position of the parked vehicle 3 in the parking lot. In the case of autonomous valet parking, the management device 2 knows the parking position of the parked vehicle 3 in the parking lot. It can be considered that estimating the brightness at the location of marker M based on the parking position of the parked vehicle 3 in the parking lot is also a characteristic unique to autonomous valet parking in parking lots.

[0033] 3. Example of vehicle 1's composition Figure 10 This is a block diagram showing an example of the configuration of vehicle 1. Vehicle 1 includes a vehicle status sensor 11, an identification sensor 12, a communication device 13, a driving device 14, and a control device 15.

[0034] The vehicle status sensor 11 detects the status of the vehicle 1. Examples of vehicle status sensors 11 include vehicle speed sensors (wheel speed sensors), steering angle sensors, yaw rate sensors, and lateral acceleration sensors.

[0035] The identification sensor 12 identifies the surrounding conditions of the vehicle 1. The identification sensor 12 includes a camera. Examples of identification sensors 12 include LiDAR (Laser Imaging Detection and Ranging), radar, and illuminance sensors.

[0036] The communication device 13 communicates with the outside of the vehicle 1. For example, the communication device 13 communicates with the management device 2.

[0037] The driving mechanism 14 includes a steering mechanism, a drive mechanism, and a braking mechanism. The steering mechanism steers the wheels of the vehicle 1. For example, the steering mechanism includes an electric power steering (EPS) system. The drive mechanism is the power source that generates driving force. Examples of drive mechanisms include an engine, an electric motor, and a hub motor. The braking mechanism generates braking force.

[0038] The control device 15 controls the vehicle 1. Specifically, the control device 15 includes one or more processors 16 (hereinafter referred to as processors 16) and one or more storage devices 17 (hereinafter referred to as storage devices 17). The processors 16 perform various processes. The storage devices 17 store various information. Examples of storage devices 17 include volatile memory, non-volatile memory, HDD (Hard Disk Drive), SSD (Solid State Drive), etc. Various processes of the control device 15 are realized by the processors 16 executing a control program, which is a computer program. The control program is stored in the storage devices 17 or recorded on a computer-readable recording medium.

[0039] The processor 16 acquires various information. The acquired information is stored in the storage device 17. The information includes map information 710, vehicle location information 720, brightness estimation information 730, and brightness information 740.

[0040] Map information 710 is map information related to the designated area AR. Map information 710 includes location information of marker M, parking bays, structures, lighting, and the parking area. Map information 710 can be provided to vehicle 1 by parking lot manager or others. Alternatively, map information 710 can also be transmitted from management device 2 to vehicle 1 via communication device 13.

[0041] Vehicle position information 720 includes the position information of vehicle 1. Vehicle position information 720 includes the position information of vehicle 1 calculated based on the vehicle state information obtained by vehicle state sensor 11. Specifically, processor 16 calculates the amount of movement of vehicle 1 based on the vehicle speed and steering angle of vehicle 1 obtained by vehicle speed sensor and steering angle sensor, thereby calculating the position information of vehicle 1. Vehicle position information 720 includes the position information of vehicle 1 calculated in this way.

[0042] Furthermore, the processor 16 corrects the position information of vehicle 1 by comparing the location of marker M shown in map information 710 with the camera's recognition location of marker M. Thus, the processor 16 performs a high-precision self-position estimation of vehicle 1. By repeatedly calculating position information based on vehicle state information and correcting based on marker recognition, the processor 16 can continuously obtain high-precision position information of vehicle 1. Vehicle position information 720 contains the high-precision position information of vehicle 1 obtained through self-position estimation.

[0043] The vehicle location information 720 may also include information related to the target path PT. The target path PT is calculated based on the current location of vehicle 1 or the location of the entry area and the location of the target parking box. The target path PT can also be pre-calculated based on the location of the entry area and the location of the target parking box before vehicle 1 enters the parking area. Alternatively, the target path PT can be calculated based on the current location of vehicle 1 and the location of the target parking box after vehicle 1 enters the parking area. The target path PT can be calculated by the management device 2 and provided to vehicle 1, or it can be calculated by the processor 16.

[0044] Brightness estimation information 730 is information used to estimate the brightness at the location of marker M. An example of brightness estimation information 730 is described later.

[0045] Brightness information 740 indicates the brightness at the location marked M. The method for obtaining brightness information 740 will be described later.

[0046] 4. Configuration example of management device 2 Figure 11 This is a block diagram illustrating an example of the configuration of the management device 2. The management device 2 includes a communication device 23, one or more processors 26 (hereinafter referred to as processors 26), and one or more storage devices 27 (hereinafter referred to as storage devices 27).

[0047] The communication device 23 communicates with the vehicle 1 via a communication network. The communication device 23 can also communicate with the parked vehicle 3. Furthermore, the communication device 23 can also communicate with infrastructure sensors. These infrastructure sensors are sensors installed in the designated area AR, including infrastructure cameras, infrastructure illuminance sensors, etc.

[0048] Processor 26 performs various processes. Storage device 27 stores various information. Examples of storage devices 27 include volatile memory, non-volatile memory, HDD, SSD, etc. Various processes of management device 2 are implemented by a control program executed by processor 26 as a computer program. The control program is stored in storage device 27 or recorded on a computer-readable recording medium.

[0049] Map information 710 can be provided to management device 2 by parking lot managers or others and stored in storage device 27. Processor 26 can communicate with vehicle 1 via communication device 23 and send map information 710 to vehicle 1.

[0050] Vehicle location information 720 includes the location information of vehicle 1, information related to the target path PT, etc.

[0051] The location information of vehicle 1 can also be obtained by the processor 26 communicating with vehicle 1 via the communication device 23. Alternatively, the location information of vehicle 1 can also be obtained by infrastructure cameras installed in the designated area AR.

[0052] The target path PT can be calculated by the processor 26 based on the current position of vehicle 1 or the position of the parking area and the position of the target parking frame. Alternatively, the target path PT calculated by the processor 16 of vehicle 1 can be obtained by the processor 26 communicating with vehicle 1.

[0053] An example of brightness estimation information 730 and a method for obtaining brightness information 740 are described later.

[0054] 5. Assisted processing of moving objects The following is a detailed description of an example of motion assistance processing based on the motion assistance system according to this embodiment.

[0055] Figure 12 This is a block diagram illustrating a functional configuration example of the mobile body assistance system according to this embodiment. The mobile body assistance system includes a brightness estimation unit 110, a brightness correction value calculation unit 120, a vehicle position acquisition unit 130, a first image acquisition unit 140, a second image generation unit 150, and a marker recognition unit 160 as functional blocks. These functional blocks can be implemented by a control program executed by the processor 16 as a computer program, or by a control program executed by the processor 26 as a computer program. Alternatively, the respective functional blocks can be implemented through distributed processing by the processor 16 and the processor 26.

[0056] Figure 13 This is a flowchart illustrating a first example of the mobile body auxiliary processing according to this embodiment. (Refer to...) Figure 12 and Figure 13 The first example of mobile body assisted processing will be described.

[0057] 5-1. Brightness estimation processing (step S110) In step S110, the brightness estimation unit 110 performs brightness estimation processing at the position of the estimation mark M. The brightness estimation processing can be performed before or after the vehicle 1 enters the parking space.

[0058] 5-1-1. Information for brightness estimation The brightness estimation information 730 is the information referenced when estimating brightness. The brightness estimation unit 110 estimates the brightness at the position of the mark M based on the position information of the mark M and the brightness estimation information 730, and obtains the brightness information 740.

[0059] Figure 14 This is a block diagram illustrating an example of the brightness estimation process performed by the brightness estimation unit 110. The brightness estimation unit 110 includes a shadow position estimation unit 111. The shadow position estimation unit 111 includes a first shadow position estimation unit 112 and a second shadow position estimation unit 113. The marker position information 711 is the position information of the marker M within the defined area AR, obtained from the map information 710. The brightness estimation information 730 includes illuminance information 731, light source position information 732, obstacle position information 735, and vehicle information 738.

[0060] Illuminance information 731 indicates at least one of the illuminance at the location marked M and the illuminance of the designated area AR. For example, the illuminance is estimated based on the date, time of day, weather information, sunshine information, etc. As another example, the illuminance can also be detected by an illuminance sensor. The illuminance sensor can be an infrastructure illuminance sensor installed in the designated area AR, or it can be an on-board illuminance sensor mounted on vehicle 1.

[0061] Light source location information 732 indicates the location of the light source. Light source location information 732 includes at least one of sun location information 733 and lighting location information 734. Sun location information 733 indicates the location of the sun, calculated based on the date and time of day. Lighting location information 734 is information related to lighting installed in the designated area AR, including information related to the location of the lighting installation. Lighting includes streetlights installed in the designated area AR. The information related to the location of the lighting installation is obtained from map information 710.

[0062] Obstacle location information 735 indicates the location of obstacles that may cast shadows within the designated area AR. Obstacle location information 735 includes at least one of structure location information 736 and parked vehicle location information 737.

[0063] Structure location information 736 indicates the location of structures situated within the designated area AR. Examples of structures include columns and walls. Structure location information 736 is obtained from map information 710.

[0064] The parking vehicle location information 737 indicates the location of the parked vehicle 3 within the designated area AR. The parking vehicle location information 737 can be obtained by communication between the management device 2 and the parked vehicle 3. Alternatively, the parking vehicle location information 737 can also be obtained by communication between the management device 2 and the infrastructure camera. The management device 2 can also send the obtained parking vehicle location information 737 to vehicle 1.

[0065] Vehicle information 738 includes at least one of the current or future positions of vehicle 1. The current position of vehicle 1 is obtained based on vehicle position information 720. The current position of vehicle 1 can be the position information of vehicle 1 calculated based on vehicle status information, the position information of vehicle 1 obtained through high-precision self-position estimation, or the information obtained through infrastructure cameras. The future position of vehicle 1 is obtained as the position of vehicle 1 on the target path PT. The target path PT is obtained based on vehicle position information 720. Vehicle information 738 may also include vehicle size information showing the dimensions of vehicle 1. The dimensions of vehicle 1 are at least one of the length, width, and height of vehicle 1. The vehicle size information can be obtained in advance by the storage device 17 of vehicle 1. The vehicle size information can also be provided to the management device 2 and stored in the storage device 27.

[0066] The shadow position estimation unit 111 performs shadow position estimation processing to estimate the position of shadows within the specified area AR. The shadow position includes a first shadow position estimated by the first shadow position estimation unit 112 and a second shadow position estimated by the second shadow position estimation unit 113.

[0067] The first shadow position estimation unit 112 estimates the position of the shadow generated by the light source and the obstacle within the designated area AR, which is the first shadow position. The position of the light source is obtained based on the light source position information 732. The position of the obstacle within the designated area AR is obtained based on the obstacle position information 735. The first shadow position estimation unit 112 performs a first shadow position estimation process, that is, it estimates the first shadow position based on the light source position information 732 and the obstacle position information 735.

[0068] The second shadow position estimation unit 113 estimates the position of the shadow generated by the light source and vehicle 1, i.e., the second shadow position. The position of the light source is obtained based on the light source position information 732. The position of vehicle 1 is obtained based on the vehicle information 738, which is either the current position or the future position of vehicle 1. If brightness estimation processing is performed before vehicle 1 enters the parking space, the position of vehicle 1 obtained by the second shadow position estimation unit 113 is the future position of vehicle 1. The second shadow position estimation unit 113 performs second shadow position estimation processing, that is, it estimates the second shadow position based on the light source position information 732 and the vehicle information 738. In the second shadow position estimation processing, in addition to the position of the light source and the position of vehicle 1, vehicle size information can also be used to estimate the second shadow position. The vehicle size information is obtained based on the vehicle information 738.

[0069] Figure 15 An example is shown where a shadow is generated at the location of mark M due to the light source and vehicle 1, and the brightness at the location of mark M changes. Since the shadow location estimation unit 111 includes a second shadow location estimation unit 113, therefore... Figure 15 In that case, the brightness at the location of marker M can also be accurately estimated.

[0070] The brightness estimation process performed by the brightness estimation unit 110 includes estimating the brightness at the location of the marker M based on illuminance information 731 and marker position information 711. The brightness estimation process may also include estimating the brightness at the location of the marker M based on the shadow position and marker position information 711 obtained through shadow position estimation processing. The brightness information 740 obtained through the brightness estimation process can be obtained for all markers M in the parking lot at once, or it can be obtained for only a portion of the markers M. In the case of obtaining information for only a portion of the markers M, for example, it may be obtained only for markers M located near the future position of vehicle 1.

[0071] 5-1-2. Scheduled warehousing time The brightness estimation unit 110 can also obtain information related to the scheduled entry time of vehicle 1 and perform brightness estimation processing using the brightness estimation information 730 at the scheduled entry time. The scheduled entry time is obtained by processor 16 or processor 26 by being sent from a user terminal or the like to management device 2 or vehicle 1.

[0072] The illuminance information at the scheduled storage time 731 is estimated based on the season, the sun's position at the storage time, the weather information at the storage time, and sunshine information.

[0073] The light source position information 732 at the scheduled storage time includes at least one of the solar position information 733 and the lighting position information 734 at the scheduled storage time. The solar position information 733 at the scheduled storage time is calculated based on the season and the scheduled storage time.

[0074] The obstacle location information 735 at the scheduled entry time includes at least one of the structure location information 736 and the parking vehicle location information 737 at the scheduled entry time.

[0075] The parking vehicle location information 737 at the scheduled entry time can be calculated by the management device 2 communicating with the user terminal, etc., to obtain the scheduled exit time of the parking vehicle 3 and the scheduled entry time of the AVP vehicle. The management device 2 can also send the obtained parking vehicle location information 737 at the scheduled entry time to vehicle 1.

[0076] The vehicle information 738 at the scheduled entry time is information related to the future location of vehicle 1. The future location of vehicle 1 is obtained as the position of vehicle 1 on the target path PT.

[0077] 5-2. Brightness correction value calculation and processing (step S120) In step S120, the brightness correction value calculation unit 120 calculates a brightness correction value. The brightness correction value is a value used to correct the brightness of an image containing the markers M acquired by the camera, and is calculated for each marker M based on the brightness information 740. The brightness correction value is set to darken an overly bright image or brighten an overly dark image. That is, the brightness correction value is set in a way that makes the markers M easier to identify. The brightness correction value can be a value used to correct the brightness of each pixel in the image, or it can be a value used to correct the hue based on the brightness of the image. The brightness correction value can be obtained all at once for all markers M in the parking lot, or it can be obtained only for a portion of the markers M. The brightness correction value calculation process can be performed before vehicle 1 enters the parking lot, or it can be performed after vehicle 1 enters the parking lot.

[0078] 5-3. Vehicle location acquisition process (step S130) In step S130, the vehicle location acquisition unit 130 acquires the location information of vehicle 1. The location information of vehicle 1 acquired by the vehicle location acquisition unit 130 is calculated based on the vehicle status information and obtained from the vehicle location information 720. Alternatively, the location information of vehicle 1 acquired by the vehicle location acquisition unit 130 can also be obtained through an infrastructure camera. Processing after step S130 is performed after the vehicle enters the parking space.

[0079] 5-4. First Image Acquisition Processing (Step S140) In step S140, the first image acquisition unit 140 uses a camera mounted on the vehicle 1 to acquire a first image assuming it contains an object marker Mt. The object marker Mt is a marker among markers M located near the current position of the vehicle 1. The object marker Mt is determined by estimating the marker M located near the current position of the vehicle based on the position information of the vehicle 1 acquired by the vehicle position acquisition unit 130 and the marker position information 711.

[0080] 5-5. Second image generation process (step S150) In step S150, the second image generation unit 150 uses the brightness correction value for the object mark Mt to correct the first image and obtain the second image.

[0081] Figure 16 This is a table showing examples of brightness correction values. Brightness correction values ​​can also be, for example,... Figure 16 The coefficients are determined for each category, such as daytime, nighttime, and the presence or absence of shadows, as shown in the table. The second image is generated, for example, by correcting the brightness of each pixel in the first image using these coefficients. The coefficients are set such that the brightness at the location marked M is 1 at a reference brightness level; the brighter the image, the smaller the coefficient value, and the darker the image, the larger the coefficient value. The second image generation unit 150 calculates the brightness of each pixel in the image acquired by the camera. When the coefficient is less than 1, a stronger correction is applied to the brighter pixels. Conversely, when the coefficient is greater than 1, a stronger correction is applied to the darker pixels. In this way, by correcting the brightness of each pixel, a second image is generated.

[0082] 5-6. Tag recognition processing (step S160) In step S160, the marker recognition unit 160 recognizes the object marker Mt based on the second image. When the marker recognition unit 160 obtains the recognition result of the object marker Mt, the processing of the current loop ends.

[0083] 5-7. Effects Through the motion-assisted processing described above, a brightness correction value is calculated based on the brightness at the location of marker M to correct the brightness of the image containing marker M (object marker Mt). By using this brightness correction value to correct the brightness of the first image, the motion-assisted system can improve the recognition accuracy of object marker Mt. By improving the recognition accuracy of object marker Mt, the accuracy of the vehicle 1's movements based on the recognition result of object marker Mt also improves.

[0084] Furthermore, in the first example, the brightness estimation processing and brightness correction value calculation processing can be performed in advance before vehicle 1 enters the parking space. By pre-calculating the brightness correction value, the time from when vehicle 1 acquires an image to when mark recognition is performed can be shortened, allowing vehicle 1 to move smoothly. Additionally, since it is not necessary to calculate the brightness correction value every time vehicle 1 moves, the processing load on the processor 16 of vehicle 1 and the processor 26 of management device 2 can also be reduced. Even if the brightness correction value calculation processing is performed after vehicle 1 enters the parking space, since the brightness information and brightness correction value are obtained using information such as weather and time, it can be performed before the image is acquired by the camera. Compared to checking the brightness of the image every time the image is acquired by the camera, the processing load on the processor 16 of vehicle 1 can be reduced.

[0085] 6. The second example of mobile body assisted processing Figure 17 This is a flowchart illustrating a second example of the mobile body auxiliary processing according to this embodiment. (Refer to...) Figure 17 The second example of mobile body assisted processing will be explained.

[0086] 6-1. Vehicle location acquisition process (step S210) In step S210, the vehicle location acquisition unit 130 acquires the location information of vehicle 1. The location information of vehicle 1, as information related to the current location of vehicle 1, is obtained based on the vehicle location information 720. Alternatively, the location information of vehicle 1 can also be obtained through an infrastructure camera. In the second example, processing after step 210 is performed after vehicle 1 has entered the warehouse.

[0087] 6-2. Brightness estimation processing (step S220) In step S220, the brightness estimation unit 110 estimates the brightness at the position of the mark M. The brightness estimation unit 110 estimates the brightness at the position of the mark M based on the position information of the mark M and the brightness estimation information 730, and obtains the brightness information 740.

[0088] Among the information included in the brightness estimation information 730, vehicle information 738 is information related to the current position of vehicle 1. The current position of vehicle 1 is obtained based on vehicle position information 720. Other information included in the brightness estimation information 730 is obtained by the brightness estimation unit using the same method as in step 110.

[0089] 5-2. Brightness correction value calculation and processing (step S230) In step S230, the brightness correction value calculation unit 120 calculates the brightness correction value. The processing in step S230 is related to... Figure 13 The processing in step S120 is the same. After step S240, the process is performed similarly. Figure 13The same process applies after step S140.

[0090] 5-3. Effects Similar to the first example, the moving body assistance system improves the accuracy of marker recognition by correcting the image brightness using a brightness correction value. By improving the accuracy of marker recognition, the accuracy of vehicle 1's movements also improves.

[0091] In the second example, brightness estimation and brightness correction value calculation are performed after the current position of vehicle 1 is obtained. Since the current position of vehicle 1 is used to estimate the brightness information 740, the error of the brightness information 740 can be reduced. In addition, in the second example, the image obtained by the camera is not required in obtaining the brightness information and brightness correction value. Compared with the case where the brightness information and brightness correction value are calculated using the image, the processing load of the processor 16 of vehicle 1 can be reduced.

[0092] 6. Other implementation methods This disclosure can also be applied to autonomous valet parking of vehicle 1 in a parking lot, in addition to other applications. For example, this disclosure can also be applied to autonomous valet parking in which an autonomous robot tows a vehicle that does not have autonomous driving capabilities. Furthermore, this disclosure can also be applied when localized processing is performed on street markers M, vehicle or robot identification markers M, etc.

[0093] In general, the term "vehicle" in the above description is replaced with "moving body". Symbol Explanation

[0094] 1. Vehicle; 2. Management device; 3. Parked vehicle; 11. Vehicle status sensor; 12. Identification sensor; 13. Communication device; 14. Driving device; 15. Control device; 16. Processor; 17. Storage device; 23. Communication device; 26. Processor; 27. Storage device; 110. Brightness estimation unit; 111. Shadow position estimation unit; 112. First shadow position estimation unit; 113. Second shadow position estimation unit; 120. Brightness correction value calculation unit; 130. Vehicle position acquisition unit; 140. First image acquisition unit; 150. Second image generation unit; 160. Marker recognition unit; 710. Map information; 711. Marker position information; 720. Vehicle position information; 730. Estimation information; 731. Illuminance information; 732. Light source position information; 733. Sun position information; 734. Illumination position information; 735. Obstacle position information; 736. Structure position information; 737. Parked vehicle position information; 738. Vehicle information; 740. Brightness information; AR designated area; M mark; Mt object mark; PT target path.

Claims

1. A mobile body assistance system for assisting in the identification of mobile bodies marked in a parking lot, the mobile bodies corresponding to autonomous valet parking vehicles in the parking lot. The mobility assistance system has one or more processors. The one or more processors are configured to perform the following processes: Processing to obtain information on the scheduled entry time of the mobile vehicle into the parking lot; Brightness estimation processing: without using images obtained by the camera mounted on the mobile body, the brightness at the location of the mark in the parking lot at the predetermined entry time is estimated before the mobile body enters the parking lot; Processing of calculating the brightness correction value of the image containing the mark based on the brightness at the position of the mark at the predetermined storage time before the moving body is stored; Processing of acquiring a first image containing object markers around the moving object using the camera; The process of generating a second image by correcting the brightness of the first image using the brightness correction value for the object marker; as well as The process of identifying the object marker based on the second image.

2. The mobile body assist system according to claim 1, wherein, The mobile body assistance system also includes a management device for communicating with the mobile body. The management device includes at least a portion of the one or more processors and performs the brightness estimation process.

3. The mobile body assist system according to claim 2, wherein, The management device also performs the process of calculating the brightness correction value.

4. The mobile body assist system according to claim 1, wherein, The brightness estimation process includes: Shadow position estimation processing to estimate the position of shadows within the parking lot; and The process of estimating the brightness at the location of the mark based on the mark location information showing the location of the mark in the parking lot and the location of the shadow.

5. The mobile body assist system according to claim 4, wherein, The shadow position estimation process includes a first shadow position estimation process that estimates the position of the shadow generated by obstacles in the parking lot.

6. The mobile body assist system according to claim 5, wherein, The first shadow position estimation process includes: Processing to obtain light source position information that indicates the position of the light source; Processing to obtain obstacle position information showing the location of the obstacle; and The process of estimating the position of the shadow generated by the light source and the obstacle based on the light source position information and the obstacle position information.

7. The mobile body assist system according to claim 6, wherein, The obstacle includes at least one of other moving bodies present in the parking lot and structures disposed in the parking lot.

8. The mobile body assist system according to any one of claims 4 to 7, wherein, The shadow position estimation process includes a second shadow position estimation process that estimates the position of at least the shadow generated by the moving body.

9. The mobile body assist system according to claim 8, wherein, The second shadow position estimation process includes: Processing to obtain light source position information indicating the position of the light source; and Processing to obtain moving body size information showing the size of the moving body; and The process of estimating the position of the shadow generated by the moving body based on the light source position information and the size information of the moving body.

10. The mobile body assist system according to claim 6 or 7, wherein, The light source location information includes at least one of the sun's location information and the lighting location information. The solar position information shows the changing position of the sun over time. The lighting location information indicates the location of the lighting within the parking lot.

11. The mobile body assist system according to claim 1, wherein, The one or more processors are further configured to: Using the camera mounted on the mobile body, images showing the surrounding conditions of the mobile body are acquired. Obtain the marker location information showing the location of the marker within the parking lot. Based on the position of the moving object and the position information of the marker, the image assuming to include the object markers around the moving object is obtained as the first image.

12. The mobile body assist system according to claim 1, wherein, The one or more processors are further configured to: Obtain the marker location information showing the location of the marker within the parking lot. Based on the recognition result of the object marker based on the second image and the marker position information, a movement position estimation process is performed to estimate the position of the moving object while correcting its position.

13. A mobile body assistance method, which assists in identifying a mobile body marked in a parking lot, the mobile body corresponding to an autonomous valet parking vehicle in the parking lot, comprising the following steps: Obtain information on the scheduled entry time of the mobile vehicle into the parking lot; Without using images obtained by a camera mounted on the mobile body, the brightness of the mark at the location in the parking lot at the predetermined entry time is estimated before the mobile body enters the parking lot; Based on the brightness at the location of the mark at the predetermined storage time, calculate the brightness correction value of the image containing the mark before the moving body is stored. The camera is used to acquire a first image containing object markers around the moving object; A second image is generated by correcting the brightness of the first image using the brightness correction value for the object marker; as well as The object marker is identified based on the second image.

Images