Information processing device, information processing method, and information processing program

Abstract

It can detect a stationary vehicle that is about to depart from a designated location. [Solution] The information processing device includes a reception unit that receives location specifications from a terminal device, and an acquisition unit that acquires real-time images captured by one or more cameras that are capturing the specified location. The system includes a detection unit that detects vehicles parked at a designated location that are expected to depart soon, based on real-time images, and a transmission unit that transmits the detection results from the detection unit to a terminal device.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and an information processing program.

Background Art

[0002] There is known a device that can determine whether another vehicle starts based on the state of another vehicle during parking and control the travel of the host vehicle.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the prior art, it is impossible to predict the occurrence of a parking space from a place where temporary parking is possible. For example, in the prior art, in order to control the driving of the host vehicle, it determines the presence or absence of another vehicle that is about to start. However, since it determines the situation of another vehicle while the host vehicle is traveling, it has no choice but to make a judgment based on the instantaneous state of another vehicle, and as a result, the determination accuracy cannot be said to be high. Also, when a driver wants to park the vehicle on the road for a short time for pick-up / drop-off or a temporary break, among the parked vehicles, the driver has to go to the actual place where he / she wants to park to find out which vehicle will start soon.

[0005] The present invention has been made in view of the above, and an object thereof is to provide an information processing apparatus, an information processing method, and an information processing program that can predict the occurrence of a parking space from a place where temporary parking is possible.

Means for Solving the Problems

[0006] To solve the above-mentioned problems and achieve the objective, the information processing device according to the present invention is characterized by comprising: a reception unit that receives location specification from a terminal device; an acquisition unit that acquires real-time images captured by one or more cameras that are capturing the location; a detection unit that detects a vehicle parked at the location that is expected to depart soon, based on the real-time images; and a transmission unit that transmits the detection result of the detection unit to the terminal device.

[0007] Furthermore, the information processing method according to the present invention is an information processing method performed by an information processing device, and is characterized by including: a reception step of receiving a location specification from a terminal device; an acquisition step of acquiring real-time images taken by one or more cameras that are photographing the location; a detection step of detecting a vehicle that is parked at the location and is expected to depart soon, based on the real-time images; and a transmission step of transmitting the detection result of the detection step to the terminal device.

[0008] Furthermore, the information processing program according to the present invention is characterized by causing a computer to execute the following: a reception procedure for receiving a location specification from a terminal device; an acquisition procedure for acquiring real-time images captured by one or more cameras that are capturing the location; a detection procedure for detecting a vehicle that is parked at the location and is expected to depart soon, based on the real-time images; and a transmission procedure for transmitting the detection result of the detection procedure to the terminal device. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 shows an example of the overall flow of the process for determining whether there is a parking space available in Embodiment 1. [Figure 2] Figure 2 is a diagram illustrating the process of determining whether or not there is a parking space available for the information processing device according to Embodiment 1. [Figure 3] Figure 3 is a block diagram showing an example configuration of the information processing system according to Embodiment 1. [Figure 4] Figure 4 shows an example of the processing content of the three-dimensional processing according to Embodiment 1. [Figure 5] Figure 5 shows an example of the processing content for assigning priority according to Embodiment 1. [Figure 6] Figure 6 shows an example of a display image when there is an undeterminable region according to Embodiment 1. [Figure 7] Figure 7 shows an example of the processing content of the privacy processing according to Embodiment 1. [Figure 8] Figure 8 shows an example of the processing content of a swipe operation according to Embodiment 1. [Figure 9] Figure 9 is a flowchart showing an example of the process for determining whether there is a parking space available according to Embodiment 1. [Figure 10] Figure 10 shows an example of the processing content of the continuous monitoring process according to Embodiment 2. [Figure 11] Figure 11 is a flowchart showing an example of the processing flow of the continuous monitoring process according to Embodiment 2. [Figure 12] Figure 12 is a diagram showing an example of the process for determining whether there is a parking space available, according to a modified example of Embodiment 2. [Figure 13] Figure 13 shows an example of the flow of the detection process for the most recently departing vehicle in Embodiment 3. [Figure 14] Figure 14 is a diagram illustrating the processing content for detecting the most recently departing vehicle in the information processing device according to Embodiment 3. [Figure 15] Figure 15 is a block diagram showing an example configuration of an information processing device according to Embodiment 3. [Figure 16] Figure 16 shows an example of the training process for an estimation model using video data. [Figure 17] Figure 17 shows an example of the training process for an estimation model using abstracted video data. [Figure 18] Figure 18 shows an example of the training process for an estimation model using symbolized video data. [Figure 19]FIG. 19 is a diagram showing an example of the content of the learning process of an estimation model including a starting probability. [Figure 20] FIG. 20 is a diagram showing an example of the content of the detection process of the most recent starting vehicle using a learned estimation model. [Figure 21] FIG. 21 is a diagram showing an example of the content of the detection process of the most recent starting vehicle using a learned model including a starting probability. [Figure 22] FIG. 22 is a diagram showing an example of the content of the image analysis process according to Embodiment 3. [Figure 23] FIG. 23 is a flowchart showing an example of the flow of the learning process of the estimation model according to Embodiment 3. [Figure 24] FIG. 24 is a flowchart showing an example of the flow of the content of the detection process of the most recent starting vehicle according to Embodiment 3. [Figure 25] ​​​​​​​​​​​​​​Figure 1 is a diagram illustrating an example of the overall flow of the process for determining the presence or absence of a parking space in Embodiment 1. As shown in Figure 1, the information processing system comprises a terminal device 10 mounted on the vehicle and a determination device 20, which is an information processing device. The terminal device 10 and the determination device 20 communicate with each other via a wireless communication network, which is Network NE. In this information processing system, the determination device 20 acquires real-time image data of a designated location, determines the presence or absence of a parking space based on the acquired image data, and transmits the determination result to the terminal device 10.

[0012] The terminal device 10 is implemented by a computer or the like installed in the vehicle. For example, the terminal device 10 may be a dedicated navigation device built into or mounted in the vehicle. For example, the terminal device 10 may be a single device having both a navigation function and a display function.

[0013] Furthermore, drivers and passengers (hereinafter sometimes referred to as "drivers, etc.") riding in a vehicle can substitute a portable terminal device they use on a daily basis (e.g., a smartphone, tablet, notebook PC, desktop PC, PDA, etc.) with a predetermined application installed, and use it as the terminal device 10. For example, a portable terminal device connected to the determination device 20 can be understood as the terminal device referred to here. When a portable terminal device is used as the terminal device 10, it is installed, for example, on the dashboard of the vehicle while driving.

[0014] The determination device 20 receives a parking location specified by the driver or other user from the terminal device 10. For example, when the determination device 20 receives a designated location from the terminal device 10, it acquires real-time images from a camera installed at the designated location or from an on-board camera of a vehicle located near the designated location. Based on the real-time images, the determination device 20 determines whether there is an available parking space and transmits the determination result to the terminal device 10.

[0015] Here, we will explain the processing of the determination device 20 using the example of a situation where a driver picks up someone at a station or similar location. As shown in Figure 1, for example, when picking someone up at a station, the driver specifies the location where they want to stop using the terminal device 10 prior to arriving at the station. The determination device 20 receives the location specification from the terminal device 10 via the network, determines whether there is an available space to stop, and transmits the determination result to the terminal device 10. The driver selects an available space to stop from the determination results displayed on the terminal device 10 (S1). After that, the driver moves to the designated available space (S2). At this time, the driver may also use the navigation function of the terminal device 10. Then, the driver temporarily stops at the designated available space and picks up the person (S3).

[0016] (Processing details of the determination device 20) The processing details of the determination device 20 according to Embodiment 1 will now be described. The determination device 20 according to Embodiment 1 receives a designated location, acquires a real-time image of the designated location, determines whether there is an available space for parking based on the real-time image, and transmits the determination result to the terminal device 10.

[0017] Here, we will explain the processing content of the determination device 20 with specific examples. Figure 2 is a diagram illustrating the processing content of the information processing device (determination device 20) according to Embodiment 1.

[0018] (1) The terminal device 10 displays a map to the driver showing areas where parking is permitted. The terminal device 10 then transmits to the determination device 20 the location (such as a rotary in front of a train station) that the driver has designated as the place where they wish to park from among the areas where parking is permitted. The determination device 20 accepts the designated location from the terminal device 10. (2) The determination device 20 acquires real-time images taken by fixed cameras, on-board cameras, etc., located at the designated location. The determination device 20 also has prior information on each location where parking is permitted, and is aware of each camera that takes pictures at each location where parking is permitted, and is able to communicate with each camera.

[0019] (3) The determination device 20 creates three-dimensional spatial information of the designated location based on the information indicating the shooting position and shooting direction attached to the acquired real-time image, and the real-time image, and refers to the vehicle information to determine whether there is an available space where a vehicle can park. (4) The determination device 20 transmits the determination result to the terminal device 10. If the determination device 20 determines that there is an available space, it may transmit information showing the location of the available space on a map along with the determination result, or it may transmit a real-time image or video that includes the available space.

[0020] (Functional configuration of information processing systems) Next, an example of the functional configuration of the information processing system according to Embodiment 1 will be described. Figure 3 is a block diagram showing an example configuration of the terminal device 10 and the determination device 20 included in the information processing system according to Embodiment 1.

[0021] (Configuration of terminal device 10) As shown in Figure 3, the terminal device 10 consists of a communication unit 11, an input unit 12, a display unit 13, a control unit 14, a storage unit 15, and the like. The communication unit 11 transmits and receives information to and from the determination device 20 via the network NE under the control of the control unit 14.

[0022] The input unit 12 includes input devices such as a touch panel, keyboard, and mouse, and accepts various data inputs in response to operations by the driver, etc. The input unit 12 then outputs the various data it has received to the communication unit 11, storage unit 15, etc., under the control of the control unit 14. For example, the driver inputs the location where they wish to stop into the input unit 12. The control unit 14 then stores the input location in the storage unit 15 and further transmits it to the determination device 20 via the communication unit 11.

[0023] The display unit 13 is composed of a display display using liquid crystal or organic EL (Electro Luminescence), a touch panel, etc., and displays various images under the control of the control unit 14. For example, when a predetermined operation is received by the driver via the input unit 12, the display unit 13 displays a map showing areas where parking is permitted. The display unit 13 also displays judgment results received from the judgment device 20, display images, etc.

[0024] The control unit 14 is implemented by a controller such as a CPU (Central Processing Unit) or MPU (Micro Processing Unit) executing various programs stored in the memory unit 15, thereby controlling the operation of the entire terminal device 10. Note that the control unit 14 may be composed of integrated circuits other than a CPU or MPU, such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

[0025] For example, when the control unit 14 receives a predetermined operation from the driver via the input unit 12, it obtains a map from the storage unit 15 showing areas where parking is permitted. The control unit 14 then displays the obtained map on the display unit 13 and, when it receives an operation via the input unit 12 to specify a desired parking location from among the permitted parking areas, it instructs the communication unit 11 to transmit the specified location to the determination device 20. Furthermore, when the control unit 14 receives a specification of an available parking space from the driver via the input unit 12, it instructs the communication unit 11 to transmit the specified available space to the determination device 20.

[0026] The memory unit 15 stores various programs executed by the control unit 14, and various data necessary for the control unit 14 to perform processing, such as maps showing areas where parking is permitted.

[0027] (Configuration of the determination device 20) As shown in Figure 3, the determination device 20 consists of a communication unit 21, a control unit 22, a storage unit 23, and the like.

[0028] The communication unit 21, under the control of the control unit 22, transmits and receives information with the terminal device 10 (communication unit 11) via the network NE. The communication unit 21 also receives real-time images captured by a fixed camera or in-vehicle camera at a designated location via the network NE, under the control of the control unit 22.

[0029] The storage unit 23 stores various programs executed by the control unit 22 (information processing programs according to this embodiment), as well as data necessary for the control unit 22 to perform processing. As shown in Figure 3, this storage unit 23 includes an image / video database 231, etc.

[0030] The image / video database 231 stores real-time images and videos (sometimes simply referred to as real-time images) captured by fixed cameras or in-vehicle cameras at specified locations, which are received by the communication unit 21. The image / video database 231 also stores the judgment results of the image judgment unit 223 and stores display images etc. generated by the generation unit 224.

[0031] The control unit 22 is implemented by a controller such as a CPU or MPU executing various programs (including the information provision program according to this embodiment) stored in the storage unit 23, and controls the operation of the entire determination device 20. Note that the control unit 22 may be composed of integrated circuits such as ASICs or FPGAs, not just a CPU or MPU.

[0032] As shown in Figure 3, the control unit 22 includes a reception unit 221, an image acquisition unit 222, an image determination unit 223, a generation unit 224, a transmission unit 225, and the like.

[0033] The reception unit 221 receives information about the location where the driver wishes to stop. In Embodiment 1, the reception unit 221 receives the location specified by the driver on the terminal device 10 via the communication unit 11 and the communication unit 21. The reception unit 221 outputs the information of the specified location to the image acquisition unit 222. The information received by the reception unit 221 may include vehicle information of the vehicle that wishes to stop, in addition to the specified location.

[0034] The image acquisition unit 222 acquires real-time images from fixed cameras, vehicle-mounted cameras, etc., located at a designated location, as received by the reception unit 221, via the communication unit 21. The real-time images may include information such as the shooting position and shooting direction. The image acquisition unit 222 outputs the acquired real-time images to the image determination unit 223.

[0035] The image determination unit 223 determines whether there is an available parking space based on the real-time image acquired by the image acquisition unit 222, and outputs the determination result to the generation unit 224 or the transmission unit 225. For example, the image determination unit 223 creates three-dimensional spatial information of a specified location based on the real-time image, for example, using known NeRF (Neural Radiance Fields) technology. The image determination unit 223 then determines whether there is an available parking space based on the created three-dimensional spatial information.

[0036] Furthermore, if the image determination unit 223 has vehicle information including the make or size of the vehicle that wishes to park, it determines whether parking is possible based on the extracted three-dimensional spatial information of the empty space and the make or size of the vehicle that wishes to park, and determines that there is an empty space where parking is possible only if parking is possible. On the other hand, if the image determination unit 223 does not have vehicle information to park, it determines that there is an empty space where parking is possible if the size of the extracted empty space exceeds a preset threshold.

[0037] Furthermore, the image determination unit 223 measures the size of the empty space based on the three-dimensional spatial information and outputs the size of the empty space along with the determination result to the generation unit 224. The image determination unit 223 may also output vehicle type information suitable for the size of the empty space to the generation unit 224.

[0038] Figure 4 is a diagram showing an example of the processing content of the three-dimensional processing according to Embodiment 1. The image determination unit 223 acquires real-time images from the image acquisition unit 222. The image determination unit 223 creates three-dimensional spatial information based on the real-time images taken by each of the multiple cameras, as well as the shooting position and shooting direction assigned to each real-time image.

[0039] Next, the image determination unit 223 extracts available space based on three-dimensional spatial information. The image determination unit 223 extracts the three-dimensional available space and measures its size. In the example in Figure 4, the image determination unit 223 determines that the available space is at least 5,000 mm deep, 2,300 mm wide, and 2,000 mm high. Subsequently, the image determination unit 223 compares this with its owned vehicle information (total length 4,600 mm, total width 1,780 mm, total height 1,420 mm) and determines that parking is possible. Finally, the image determination unit 223 determines that there is available space where parking is possible.

[0040] Furthermore, if multiple available parking spaces are identified, the image determination unit 223 may assign a predetermined priority order to each space. For example, when a driver specifies a desired parking location, they can pre-determine the priority order, such as proximity to the station, size of the space, or the distance from the station that can be covered by rain.

[0041] Figure 5 shows an example of the processing content for assigning priority according to Embodiment 1. For example, as shown in Figure 5A, when multiple available parking spaces are extracted, the image determination unit 223 compares them with map information and measures the walking distance from the station to each available space. The image determination unit 223 then assigns a ranking to the spaces in order of proximity to the station and outputs it to the generation unit 224.

[0042] Furthermore, if the image determination unit 223 is unable to acquire sufficient real-time images to determine the presence or absence of empty space, such as due to insufficient real-time images or blind spots, it outputs to the generation unit 224 that there are areas that could not be determined (undeterminable areas) and the locations of these undeterminable areas. The generation of information indicating the locations of undeterminable areas will be described later.

[0043] Furthermore, if the image determination unit 223 cannot extract an available space, or if an available space is extracted but it is determined that parking is not possible after comparing it with the stored vehicle information, it determines that there is no available space for parking. The image determination unit 223 outputs this determination result to the transmission unit 225.

[0044] Returning to Figure 3, the generation unit 224 acquires the judgment result from the image judgment unit 223, generates a display image, and outputs it to the transmission unit 225. For example, if the judgment result indicates that there is an available space where parking is possible, the generation unit 224 generates map information that reflects the location of that space on the map. The generation unit 224 may also acquire a real-time image or video including that space and output it to the transmission unit 225 as a display image. Furthermore, the generation unit 224 may insert arrows, borders, etc., indicating the available space where parking is possible into the selected real-time image.

[0045] Furthermore, the generation unit 224 may generate a display image using three-dimensional information of the empty space as shown in Figure 4. Also, if vehicle information is not held, the generation unit 224 may add the measurement results of the length of each side to the three-dimensional information. In addition, the generation unit 224 may create a display image that includes vehicle type information suitable for the size of the empty space input from the image determination unit 223.

[0046] For example, as shown in Figure 5B, the generation unit 224 may rearrange the priority of each empty space input from the image determination unit 223 from highest to lowest, and then generate a display image that is easy for drivers to see. In the example in Figure 5B, the generation unit 2242 generates a display image showing empty space No. 1, empty space No. 2, and empty space No. 3 in order of highest priority.

[0047] Here, we will explain an example of a case where information indicating the location of an undeterminable region is input from the image determination unit 223 to the generation unit 24. Figure 6 is a diagram showing an example of a display image when there is an undeterminable region according to Embodiment 1. When the image determination unit 223 inputs information that there is an undeterminable region and the location information of the undeterminable region, the generation unit 224 identifies the location on the map based on the location information of the undeterminable region. As shown in Figure 6, the generation unit 224 inserts a mark or the like that indicates the identified location on the map into the map information and generates a display image that makes the undeterminable region easily identifiable. For example, the generation unit 224 makes the undeterminable region easily identifiable on the map by filling it in black or outlining it.

[0048] Furthermore, the generation unit 224 can also transmit an image to the terminal device as a display image, after applying privacy processing to a predetermined protected object captured in the real-time image.

[0049] Here, we will explain an example of a case where a real-time image containing private information such as license plates and people's faces is input from the image determination unit 223. Figure 7 is a diagram showing an example of the processing content of privacy processing according to Embodiment 1. As shown in Figure 7, when the generation unit 224 detects private information such as people's faces and license plates from the real-time image input from the image determination unit 223, it performs blurring, mosaic processing, etc. on the corresponding parts to generate a display image in which the private information cannot be viewed.

[0050] Furthermore, the generation unit 224 can also generate a display image based on another real-time image and transmit it from the transmission unit 225 if, after a display image has been transmitted from the transmission unit 225 and displayed on the terminal device 10, the terminal device 10 has requested an image switching operation.

[0051] Here, we will describe an example of a case where information indicating that there is a parking space available is transmitted to the terminal device 10. Figure 8 is a diagram showing an example of the processing content of a swipe operation according to Embodiment 1. As shown in Figure 8A, the terminal device 10 displays the parking space determination result and map information. As an example, the driver or the like checks the result and performs a swipe operation (an operation of sliding a finger placed on the screen in any direction) on the display screen of the terminal device 10 to find out the details of the parking space available. The terminal device 10 then transmits to the determination device 20 that a swipe operation has been performed.

[0052] Then, the generation unit 224 of the determination device 20 transmits an image that includes a parking space and was captured by another camera to the terminal device 10. As a result, as shown in Figure 8B, an image of the parking space taken from a different angle and distance than the original image is displayed on the terminal device 10. Similarly, in Figure 8C, an image of the parking space taken by a different camera than the image in Figure 8B is displayed through the same process.

[0053] Furthermore, the generation unit 224 may generate a display image by combining map information including location information of available parking spaces and real-time images including multiple selected available parking spaces, in order to respond to swipe operations by drivers, etc. For example, the generation unit 224 may combine map information as shown in Figure 8A with an image showing the entire specified location including available parking spaces as shown in Figure 8B as the first image, and an image centered on the available parking spaces as shown in Figure 8C as the second image. The generation unit 224 may also combine real-time video including available parking spaces.

[0054] In this case, the terminal device 10 receives the image generated by the generation unit 224 and switches between displaying the map information shown in Figure 8A, the first image shown in Figure 8B, and the second image shown in Figure 8C, in accordance with the swipe operation of the driver or the like.

[0055] Furthermore, the generation unit 224 may generate a generated image by abstracting the real-time image input from the image determination unit 223. For example, if the available parking space is difficult to see in the real-time image, the generation unit 224 may create an easy-to-view display image by highlighting the available space and other elements based on the real-time image including the available parking space, and abstracting the surrounding area.

[0056] Returning to Figure 3, the transmission unit 225 transmits the judgment result input from the image determination unit 223 and the display image input from the generation unit 224 to the terminal device 10. For example, if the image determination unit 223 inputs a judgment result of "no available parking space", the transmission unit 225 transmits that result to the terminal device 10. In addition, if the information input from the image determination unit 223 includes an area that cannot be determined, the transmission unit 225 may add map information of the area that cannot be determined, generated by the generation unit 224, to the judgment result of "no available parking space".

[0057] Furthermore, the transmission unit 225 transmits the judgment result "Available parking space" input from the generation unit 224 and the display image generated by the generation unit 224 to the terminal device 10. In addition, if the information input from the image judgment unit 223 includes an area that cannot be determined, as shown in Figure 6, map information including the area that cannot be determined generated by the generation unit 224 may be added to the display of the judgment result "Space available".

[0058] (Specific example) Next, let's explain a specific example of a father picking up his daughter at the rotary in front of the station. Prior to arriving at the rotary in front of the station, the father specifies the desired parking location, "the rotary in front of the station," to the terminal device 10. The reception unit 221 of the determination device 20 then receives the specified location, "the rotary in front of the station," via the communication unit 11 of the terminal device 10 and the communication unit 21 of the determination device 20. At the same time, the reception unit 221 of the determination device 20 also receives vehicle information that has been pre-registered in the terminal device 10.

[0059] Next, the image acquisition unit 222 acquires real-time images taken from fixed cameras in the "station square rotary" or from on-board cameras mounted on vehicles in the station square rotary.

[0060] Next, the image determination unit 223 of the determination device 20 creates three-dimensional spatial information from the acquired real-time image and extracts available space. The image determination unit 223 also measures the size of the extracted available space and determines whether it matches the vehicle size included in the previously acquired vehicle information to determine whether parking is possible. If the image determination unit 223 determines that there is available space for parking, it outputs the real-time image to the generation unit 224 along with the determination result "Available space for parking".

[0061] Next, the generation unit 224 of the determination device 20 selects several images from the multiple real-time images containing available parking spaces input from the image determination unit 223. The generation unit 224 inserts borders indicating the available parking spaces into the selected real-time images and generates a display image. Furthermore, the generation unit 224 generates map information that identifies the locations of the available parking spaces. The generation unit 224 outputs the display image and map information along with the "space available" determination result to the transmission unit 225.

[0062] Next, the transmission unit 225 of the determination device 20 transmits the "space present" determination result, display image, and map information input from the generation unit 224 to the terminal device 10.

[0063] The father selects an available space from the results displayed on the terminal device 10. At that time, the terminal device 10 may start navigating toward the selected available space.

[0064] (Processing flow of the determination device 20) Next, an example of the processing procedure by the determination device 20 according to Embodiment 1 will be described using Figure 9. Figure 9 is a flowchart showing an example of the process for determining whether or not there is a parking space available according to Embodiment 1. Note that each step in the flowchart shown in Figure 9 can be executed in a different order, and additional or omitted processes may be included.

[0065] First, the determination device 20 receives the designation of the desired parking location and vehicle information from the terminal device 10 (S101). Next, the determination device 20 acquires real-time images from fixed cameras, on-board cameras, etc., at the designated location (S102). Subsequently, the determination device 20 generates three-dimensional spatial information based on the real-time images (S103).

[0066] The determination device 20 then extracts available spaces from the three-dimensional spatial information, refers to the vehicle information, and determines whether there is a available space for parking (S104). If it determines that there is a available space for parking (S104; YES), the determination device 20 determines whether there are multiple available spaces for parking (S105). On the other hand, if there is no available space for parking (S104; NO), the determination device 20 proceeds to S110 and continues processing.

[0067] If the determination device 20 determines that there are multiple available parking spaces (S105; YES), it assigns a priority to each available space based on a predetermined priority order (S106). On the other hand, if the determination device 20 determines that there are no multiple available parking spaces (S105; NO), it continues the process described in S107.

[0068] The determination device 20 detects the object to be protected from the real-time image (S107). If the object to be protected is detected (S107; YES), the determination device 20 performs privacy processing on the real-time image (S108). On the other hand, if the object to be protected is not detected from the real-time image (S107; NO), the determination device 20 continues the process from S109.

[0069] The determination device 20 generates a display image from a real-time image that includes available parking spaces (S109). Next, the determination device 20 determines whether or not there are areas that cannot be determined (S110). If there are areas that cannot be determined (S110; YES), the determination device 20 transmits the determination result, the display image, and map information including the areas that cannot be determined to the terminal device 10 (S111), and ends the process.

[0070] On the other hand, if there are no areas that cannot be determined (S110; NO), the determination device 20 transmits the determination result and display image to the terminal device 10 (S112) and terminates the process.

[0071] (effect) The above-described embodiment 1 provides the following effects. The determination device 20 can obtain information about available parking spaces before arriving at a location when a temporary stop is desired in an area where temporary stopping is permitted, such as for picking someone up. Therefore, if there is an available parking space in the desired area, the driver can go directly to the available space, improving the efficiency of pick-ups and drop-offs. Furthermore, because the driver can know about available parking spaces in advance, they can stop quickly even in places they are visiting for the first time. This also contributes to reducing congestion in places such as station plazas.

[0072] Furthermore, if there are no available parking spaces in the desired area, drivers can search for available spaces in nearby areas where parking is permitted and obtain information on available spaces. Alternatively, they can park their vehicle in a paid parking lot instead of going to an area without available spaces in order to search for a parking space. This improves user convenience.

[0073] [Embodiment 2] By the way, Embodiment 1 described a process in which the driver specifies a place where they wish to stop, determines whether there is an available parking space at the specified location, and displays the determination result to the driver. However, the availability of parking spaces changes in real time, and it is possible that there may be no available parking space when the driver arrives at the location.

[0074] Therefore, in Embodiment 2, the process by which the determination device 20 continuously monitors the status of the available parking space until the driver arrives at the available parking space will be described.

[0075] The overall configuration and functional configuration of the information processing system according to Embodiment 2 are the same as those of the terminal device 10 and determination device 20 according to Embodiment 1, so a detailed explanation will be omitted. Below, we will explain the processing content of the determination device 20, which differs from that of Embodiment 1.

[0076] (Processing details of the determination device 20) The processing details of the determination device 20 according to this second embodiment will now be described. The determination device 20 according to this second embodiment receives a designation of a parking space from the results of the determination of available parking spaces, and monitors a real-time image of that parking space until the vehicle arrives at that space. For example, the determination device 20 may determine whether there has been a change in the parking space, and if it determines that there has been a change, it may determine whether it is possible to park in the space after the change, and transmit the fact that there has been a change and the determination result to the terminal device 10. Alternatively, the determination device 20 may transmit a message indicating that there has been a change in the parking space as soon as it detects a change.

[0077] Here, we will explain the processing content of the determination device 20 with specific examples. Figure 10 is a diagram showing an example of the processing content of the continuous monitoring process according to Embodiment 2.

[0078] As a prerequisite, the driver or other user selects a parking space from the results of the determination of available parking spaces via the input section of the terminal device 10 and begins moving toward the parking space. The terminal device 10 transmits the selected parking space to the determination device 20. The receiving section 221 of the determination device 20 then accepts the designation of the parking space.

[0079] As shown in Figure 10(1), the image acquisition unit 222 continues to acquire real-time images of the designated available parking space from a fixed camera, an on-board camera, etc., that can capture the designated space. The image determination unit 223 determines the status of the designated space based on the acquired real-time images. The process by the image acquisition unit 222 and the image determination unit 223 to determine the status of the designated space continues until the vehicle arrives at the designated space.

[0080] When a real-time image like the one shown in Figure 10 (2) is input from the image acquisition unit 222, the image determination unit 223 determines that the status of the available space has changed. At the stage when the image determination unit 223 determines that a change has been determined in the designated available space, it may transmit a message to the determination device 20 via the transmission unit 225 indicating that a change has been determined.

[0081] As shown in Figure 10 (3), the image determination unit 223 may determine whether or not there is still space available for parking after a change has occurred in the designated empty space, using the same process as in Embodiment 1. For example, if the designated empty space was large or the newly parked vehicle was small, it may be possible for a vehicle to park in the empty space after the change has been determined. On the other hand, if the designated parking space was originally narrow or the newly parked vehicle was large, it may be impossible for a vehicle to park in the empty space after the change has been determined.

[0082] In the example of (4) in Figure 10, the image determination unit 223 determines that the designated empty space after detecting the change is unsuitable for vehicle parking, and outputs the determination result to the generation unit 224 along with the real-time image.

[0083] As shown in Figure 10 (4), the generation unit 224 generates map information and display images based on the judgment result input from the image judgment unit 223 and the real-time image. In this case, the generation unit 224 creates a display image indicating that the designated empty space has changed and that there are no more empty spaces available for parking, and transmits it to the terminal device 10 via the transmission unit 225 for display.

[0084] (Processing flow of the determination device 20) Next, an example of the processing procedure by the determination device 20 according to Embodiment 2 will be described using Figure 11. Figure 11 is a flowchart showing an example of the flow of processing content for the continuous monitoring process according to Embodiment 2. Note that each step in the flowchart shown in Figure 11 can be executed in a different order, and additional or omitted processes may be included.

[0085] First, the determination device 20 receives a designation from the terminal device 10 of a suitable parking space where the user wishes to park based on the determination result (S201). Next, the determination device 20 acquires real-time images taken from a fixed camera, an in-vehicle camera, etc., that can capture the designated parking space (S202). Subsequently, the determination device 20 determines whether or not there has been a change in the status of the designated parking space based on the real-time images (S203).

[0086] If there is no change in the designated empty space (S203; NO), the determination device 20 proceeds to S206 and continues processing. On the other hand, if there is a change in the designated empty space (S203; YES), the determination device 20 determines whether the designated empty space after the change is available for parking (S204). The determination device 20 then transmits to the terminal device 10 the fact that there has been a change in the designated empty space and the determination result (S205). Next, the determination device 20 confirms whether the vehicle has arrived at the designated empty space (S206).

[0087] If the vehicle has not arrived at the designated empty space (S206; NO), the determination device 20 repeats the process from S202. On the other hand, if the vehicle has arrived at the designated empty space (S206; YES), the determination device 20 terminates the process.

[0088] (effect) The embodiment 2 described above provides the following effects. The determination device 20 can inform the driver that the situation has changed while the vehicle is moving towards a designated available parking space, and whether the space is still available for parking after the change in circumstances. This allows the driver to take measures such as specifying an alternative location before arriving at the designated parking space, thereby improving convenience.

[0089] (modified version) By the way, as a variation of Embodiment 1 and Embodiment 2, the functions of the terminal device 10 can be introduced as an application for a mobile phone or the like, making it possible, for example, for the person going to pick someone up and the person waiting to be picked up to share information.

[0090] Figure 12 shows an example of the process for determining whether there is a parking space available, according to a modified version of Embodiment 2. As shown in Figure 12, the terminal devices 10 in Embodiment 2, one for the party going to pick up the other and one for the party waiting to pick up the other, are connected via a network.

[0091] This allows both the person going to pick up the other and the person waiting to pick up to specify the desired stopping location and perform other operations. Furthermore, both the person going to pick up and the person waiting to pick up can check information such as the determination result of whether there is an available space and the current location of the vehicle going to pick up the other. Below, we will describe the processing content of the determination device 20, which differs from that of Embodiments 1 and 2.

[0092] As shown in Figure 12(1), the receiving unit 221 of the determination device 20 receives, for example, a specification of the desired stopping location from the terminal device 10 on the waiting side. Then, as shown in Figure 12(2), the receiving unit 221 notifies, for example, the terminal device 10 on the going side that the desired stopping location has been specified. After that, the receiving unit 221 outputs the specified location to the image acquisition unit 222, similar to the processing in Embodiment 1.

[0093] If one terminal device 10 selects an available parking space from the selection results, the receiving unit 221 of the determination device 20 transmits to the other terminal device 10 that an available space has been selected, indicates the location of the available space, and requests confirmation. Subsequently, the receiving unit 221 outputs the selected location to the image acquisition unit 222, similar to the process in Embodiment 2.

[0094] The transmission unit 225 transmits the processing results of the image determination unit 223 or the generation unit 224 and the display image to both terminal devices 10.

[0095] (Specific example) For example, when a father goes to pick up his daughter, the daughter specifies the desired parking location using her own terminal device 10. The determination device 20 then notifies the father's terminal device 10 that a location has been specified and provides the location. The determination device 20 then sends the available parking spaces to both the father's and daughter's terminal devices 10 as a result of the determination. The daughter then specifies an available parking space. Subsequently, the determination device 20 notifies the father's terminal device 10 that an available parking space has been specified by the daughter and shows the location of the specified space. The father inputs into his terminal device 10 that he has confirmed the information provided. The determination device 20 then notifies the daughter's terminal device 10 that the father has confirmed the information. After that, the terminal device 10 may start navigation toward the available parking space.

[0096] (effect) The determination device 20 transmits information about available parking spaces to the terminal devices 10 of both the vehicle going to pick up the vehicle and the vehicle waiting, thereby sharing the information. This allows the vehicle to be found more quickly, even at large stations where the parking location of the picking vehicle may be difficult to determine, because the waiting vehicle already knows its location. This leads to shorter stopping times and contributes to alleviating congestion.

[0097] Furthermore, by sharing information, it becomes possible to dynamically change the destination if a passenger has to get off at a different station than their intended destination due to train delays or if their arrival at their destination is significantly delayed due to traffic congestion. This enables situation-appropriate pick-up and drop-off, greatly improving convenience. In addition, even if the destination is an unfamiliar location, it becomes possible to identify a suitable parking space, resulting in shorter stopping times, more efficient travel, and improved usability.

[0098] [Embodiment 3] By the way, the information processing systems described in Embodiments 1 and 2 above describe an example of determining and notifying whether there is an available parking space at a specified location, but the system is not limited to this. For example, the disclosed information processing system can detect a vehicle that is currently parked at the desired parking location and is expected to depart soon, in the event that no available parking space is found at the desired parking location. Therefore, Embodiment 3 describes an information processing system for detecting a vehicle that is expected to depart soon.

[0099] Figure 13 shows an example of the overall flow of the detection process for the most recently departing vehicle in Embodiment 3. As shown in Figure 13, the information processing system comprises a terminal device 10 mounted on the vehicle and a determination device 20, which is an information processing device. The terminal device 10 and the determination device 20 communicate via a wireless communication network, which is Network NE. The learning device 30, which will be described later, communicates with the determination device 20 via Network NE and performs the generation of a trained model used by the determination device 20.

[0100] The terminal device 10 is implemented by a computer or the like mounted in the vehicle, similar to Embodiment 1. For example, the terminal device 10 may be a dedicated navigation device built into or mounted in the vehicle. For example, the terminal device 10 may be a single device having both a navigation function and a display function.

[0101] The determination device 20 detects the most recently departing vehicle based on real-time images captured by a camera or in-vehicle camera installed at a designated location specified by the terminal device 10, and transmits the detection result to the terminal device 10. For example, if no available parking space is extracted using the method of Embodiment 1, the determination device 20 uses a machine learning model generated by a learning device 30 (not shown) to detect the most recently departing vehicle.

[0102] In such an information processing system, for example, terminal device 10 notifies determination device 20 when it approaches a designated location where no available parking space was found. Determination device 20 acquires real-time images taken by fixed cameras, on-board cameras, etc., at the designated location, detects the most recently departing vehicle based on the acquired real-time images, and transmits the detection result to terminal device 10.

[0103] Here, we will explain the processing of the determination device 20 using the example of a situation where a driver picks up someone at a station. As shown in Figure 13, for example, when picking someone up at a station, the driver specifies the desired stopping location on the terminal device 10 prior to arriving at the station, but it is determined that there is no available space to stop at the desired location. In this case, the driver proceeds to the station without a available space to stop (S1). The determination device 20 then receives notification via the network that the terminal device 10 is approaching the desired stopping location, detects the nearest departing vehicle, and transmits the detection result to the terminal device 10 (S2). The driver proceeds to the location where the nearest departing vehicle was detected, and stops after the nearest departing vehicle has departed (S3). After that, the driver picks up the person in the vehicle (S4).

[0104] (Processing details of the determination device 20) The processing details of the determination device 20 according to Embodiment 3 will now be described. The determination device 20 according to Embodiment 3 receives notification that a vehicle is approaching a designated location, acquires a real-time image of the designated location, detects the most recently departing vehicle based on the real-time image, and transmits the detection result to the terminal device 10.

[0105] Here, we will explain the processing content of the determination device 20 with specific examples. Figure 14 is a diagram illustrating the processing content of the determination device 20 according to Embodiment 3.

[0106] (1) The terminal device 10 mounted on the vehicle notifies the determination device 20 that it has approached within a predetermined distance of the designated location. Alternatively, the terminal device 10 may sequentially transmit current location information to the determination device 20, and the determination device 20 may detect the vehicle's approach to the designated location. (2) When the determination device 20 detects the vehicle's approach to the designated location, it acquires real-time images captured by fixed cameras, on-board cameras, etc., located at the designated location.

[0107] (3) The determination device 20 detects the most recently departing vehicle based on the behavior of the occupants of the stationary vehicle included in the acquired real-time image. (4) The determination device 20 transmits the detection result to the terminal device 10. When the most recently departing vehicle is detected, the determination device 20 may transmit information showing the location of the most recently departing vehicle on a map along with the detection result, or it may transmit a real-time image or video that includes the most recently departing vehicle.

[0108] (Functional configuration of information processing systems) Next, an example of the functional configuration of the information processing system according to Embodiment 3 will be described. Figure 15 is a block diagram showing an example configuration of the terminal device 10, determination device 20, and learning device 30 included in the information processing system according to Embodiment 3.

[0109] (Configuration of terminal device 10) Since the terminal device 10 has the same configuration as in Embodiment 1, we will describe the control unit 14 which performs different processing.

[0110] For example, if the control unit 14 approaches a designated location stored in the storage unit 15 while no available parking space has been found, it instructs the communication unit 11 to notify the determination device 20 that the vehicle has approached the designated location.

[0111] (Configuration of the learning device 30) As shown in Figure 15, the learning device 30 consists of a communication unit 31, a control unit 32, a storage unit 33, etc. The learning device 30 acquires multiple training data, including video data containing people and a parked vehicle, and a correct label indicating whether or not the parked vehicle will start moving within a predetermined time. The learning device 30 then uses the acquired training data to learn an estimation model that estimates from the video data whether or not the parked vehicle in the video data will start moving within a predetermined time, and generates a trained estimation model.

[0112] The communication unit 31, under the control of the control unit 32, transmits and receives information with the determination device 20 (communication unit 21) via the network NE.

[0113] The storage unit 33 stores various programs executed by the control unit 32 (information processing programs according to this embodiment), as well as data necessary for the control unit 32 to perform processing. As shown in Figure 15, this storage unit 33 includes a training DB 331, a trained model DB 332, and the like.

[0114] The training DB331 stores training data, etc., in which explanatory variables are associated with ground truth data (target variable). For example, the training DB331 stores multiple training data, etc., including image data or video data (sometimes simply referred to as video data) and ground truth labels. Alternatively, the training DB331 may also store training data, including image or video data and ground truth labels, generated by the training data processing unit 322, which will be described later. The trained model DB332 stores estimation models, scene detection models, etc., that have been trained using the training data in the learning unit 323. Neural networks, etc., can be used for each of these models, such as estimation models and scene detection models.

[0115] The control unit 32 is implemented by a controller such as a CPU or MPU executing various programs (including the information provision program according to this embodiment) stored in the memory unit 33, and controls the operation of the entire learning device 30. Note that the control unit 32 is not limited to a CPU or MPU, but may also be composed of integrated circuits such as ASICs or FPGAs.

[0116] As shown in Figure 15, this control unit 32 includes a training data acquisition unit 321, a training data processing unit 322, a learning unit 323, and the like.

[0117] The training data acquisition unit 321 acquires training data from the training DB 331. For example, the training data includes video data that includes a person being filmed and a parked vehicle, and a correct label indicating whether or not the parked vehicle will start moving within a predetermined time. The correct label may also include a probability of departure, which is the likelihood that the parked vehicle will start moving within a predetermined time. The training data acquisition unit 321 outputs the training data to the training data processing unit 322 or the learning unit 323.

[0118] For example, the training data acquisition unit 321 acquires training data that combines scenes showing the movements of occupants, etc., with correct labels, as shown in the training DB 331 in Figure 16. Scenes showing the movements of occupants, etc., include, for example, scenes where a person gets out from a seat other than the driver's seat, or scenes where a seatbelt is fastened. The correct labels may indicate whether the vehicle starts or does not start within a predetermined time, or they may include the probability of a stationary vehicle starting within a predetermined time. For example, the correct labels may indicate whether the vehicle starts within 3 minutes, or whether it does not start within 3 minutes.

[0119] The training data processing unit 322 processes the training data to improve learning efficiency. The training data processing unit 322 generates processed training data by applying abstraction or other methods to the training data input from the training data acquisition unit 321 and outputs it to the learning unit 323. Furthermore, the training data processing unit 322 may output the processed training data to the training DB 331.

[0120] For example, the training data processing unit 322 generates abstracted video data by abstracting the input video data (A in Figure 17), as shown in Figure 17B. The training data processing unit 322 may also recognize people and vehicles from the video data and generate abstracted video data by abstracting the recognized people and vehicles. The training data processing unit 322 generates training data by combining the generated abstracted video data with the correct labels.

[0121] The training data processing unit 322 abstracts a video of a seatbelt being fastened, as shown in Figure 17A. Then, as shown in Figure 17B, the training data processing unit 322 generates training data that includes the abstracted video data and the correct labels.

[0122] Furthermore, for example, the training data processing unit 322 generates symbolized video data by concatenating the feature points of the input video data, as shown in Figure 18B.

[0123] The training data processing unit 322 recognizes people from video data and generates person symbols by concatenating the feature points of the recognized people. The training data processing unit 322 recognizes vehicles from video data and generates vehicle symbols by concatenating the feature points of the recognized vehicles. The training data processing unit 322 generates training data that includes symbolized video data containing person symbols and vehicle symbols, and ground truth labels.

[0124] The training data processing unit 322 uses, for example, general image recognition technology to recognize a person and a vehicle from a video of a person getting out of the driver's seat, as shown in Figure 18A. The training data processing unit 322 then generates symbolized video data by concatenating the feature points of the recognized person and vehicle, as shown in Figure 18B. The training data processing unit 322 generates training data that includes the symbolized video data and the correct labels.

[0125] Returning to Figure 15, the learning unit 323 uses the training data or the training data processing unit 322 to learn an estimation model that estimates whether or not a stationary vehicle included in the input video data will start moving, and generates a trained estimation model. The learning unit 323 stores the generated trained estimation model in the trained model DB 332.

[0126] Figure 16 shows an example of the learning process for an estimation model using video data. For example, as shown in Figure 16, the learning unit 323 generates an estimation model by performing supervised learning processing using the training data acquired by the training data acquisition unit 321 from the training DB 331 as training data.

[0127] For example, the learning unit 323 repeatedly performs supervised learning using multiple training data sets as training data, each containing either video data and the correct label "depart" or video data and the correct label "do not depart". To give one example, the learning unit 323 inputs video data and the correct label "depart" into the estimation model and obtains estimation results (probability of departure (%), probability of not depart (%)). Then, the learning unit 323 updates the parameters of the estimation model so that the estimation results match the correct label "depart".

[0128] Figure 17 shows an example of the learning process for an estimation model using abstracted video data. For example, as shown in Figure 17, the learning unit 323 generates an estimation model by performing supervised learning processing using the training data processing unit 322, which uses the abstracted video data and training data containing the correct labels generated by the training data processing unit 322 as training data.

[0129] For example, the learning unit 323 iteratively performs supervised learning using multiple training data sets, each containing either abstracted video data and the correct label "depart" or abstracted video data and the correct label "do not depart," as training data. To give one example, the learning unit 323 inputs the abstracted video data and the correct label "depart" into the estimation model and obtains estimation results (probability of departure (%), probability of not depart (%)). Then, the learning unit 323 updates the parameters of the estimation model so that the estimation results match the correct label "depart."

[0130] Figure 18 shows an example of the learning process for an estimation model using symbolized video data. For example, as shown in Figure 18, the learning unit 323 generates an estimation model by performing supervised learning processing using the training data processing unit 322, which uses the symbolized video data and training data containing the correct labels generated by the training data processing unit 322 as training data.

[0131] For example, the learning unit 323 iteratively performs supervised learning using multiple training data sets, each containing either symbolized video data and the correct label "depart" or symbolized video data and the correct label "do not depart," as training data. To give one example, the learning unit 323 inputs the symbolized video data and the correct label "do not depart" into the estimation model and obtains estimation results (probability of departure (%), probability of not depart (%)). Then, the learning unit 323 updates the parameters of the estimation model so that the estimation results match the correct label "do not depart."

[0132] Figure 19 shows an example of the learning process for an estimation model that includes the launch probability. For example, as shown in Figure 19, the learning unit 323 generates an estimation model by performing supervised learning processing using the training data acquired by the training data acquisition unit 321 from the training DB 331 as training data.

[0133] As shown in Figure 19, the ground truth labels in the training data include the probability of starting within a given time. By including the probability of starting in the ground truth labels that indicate a vehicle will start within a given time, the ground truth labels for "starting" can be further refined.

[0134] For example, the learning unit 323 iteratively performs supervised learning using multiple training data sets, including video data and the correct label "Depart (including the probability of departure)," and video data and the correct label "Do not depart," as training data. To give one example, the learning unit 323 inputs video data and the correct label "Depart (80% probability of departure)" into the estimation model and obtains the estimation result. Then, the learning unit 323 updates the parameters of the estimation model so that the estimation result matches the correct label "Depart (80% probability of departure)."

[0135] Thus, the learning unit 323 can generate not only a binary estimation model that performs a binary estimation of whether to launch or not, depending on the user and the type of service provided, but also a multi-valued estimation model that estimates the probability of launching.

[0136] Furthermore, the learning unit 323 may generate a scene detection model that can extract various scenes based on training data acquired from the training data acquisition unit 321, which includes scenes that may occur while the vehicle is stopped, such as scenes where a person gets out of the vehicle from a location other than the driver's seat, scenes where a person gets in from a location other than the driver's seat, or scenes where an occupant fastens a seatbelt.

[0137] (Configuration of the determination device 20) Only the configuration or processing of the determination device 20 that differs from the configuration of Embodiment 1 will be described. As shown in Figure 15, the determination device 20 has an estimation processing unit 400 in the control unit 22 and a learned model DB 232 in the storage unit 23.

[0138] The communication unit 21, under the control of the control unit 22, transmits and receives information between the terminal device 10 and the learning device 30 via the network.

[0139] The image / video DB231 in the memory unit 23 stores the detection results from the detection unit 403 or the estimation results from the estimation unit 402, and also stores display images and the like generated by the image processing unit 401.

[0140] The pre-trained model DB232 in the memory unit 23 acquires and stores pre-trained models generated by the learning device 30, for example, via the communication unit 31 of the learning device 30 which can communicate with the communication unit 21. The acquisition of pre-trained models is not limited to this example and may be performed by other methods, such as acquiring them via any recording medium. For example, the pre-trained model DB232 stores pre-trained estimation models as explained in Figures 16 to 19, and scene detection models that can extract the various scenes described above.

[0141] The receiving unit of this invention corresponds to the receiving unit 221, and the acquisition unit corresponds to the image acquisition unit 222. The detection unit of this invention corresponds to the estimation unit 402 and the detection unit 403. Furthermore, the transmission unit of this invention corresponds to the display image generation unit 404 and the transmission unit 225.

[0142] The reception unit 221 receives notification from the terminal device 10 that a vehicle is approaching the designated parking location. Alternatively, the reception unit 221 may sequentially acquire current location information from the terminal device 10 and determine whether the vehicle has approached the designated parking location. In Embodiment 3, when no available parking space is found at the location specified by the driver, the reception unit 221 receives notification from the terminal device 10 via the communication unit 11 and the control unit 21 that the vehicle has approached the location. The reception unit 221 outputs information of the designated location to the image acquisition unit 222.

[0143] The image acquisition unit 222 acquires real-time images from fixed cameras, in-vehicle cameras, etc., located at a designated location, as received by the reception unit 221, via the communication unit 21. The image acquisition unit 222 outputs the acquired real-time images to the estimation processing unit 400.

[0144] The image processing unit 401 processes the real-time image input from the image acquisition unit 222 and outputs it to the estimation unit 402. The image processing unit 401 performs abstraction processing on the input real-time image to generate an abstracted real-time image. The image processing unit 401 may also recognize people and vehicles from the real-time image and generate an abstracted real-time image in which people and vehicles are abstracted.

[0145] Furthermore, for example, the image processing unit 401 performs the symbolization process described in Figure 18 on the input real-time image to generate a symbolized real-time image. Alternatively, the image processing unit 401 may extract people and vehicles from the real-time image and generate a symbolized real-time image by concatenating the feature points of the people and vehicles.

[0146] The estimation unit 402 inputs real-time images received from the image acquisition unit 222 into a trained estimation model to estimate whether a stationary vehicle will start moving within a predetermined time. Figure 20 shows an example of the process for detecting the most recently moving vehicle using the trained estimation model. For example, the estimation unit 402 inputs real-time images, including a scene where three cars are parked in a row as shown in Figure 20A, into the trained estimation model.

[0147] The trained estimation model then determines from the input real-time image whether a stationary vehicle is the most recently departing vehicle. For example, as shown in Figure 20B, the trained model outputs "departing" or "not departing" for each extracted stationary vehicle.

[0148] For example, in the example shown in Figure 20B, the estimation unit 402 outputs "does not start," "starts," and "does not start" as output results to the detection unit 403 for each of the three parallel-parked vehicles. The output result "starts" is calculated by the estimation unit 402 based on the scene in which a person gets out of a vehicle other than the driver's seat.

[0149] Figure 21 shows an example of the process for detecting the most recently departing vehicle using a trained estimation model that includes the probability of departure. Figure 21 shows the case where the output result of the estimation unit 402 includes the probability of departure. For example, the estimation unit 402 inputs a real-time image including a scene in which three cars are parked in a row, as shown in Figure 21A, to the trained estimation model. The trained estimation model then estimates the probability that a stationary vehicle is the most recently departing vehicle from the input real-time image.

[0150] For example, in the example shown in Figure 21B, the trained model outputs the probability that each extracted vehicle will start moving. For example, as shown in Figure 21B, the estimation unit 402 outputs "Start moving: 80%", "Start moving: 20%", and "Start moving: 5%" as output results to the detection unit 403 for each of the three parallel-parked vehicles. The output result "Start moving: 80%" is calculated by the estimation unit 402 from the scene in which the occupants of the extracted parked vehicles fasten their seat belts.

[0151] The estimation unit 402 may also input an abstracted real-time image or a symbolized real-time image generated by the processing of the image processing unit 401 into the trained estimation model.

[0152] Returning to Figure 15, the detection unit 403 detects the most recently departing vehicle based on the real-time image acquired by the image acquisition unit 222 or the output result of the estimation unit 402, and outputs the detection result to the display image generation unit 404 or the transmission unit 225.

[0153] For example, the detection unit 403 detects vehicles that the estimation unit 402 has determined to "start moving" or vehicles whose probability of starting is above a threshold (e.g., 80% or higher) as the most recently started vehicle, based on the output of the estimation unit 402. The detection unit 403 does not detect vehicles that the estimation unit 402 has determined to "not start moving" or vehicles whose probability of starting is below a threshold (e.g., less than 80%) as the most recently started vehicle.

[0154] For example, the detection unit 403 detects only the stationary vehicles that are marked "will start" from the output results of "will not start" and "will not start" in Figure 20B as the most recently starting vehicles. In other words, as shown in Figure 20C, it detects vehicles that are marked "will start" and from which a person gets out of a seat other than the driver's seat as the most recently starting vehicles.

[0155] Furthermore, for example, if the output of the pre-trained estimation model of the estimation unit 402 includes the probability of departure, the detection unit 403 will detect only the stationary vehicle with a "Depart: 80%" probability from the output of "Depart: 80%", "Depart: 20%", and "Depart: 5%" in Figure 21B as the vehicle that has just departed. In other words, as shown in Figure 21C, the detection unit 403 will detect the vehicle with a "Depart: 80%" probability and in which the occupants have been observed fastening their seatbelts as the vehicle that has just departed.

[0156] Furthermore, for example, if the detection unit 403 can obtain the duration of stopping of a stationary vehicle based on the acquired real-time image, it may apply weighting to the output result according to the duration of stopping. For example, the detection unit 403 calculates an output result weighted by "corrected starting probability = k × starting probability". For example, if the duration of stopping is 3 minutes or less, "k=1" is set, and if the duration of stopping is longer than 3 minutes, "k=1.4" is set.

[0157] For example, the detection unit 403 assumes that it has determined from the acquired real-time image that the duration of a stationary vehicle's stay is 4 minutes. If the output of the trained estimation model for the vehicle is "will start: 60%", the detection unit 403 sets "k=1.4" because the duration of the stop is longer than 3 minutes, and performs "corrected start probability = k × 60". Since the corrected start probability becomes "84%", it considers that the probability of starting is above the threshold (for example, 80% or more) and detects the vehicle as the vehicle that will start soon.

[0158] Furthermore, for example, the detection unit 403 can also detect the most recently departing vehicle if it detects a predetermined situation as a result of performing image analysis on the real-time image acquired by the image acquisition unit 222.

[0159] Figure 22 shows an example of the image analysis processing content according to Embodiment 3. As shown in Figure 22A, the detection unit 403 performs image analysis processing based on the acquired real-time image, and if it detects a scene in which a person gets out from a place other than the driver's seat, it detects the vehicle from which the person got out as the most recently departing vehicle.

[0160] As shown in Figure 22B, the detection unit 403 performs image analysis processing based on the acquired real-time images, and if it detects a scene where a person gets in from a position other than the driver's seat, it detects the vehicle with the person inside as the most recently departing vehicle.

[0161] As shown in Figure 22C, the detection unit 403 performs image analysis processing based on the acquired real-time image, and if it detects a scene in which an occupant is fastening a seat belt, it detects the vehicle in which the occupant is fastening a seat belt as the most recently departing vehicle.

[0162] Furthermore, the detection unit 403 may use a scene detection model to detect predetermined scenes from a video, and then detect the most recently departing vehicle from that scene. The scene detection model is included in the learning model of the present invention. For example, the detection unit 403 inputs the acquired real-time images into the scene detection model. The scene detection model then extracts scenes from the real-time images such as a person getting out from a seat other than the driver's seat, a person getting in from a seat other than the driver's seat, or an occupant fastening a seatbelt.

[0163] Based on the extracted scenes, the detection unit 403 detects vehicles in real-time images where actions such as a person getting out of the vehicle from a location other than the driver's seat or a person getting into the vehicle from a location other than the driver's seat are taking place, and identifies these as the most recently departing vehicles.

[0164] Alternatively, the detection unit 403 can also detect the most recent departing vehicle when it detects a scene in which a predetermined scene is expected to be present, using a scene detection model that detects a predetermined scene from the video.

[0165] Furthermore, if the detection unit 403 detects the most recently departing vehicle, it outputs the detection result, along with the real-time image obtained, to the display image generation unit 404. If the detection unit 403 does not detect the most recently departing vehicle, it outputs to the transmission unit 225.

[0166] Returning to Figure 15, the display image generation unit 404 generates a display image based on the detection results, output results, and real-time images input from the detection unit 403 or estimation unit 402. For example, the display image generation unit 404 selects an image from among multiple real-time images that clearly shows the most recently departing vehicle and outputs it to the transmission unit 225 as a display image.

[0167] For example, the display image generation unit 404 generates map information that reflects the location of the most recently departed vehicle detected on the map. Alternatively, the display image generation unit 404 may combine the selected real-time image with the generated map information and output it to the transmission unit 225 as a display image. Furthermore, the display image generation unit 404 may insert arrows, borders, etc., indicating the most recently departed vehicle detected in the real-time image, etc.

[0168] Furthermore, for example, if the detection result of the detection unit 403 includes the departure probability, the display image generation unit 404 may include the departure probability in the selected real-time image. The display image generation unit 404 may also insert the departure probability into an arrow, frame, etc., indicating the most recently detected departing vehicle in the real-time image, etc.

[0169] The transmission unit 225 transmits the detection result or output result input from the detection unit 403, the estimation unit 402, or the display image generation unit 404, as well as the detection result and display image input from the display image generation unit 404, to the terminal device 10. For example, if the detection unit 403 inputs a detection result of "no recently departing vehicle," the transmission unit 225 may transmit that result to the terminal device 10.

[0170] Furthermore, the transmission unit 225 transmits to the terminal device 10 the detection result of "vehicle departing recently" input from the display image generation unit 404, along with the display image generated by the display image generation unit 404.

[0171] (Specific example) Next, let's explain a specific example of a father picking up his daughter. Prior to arriving at the rotary in front of the station, the father specifies the "rotary in front of the station" where he wishes to park to the terminal device 10. If no available parking space is found in the rotary in front of the station, the terminal device 10 transmits to the determination device 20 that the vehicle is approaching the designated rotary in front of the station. The reception unit 221 of the determination device 20 receives the information about the approach to the designated location and outputs the designated location to the image acquisition unit 222. The image acquisition unit 222 then acquires real-time images taken from a fixed camera at the designated location, the "rotary in front of the station," or from an on-board camera mounted on a vehicle in the rotary in front of the station.

[0172] Next, the estimation unit 402 of the determination device 20 uses an estimation model to estimate which vehicles are predicted to depart within 3 minutes from the stationary vehicles. Then, the detection unit 403 receives the estimation result and detects the most recently departed vehicle. The detection unit 403 outputs the detection result and real-time image to the display image generation unit 404. The display image generation unit 404 generates map information and a display image based on the detection result and real-time image and outputs them to the transmission unit 225.

[0173] Then, the transmission unit 225 of the detection device 20 transmits the detection result and display image to the terminal device 10. Based on the detection result and display image displayed on the terminal device 10, the father can head towards the location where the most recently departing vehicle is stopped and stop after the most recently departing vehicle has departed.

[0174] (Processing flow of the learning device 30) Next, an example of the processing procedure by the learning device 30 according to Embodiment 3 will be described using Figure 23. Figure 23 is a flowchart showing an example of the learning process flow of the estimation model according to Embodiment 3. Note that each step in the flowchart shown in Figure 23 can be executed in a different order, and additional or omitted processes may be included.

[0175] First, the learning device 30 records multiple training data sets, including video data and correct labels (S201). If the learning device 30 does not process the video data included in the training data (S202; NO), it proceeds to S204 and continues processing. If the learning device 30 processes the video data included in the training data (S202; YES), it abstracts or symbolizes the video data (S203).

[0176] The learning device 30 iteratively learns the estimation model using the training data as the training data (S204). Then, the learning device 30 generates a trained estimation model (S205) and terminates the process.

[0177] (Processing flow of the determination device 20) Next, an example of the processing procedure by the determination device 20 according to Embodiment 3 will be described using Figure 24. Figure 24 is a flowchart showing an example of the flow of the detection process for the most recently departing vehicle according to Embodiment 3. Note that each step in the flowchart shown in Figure 24 can be executed in a different order, and additional or omitted processes may be included.

[0178] The determination device 20 receives from the terminal device 10 the designation of the location where the vehicle is to be parked and the notification that the vehicle is approaching the designated location (S301). Next, the determination device 20 acquires real-time images from fixed cameras, on-board cameras, etc. at the designated location (S302). Subsequently, if the determination device 20 processes the real-time images (S303; YES), it abstracts or symbolizes the real-time images (S304). If the determination device 20 does not process the real-time images (S303; NO), it proceeds to S305 and continues processing.

[0179] If the determination device 20 does not want to process the real-time image using the trained model (S305; NO), it performs image analysis processing on the real-time image (S306). If the determination device 20 wants to process the real-time image using the trained model (S305; YES), it inputs the image into the estimation model (S307; YES). If it does not want to input the image into the estimation model (S307; NO), it inputs the image into the scene detection model (S308).

[0180] If the determination device 20 detects the most recently departing vehicle (S309; ​​YES), it generates a display image based on the real-time image (S310). If the determination device 20 does not detect the most recently departing vehicle (S309; ​​NO), it proceeds to S311 and continues processing.

[0181] If a display image is available, the determination device 20 transmits the detection result, including the display image, to the terminal device 10 (S311), and terminates the process.

[0182] (effect) The above-described embodiment 3 provides the following effects: The learning device 30 generates an estimation model that can estimate the possibility of a stationary vehicle starting. This enables the determination device 20 to quickly and accurately detect a vehicle that is likely to start soon from the input real-time image, etc. Furthermore, by training the estimation model on the possibility of starting, the detection result of the most recently starting vehicle can include the probability of starting, improving convenience for drivers and the efficiency of stopping.

[0183] Furthermore, even if no available parking space is detected in the area where the driver wishes to stop, the detection device 20 can detect the most recently departing vehicle. As a result, the driver can quickly recognize when an available parking space has become available in their original desired parking area before heading to another area where parking is permitted, and can decide to proceed to their original desired parking area. This allows the driver to efficiently search for a parking space.

[0184] Furthermore, even while the vehicle is circling to allow drivers to stop, it can quickly find a parking space. This improves usability.

[0185] [Embodiment 4] Now, while embodiments of the present invention have been described, the present invention may be implemented in various other forms besides those described above.

[0186] (Numerical values, etc.) The numerical values ​​and thresholds (for example, arbitrary numerical values) used in the above embodiment are merely examples and can be changed as desired.

[0187] Furthermore, the detection process for the most recently departing vehicle described in Embodiment 3 may be performed at the stage when no available parking space is detected in Embodiment 1. If the image determination unit 223 of the determination device 20 is unable to extract an available space, the image determination unit 223 may output a real-time image to the estimation processing unit 400. The trained estimation model may also estimate whether each of the multiple vehicles included in a real-time image corresponds to the most recently departing vehicle from a single real-time image.

[0188] It is also possible to retain some of the functions of the terminal device 10 according to Embodiments 1, 2, and 3, and provide the other functions in the determination device 20. In this case, the terminal device 10 may be equipped only with a display function, an input function, and a communication function.

[0189] [Hardware configuration] Furthermore, the information processing devices (including the terminal device 10, the determination device 20, and the learning device 30) according to Embodiments 1, 2, 3, and 4 described above are implemented by a computer 1000 having a configuration such as that shown in Figure 25. The following explanation will use the information processing device as an example. Figure 25 is a hardware configuration diagram showing an example of a computer that implements the functions of the information processing devices of Embodiments 1, 2, 3, and 4. The computer 1000 has a CPU 1100, RAM 1200, ROM 1300, HDD 1400, communication interface (I / F) 1500, input / output interface (I / F) 1600, and media interface (I / F) 1700.

[0190] The CPU 1100 operates based on programs stored in the ROM 1300 or HDD 1400, controlling various components. The ROM 1300 stores boot programs executed by the CPU 1100 when the computer 1000 starts up, as well as programs that depend on the computer 1000's hardware.

[0191] The HDD1400 stores programs executed by the CPU1100, as well as data used by such programs. The communication interface1500 receives data from other devices via a predetermined communication network and sends it to the CPU1100, and transmits data generated by the CPU1100 to other devices via the predetermined communication network.

[0192] The CPU 1100 controls output devices such as displays and printers, and input devices such as keyboards and mice, via the input / output interface 1600. The CPU 1100 acquires data from input devices via the input / output interface 1600. The CPU 1100 also outputs the generated data to output devices via the input / output interface 1600.

[0193] The media interface 1700 reads a program or data stored in the recording medium 1800 and provides it to the CPU 1100 via the RAM 1200. The CPU 1100 loads the program from the recording medium 1800 onto the RAM 1200 via the media interface 1700 and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or PD (Phase Change Rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0194] For example, when computer 1000 functions as an information processing device according to Embodiments 1, 2, 3, and 4, the CPU 1100 of computer 1000 realizes the functions included in the control units 14, 22, 32, and 22 of each device by executing programs loaded onto the RAM 1200. The CPU 1100 of computer 1000 reads and executes these programs from the recording medium 1800, but as another example, these programs may be obtained from other devices via a predetermined communication network.

[0195] 〔others〕 Furthermore, among the processes described in each of the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, specific names, and information including various data and parameters shown in the above document and drawings can be changed at will unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown.

[0196] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0197] Furthermore, the above embodiments can be combined as appropriate, provided that the processing content is not contradictory.

[0198] Although some embodiments of the present invention have been described in detail above with reference to the drawings, these are illustrative examples, and the present invention can be implemented in various other forms with modifications and improvements based on the knowledge of those skilled in the art, starting with the embodiments described in the disclosure section of the invention.

[0199] Furthermore, the terms "section, module, unit" mentioned above can be replaced with "means" or "circuit," etc. For example, the receiving unit can be replaced with receiving means or receiving circuit. [Explanation of symbols]

[0200] 10 Terminal devices 11 Communications Department 12 Input section 13 Display section 14 Control Unit 15 Storage section 20 Judgment device 21 Communications Department 22 Control Unit 221 Reception Department 222 Image acquisition unit 223 Image determination unit 224 Generation part 225 Transmitter 23 Memory section 231 Image / Video Database 232 Pre-trained Model Database 30 Learning device 31 Communications Department 32 Control Unit 321 Training Data Acquisition Unit 322 Training Data Processing Department 323 Learning Department 33 Storage section 331 Training DB 332 Pre-trained Model Database 400 Estimation Processing Unit 401 Image Processing Department 402 Estimation section 403 Detection Unit 404 Display image generation unit NE Network

Claims

1. A reception desk that accepts location requests from terminal devices, An acquisition unit that acquires real-time images captured by one or more cameras that are capturing the aforementioned location, A detection unit that detects a vehicle that is parked at the location and is expected to depart soon, based on the real-time image, A transmission unit that transmits the detection result of the detection unit to the terminal device, An information processing device having

2. The aforementioned transmitting unit When the most recently departing vehicle is detected, an image based on at least one of the real-time images showing the most recently departing vehicle is transmitted to the terminal device as a display image. The information processing apparatus according to claim 1.

3. The aforementioned transmitting unit When the most recently departing vehicle is detected, map information indicating the location of the most recently departing vehicle is transmitted to the terminal device. The information processing apparatus according to claim 1.

4. The detection unit is The real-time images are analyzed, and if a scene is detected in any of the vehicles parked at the location in which a person is getting out from a seat other than the driver's seat, a person is getting in from a seat other than the driver's seat, or an occupant is fastening a seatbelt, the vehicle in which such a scene is detected is identified as the most recently departing vehicle. The information processing apparatus according to claim 1.

5. The detection unit is The system includes a learning model that, upon input of the aforementioned real-time images, is trained to extract from the real-time images scenes where a person gets out from a seat other than the driver's seat, a person gets in from a seat other than the driver's seat, or an occupant fastens a seatbelt. The information processing apparatus according to claim 4.

6. The detection unit is A trained model, which has been trained to estimate whether the input image contains the most recently departing vehicle based on the input image, is input with the real-time image and, based on the estimation result obtained, the most recently departing vehicle is detected from the real-time image. The information processing apparatus according to claim 1.

7. The detection unit is A trained model, which is trained to estimate the likelihood of a vehicle in an input image starting based on the input image, is input with the real-time image and the estimation results obtained are referenced. If there is a vehicle whose likelihood of starting is above a threshold, that vehicle is detected as the most recently started vehicle. The information processing apparatus according to claim 1.

8. The aforementioned transmitting unit The possibility of departure of the most recently departed vehicle is transmitted to the terminal device. The information processing apparatus according to claim 7.

9. An information processing method performed by an information processing device, A reception process that accepts location specifications from a terminal device, The acquisition process includes acquiring real-time images captured by one or more cameras that are filming the aforementioned location, A detection step to detect a vehicle that is parked at the location and is expected to depart soon, based on the real-time image, A transmission step which transmits the detection result of the above detection step to the terminal device, An information processing method characterized by including

10. The reception procedure for receiving location specifications from the terminal device, A procedure for acquiring real-time images captured by one or more cameras while filming the aforementioned location, A detection procedure for detecting a vehicle that is parked at the location and is expected to depart soon, based on the real-time image, A transmission procedure for transmitting the detection result of the above detection procedure to the terminal device, An information processing program characterized by causing a computer to execute it.

Images