Information processing device, information processing system and method, and non-transitory storage medium

Abstract

The present invention provides an information processing device, an information processing system and method, and a non-temporary storage medium, which can generate an image without a moving object using an image obtained from a vehicle. The system includes an acquisition unit, a detection unit, a selection unit, and a synthesis unit. The acquisition unit acquires captured images that meet predetermined conditions, including a capture condition, a capture condition, and a moving object condition related to the moving object in the captured image, from captured images taken by multiple vehicles, and vehicle information including position information corresponding to the captured image. The detection unit detects a moving object existing in the captured image obtained. The selection unit selects a captured image with a predetermined similarity or higher from other captured images obtained that correspond to the shooting position of the captured image in which the moving object was detected, based on the captured image obtained and the vehicle information. The synthesis unit removes the detected moving object from the captured image, extracts an image corresponding to the removed area from the selected captured image, and synthesizes the image.

Description

Technical Field

[0001] The present disclosure relates to an information processing device, an information processing system, and an information processing program that collect images captured by a vehicle and store them in a viewable manner. Background Art

[0002] In Japanese Patent Gazette No. 2012-129961, an image database construction system having an image database is proposed, which includes: a receiving mechanism for receiving information sent from a terminal; an image discrimination mechanism for discriminating whether to adopt the latest image of the place where the image is captured based on the information received by the receiving mechanism; and an image storage mechanism for storing the image sent from the terminal as the latest image of the place where the above-mentioned image is captured when the image discrimination mechanism determines that the latest image is adopted.

[0003] However, in the technology disclosed in Japanese Patent Application Laid-Open No. 2012-129961, since the background image is blocked by moving objects such as people and vehicles, an image without the moving object cannot be obtained, and thus there is room for improvement. Summary of the Invention

[0004] The present disclosure provides an information processing device, an information processing system, and an information processing program capable of generating an image in which no moving object exists using an image acquired from a vehicle.

[0005] The information processing device of the first embodiment includes: an acquisition unit, which acquires, from captured images captured by multiple vehicles, captured images that meet predetermined conditions, including a capture condition, a capture condition, and a moving body condition related to the moving body in the captured image, and vehicle information including position information corresponding to the captured image; a detection unit, which detects the moving body existing in the captured image acquired by the above-mentioned acquisition unit; a selection unit, which selects, based on the captured image acquired by the above-mentioned acquisition unit and the above-mentioned vehicle information, the captured image having a similarity greater than or equal to a predetermined degree from the other captured images acquired by the above-mentioned acquisition unit that correspond to the capturing position of the captured image in which the moving body is detected by the above-mentioned detection unit; and a synthesis unit, which removes the moving body detected by the above-mentioned detection unit from the above-mentioned captured image, extracts an image corresponding to the removed area from the above-mentioned captured image selected by the above-mentioned selection unit, and synthesizes the image.

[0006] According to the first embodiment, an acquisition unit acquires captured images captured by multiple vehicles, the captured images satisfying predetermined conditions including a capture condition, a capture condition, and a moving body condition related to the moving body in the captured images, and vehicle information including position information corresponding to the captured images.

[0007] In the detection unit, a moving body present in the captured image acquired by the acquisition unit is detected, and in the selection unit, a captured image having a similarity greater than or equal to a predetermined degree is selected from other captured images acquired by the acquisition unit and corresponding to a shooting position of the captured image in which the moving body is detected by the detection unit, based on the captured image acquired by the acquisition unit and vehicle information.

[0008] Furthermore, the synthesis unit removes the moving object detected by the detection unit from the captured image, extracts an image corresponding to the removed area from the captured image selected by the selection unit, and synthesizes the extracted images. By synthesizing the captured images in this manner, an image without the moving object can be generated using the captured image obtained from the vehicle.

[0009] The acquisition unit may use a shooting recency condition, a shooting condition, and a moving object condition as scores, and acquire the captured image having the score being equal to or greater than a predetermined threshold.

[0010] Therefore, since the conditions of the recency of the shooting, the shooting conditions and the moving object conditions can be scored and evaluated, it is easy to obtain a captured image with good recency, good shooting conditions and a small number of pixels occupied by moving objects in the captured image.

[0011] Furthermore, the scores can be calculated such that the newer the shooting date, the higher the score for the shooting condition; the closer the brightness is to a predetermined brightness suitable for shooting conditions; the slower the vehicle speed, the higher the score for the shooting condition; and the smaller the number of pixels occupied by the moving object in the captured image, the higher the score for the moving object condition. In this way, the shooting condition, the shooting condition, and the moving object condition can be evaluated using a single score.

[0012] Furthermore, the acquisition unit may perform acquisition a predetermined number of times within a predetermined period, thereby enabling appropriate captured images to be acquired from a plurality of vehicles traveling at the target point within the predetermined period.

[0013] Furthermore, the acquisition unit may acquire the captured images by changing the threshold value so as to acquire the captured images a predetermined number of times within a predetermined period. This allows a required number of captured images to be acquired within the predetermined period.

[0014] Furthermore, the selection unit may prioritize at least one of the images of the same or similar vehicle models and the images taken at the same or similar times, thereby enabling the selection of images with a higher degree of similarity compared to selecting images of different vehicle models or at different times.

[0015] Furthermore, the selection unit may preferentially select the captured images whose vanishing points are located within a predetermined range, thereby enabling the selection of captured images with a higher degree of similarity than when selecting captured images whose vanishing points are located at completely different positions.

[0016] Furthermore, the selection unit may extract a predetermined tracking area from the captured image and select the captured image whose feature quantity in the tracking area exceeds a predetermined similarity. This allows for selection of appropriate captured images while reducing processing load. In this case, the tracking area may be an area of the captured image other than the area in which at least one of the vehicle and the adjacent vehicle is captured.

[0017] In addition, it can also be formed into an information processing system, including: the above-mentioned information processing device; and a vehicle-mounted device, including a shooting unit that is mounted on a vehicle and shoots the surroundings of the vehicle to generate the above-mentioned shot image and a detection unit that detects vehicle information including the position information of the vehicle at the time of shooting.

[0018] Alternatively, the information processing program may be formed to cause a computer to function as each unit of the above-mentioned information processing device.

[0019] As described above, according to the present disclosure, it is possible to provide an information processing device, an information processing system, and an information processing program that can generate an image in which no moving object exists, using an image acquired from a vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] An exemplary embodiment of the present invention is described in detail based on the following figures, wherein:

[0021] Figure 1 This is a diagram showing a schematic configuration of an information processing system according to this embodiment.

[0022] Figure 2 This is a block diagram showing the configuration of a vehicle-mounted device and a center server in the information processing system according to the present embodiment.

[0023] Figure 3 This is a block diagram showing the configuration of a control unit of an in-vehicle device and a central processing unit of a center server in the information processing system according to the present embodiment.

[0024] Figure 4 This is a diagram for explaining a method for generating a common image by a common image generating unit.

[0025] Figure 5 This is a flowchart showing an example of the flow of imaging processing performed by the vehicle-mounted device in the information processing system according to this embodiment.

[0026] Figure 6 This is a flowchart showing an example of a process flow when collecting captured images from a vehicle-mounted device, performed by a center server in the information processing system according to this embodiment.

[0027] Figure 7 This is a flowchart showing an example of the flow of processing performed by the vehicle-mounted device in the information processing system according to the present embodiment when transmitting a captured image in response to a request from the center server.

[0028] Figure 8 This is a flowchart showing an example of a process flow when generating a common image by the common image generating unit of the center server in the information processing system according to the present embodiment.

[0029] Figure 9 This is a flowchart showing an example of a specific processing flow of the moving image frame matching process.

[0030] Figure 10 A diagram illustrating an example of a non-tracking region. DETAILED DESCRIPTION

[0031] Hereinafter, an example of an embodiment of the present disclosure will be described in detail with reference to the drawings. Figure 1 This is a diagram showing a schematic configuration of an information processing system according to this embodiment.

[0032] In the information processing system 10 according to this embodiment, vehicle-mounted devices 16 mounted on vehicles 14 are connected to a center server 12 as an information processing apparatus via a communication network 18. In this embodiment, the vehicle-mounted devices 16 mounted on a plurality of vehicles 14 and the center server 12 can communicate with each other.

[0033] In the information processing system 10 according to this embodiment, the center server 12 performs processing to collect various data stored in a plurality of vehicle-mounted devices 16. Examples of the various data stored in the vehicle-mounted devices 16 include image information representing captured images and vehicle information representing the status of each vehicle 14. In this embodiment, the center server 12 uses the captured images collected from the vehicle-mounted devices 16 to generate captured images that do not include vehicles 14 or moving objects such as pedestrians.

[0034] Next, the configuration of each unit of the information processing system 10 according to this embodiment will be described in detail. Figure 2 This is a block diagram showing the configuration of the vehicle-mounted device 16 and the center server 12 in the information processing system 10 according to the present embodiment.

[0035] The vehicle-mounted device 16 includes a control unit 20 , a vehicle information detection unit 22 , an imaging unit 24 , a communication unit 26 , and a display unit 28 .

[0036] The vehicle information detection unit 22 detects vehicle information related to the vehicle 14, including at least the position information of the vehicle 14. As an example of vehicle information, for example, the position information of the vehicle 14, the vehicle speed, acceleration, steering angle, accelerator opening, the distance to the obstacles around the vehicle, the path, and other vehicle information are detected. Specifically, the vehicle information detection unit 22 can apply a variety of sensors or devices that obtain information indicating the state of the surrounding environment of the vehicle 14. As an example of a sensor or device, there can be cited sensors such as speed sensors and acceleration sensors mounted on the vehicle 14, or GNSS (Global Navigation Satellite System) devices, on-board communicators, navigation systems, and radar devices. The GNSS device receives GNSS signals from multiple GNSS satellites to locate the position of the vehicle 14. The on-board communicator is a communication device that performs at least one of vehicle-to-vehicle communication with other vehicles 14 and road-to-vehicle communication with roadside devices via the communication unit 26. The navigation system includes a map information storage unit that stores map information. Based on the location information obtained from the GNSS device and the map information stored in the map information storage unit, the system displays the location of the vehicle 14 on a map and guides the vehicle 14 along a route to its destination. Furthermore, the radar system includes multiple radars with different detection ranges. These radars detect objects such as pedestrians and other vehicles 14 around the vehicle 14 and obtain the relative positions and speeds of the detected objects relative to the vehicle 14. The radar system also includes a built-in processing unit that processes the detection results of surrounding objects. Based on changes in the relative positions and speeds of each object, as determined by the most recent detection results, this processing unit removes noise, roadside objects such as guardrails, and other objects from the monitoring list, and tracks pedestrians and other vehicles 14 as the monitoring targets. The radar system then outputs information such as the relative positions and speeds of each monitoring target object.

[0037] The imaging unit 24 is, for example, mounted on the vehicle 14 and captures the vehicle's surroundings, such as the front of the vehicle 14, generating dynamic image data representing the captured image as image information. For example, a camera such as a drive recorder can be used as the imaging unit 24. Furthermore, the imaging unit 24 can also capture the vehicle's surroundings, at least to the side and rear of the vehicle 14. Furthermore, the imaging unit 24 can also capture the interior of the vehicle. In this embodiment, the image information generated by the imaging unit 24 is temporarily stored in the control unit 20, but it can also be uploaded to the center server 12, etc., without being stored.

[0038] The communication unit 26 communicates with the center server 12 via the communication network 18 to transmit and receive various data, such as image information obtained by the imaging unit 24 and vehicle information detected by the vehicle information detection unit 22. The communication unit 26 can also establish communication between vehicles to perform inter-vehicle communication.

[0039] The display unit 28 provides various information to the occupants by displaying various information, such as information provided from the center server 12 .

[0040] like Figure 3 As shown, the control unit 20 is composed of a general microcomputer including a CPU (Central Processing Unit) 20A, a ROM (Read Only Memory) 20B, a RAM (Random Access Memory) 20C, a memory 20D, an interface (I / F) 20E, and a bus 20F.

[0041] The control unit 20 performs processing such as uploading various information to the center server 12 by having the CPU 20A as the second processor expand the program stored in the ROM 20B as the second memory into the RAM 20C and execute it. Alternatively, the program may be expanded from the storage 20D as the second memory into the RAM 20C.

[0042] On the other hand, Figure 2 As shown, the center server 12 includes a central processing unit 30 , a central communication unit 36 , and a DB (database) 38 .

[0043] like Figure 3 As shown, the central processing unit 30 is composed of a general microcomputer including a CPU 30A, a ROM 30B, a RAM 30C, a memory 30D, an interface (I / F) 30E, a bus 30F, etc. Alternatively, a GPU (Graphics Processing Unit) may be applied to the CPU 30A.

[0044] The central processing unit 30 functions as a captured image acquisition unit 40, an acquisition condition management unit 50, and a common image generation unit 60 by CPU 30A, which serves as the first processor, expanding programs stored in ROM 30B, which serves as the first memory, or storage 30D, which serves as the first memory, into RAM 30C and executing the programs. The captured image acquisition unit 40 and the acquisition condition management unit 50 correspond to the acquisition unit. The common image generation unit 60 corresponds to the detection unit, the selection unit, and the synthesis unit, and will be described in detail later.

[0045] The captured image acquisition unit 40 acquires captured images and vehicle information including position information corresponding to the captured images from a plurality of vehicles 14, based on conditions set by the acquisition condition management unit 50, and stores the captured images in the DB 38. The captured image acquisition unit 40 can acquire the images a predetermined number of times within a predetermined period. This allows appropriate captured images to be acquired from a plurality of vehicles traveling at a target location within the predetermined period.

[0046] The acquisition condition management unit 50 manages the acquisition conditions for captured images acquired from multiple vehicles 14. Specifically, the acquisition condition management unit 50 sets the conditions for acquiring captured images from vehicles 14 so that captured images satisfy predetermined conditions, including freshness, shooting conditions, and moving object conditions related to moving objects in the captured images. For example, the acquisition condition management unit 50 manages the acquisition of images such that the freshness condition is the most recent captured image, the shooting conditions are favorable (such as daytime, clear skies, and low speeds), and the moving object condition is such that the number of pixels occupied by moving objects such as pedestrians and vehicles 14 is small. The acquisition condition management unit 50 assigns scores to the multiple conditions, including the freshness condition, shooting conditions, and moving object condition, and the captured image acquisition unit 40 manages the acquisition of images such that the scores exceed a predetermined threshold. This allows for scoring of multiple conditions, making it easier to acquire captured images with favorable freshness, favorable shooting conditions, and a small number of pixels occupied by moving objects in the captured images. For example, scores are calculated such that the newer the shooting date, the higher the score for the recency condition; the closer the brightness is to a predetermined brightness suitable for shooting conditions; the slower the vehicle speed, the higher the score for the shooting condition; and the smaller the number of pixels occupied by the aforementioned moving object in the captured image, the higher the score for the moving object condition. This allows a single score to evaluate the recency condition, the shooting condition, and the moving object condition. Scores for multiple conditions, such as the recency condition, the shooting condition, and the moving object condition, are calculated using, for example, vehicle information obtained from the vehicle-mounted device 16 along with the captured image.

[0047] The common image generation unit 60 detects a moving object in a captured image and, based on the captured images and vehicle information stored in the database 38, selects a captured image with a predetermined degree of similarity or greater from other captured images stored in the database 38, corresponding to the capture location of the captured image in which the moving object was detected. In other words, captured images are selected that are similar in themselves and captured at the same or similar locations. Specifically, the captured images are selected through a dynamic image frame matching process. The dynamic image frame matching process extracts captured images within a predetermined range (e.g., 10 meters before and after) of the captured images of the vehicle 14 traveling at the same location based on the position information. Feature quantities (specifically, a collection of multiple local feature quantity vectors, local feature quantities at multiple locations) are calculated for each image, and matching of the feature quantities of the predetermined tracking area is determined. Based on the matching results, a captured image with a high degree of similarity is selected. Here, as an example, the predetermined tracking area is an area other than the area where the vehicle 14 is reflected, such as the hood. When selecting images from images of other vehicles 14 that correspond to the capture position of the image in which the moving object was detected, priority can be given to at least one of images of the same or similar vehicle model and images captured at the same or similar time. This allows for the selection of images with a higher degree of similarity compared to selecting images of different vehicle models or at different times. Furthermore, when selecting images with a predetermined degree of similarity or greater, the vanishing point can be used to prioritize images whose vanishing point is within a predetermined range (for example, within a predetermined range such as 10 to 20 pixels offset from the position of the vanishing point relative to the captured image). This allows for the selection of images with a higher degree of similarity compared to selecting images whose vanishing point is located at a completely different position. Alternatively, when performing dynamic image frame matching, the vanishing point position can be used to correct for the offset in the captured images before performing matching. Furthermore, if images from the same viewpoint, such as in the same lane, are unavailable, matching can be performed after performing lateral corrections.

[0048] In addition, the common image generation unit 60 performs a removal process, wherein the removal process is to identify a moving object in a captured image and remove the moving object from the captured image, and a synthesis process is to extract an image corresponding to the removed area from the captured image selected by the dynamic image frame matching process and synthesize it. The image generated by the removal process and the synthesis process is stored as a common image in the DB 38. For example, Figure 4As shown, when there is a front vehicle 14 and a pedestrian 64 in the focus shot 62 among the uploaded shot images, a removed shot image 66 is generated by removing the pedestrian 64 and the vehicle 14 from the focus shot 62. In addition, when there is a pedestrian 64 in the selected shot 68 selected by the dynamic image frame matching process, a removed selected shot image 70 is generated by removing the pedestrian 64. Furthermore, the images corresponding to the pedestrian 64 and the vehicle 14 in the removed shot image 66 are extracted from the removed selected shot image 70 and synthesized with the removed shot image 66 to generate a common image 72. When extracting from the removed selected shot image 70 and synthesizing with the removed shot image 66, abstract features are extracted and synthesized in combination with conditions such as the brightness of the image, such as lighting conditions. In this way, a common image 72 in which there is no vehicle 14 or pedestrian 64 can be generated and stored in DB38. Among them, Figure 4 This figure is used to illustrate the common image generation method used by the common image generation unit 60. While the following description assumes that the removal and synthesis of moving objects are performed based on the shape of the moving object, a method may also be employed in which bounding boxes (bounding boxes) of the recognized moving object are removed and the bounding box area is extracted from images captured by other vehicles 14 for synthesis. Furthermore, in this embodiment, when removing the moving object based on its shape, a sufficient area is removed to include a shape larger than the moving object and encompass its periphery. However, removal may also be performed in a shape that matches the outline of the moving object.

[0049] The central communication unit 36 establishes communication with the vehicle-mounted device 16 via the communication network 18 to transmit and receive information such as image information and vehicle information.

[0050] The DB 38 stores data acquired from each vehicle 14 by requesting transmission of information to each vehicle 14, and also stores the common image 72 generated by the common image generation unit 60. Examples of the data acquired from and stored in the vehicles 14 include image information representing images captured by the image capture unit 24 of each vehicle 14 and vehicle information detected by the vehicle information detection unit 22.

[0051] Among them, it is preferable that the image used for map generation etc. is the most recent image and is an image captured under predetermined favorable shooting conditions such as daytime, sunny weather, and low driving speed.

[0052] In contrast, in this embodiment, since common image 72 is generated without moving objects such as the preceding vehicle 14 and pedestrians 64, it is preferred to obtain images captured without moving objects. Therefore, in addition to the aforementioned capturing conditions, the acquisition conditions can be managed to minimize uploading under the following conditions. To minimize this, for example, if only captured images that do not meet the conditions exist within the past month, the captured images that do not meet the conditions will be used to generate the common image.

[0053] As an example of the acquisition condition, pedestrian detection information is detected in an ADAS (Advanced Driver-Assistance Systems) function including a function of detecting a pedestrian 64 to avoid a collision, and a captured image in which the pedestrian 64 is detected is used.

[0054] As another example of the acquisition condition, in the same manner, when the vehicle is following the preceding vehicle, especially when the distance between vehicles is narrow or when the vehicle is in a traffic jam, the surrounding lanes may be considered in addition to the own lane.

[0055] As another example of the acquisition condition, density information of pedestrians 64 and vehicles 14 is acquired from another database (eg, a mobile spatial statistics database) to avoid capturing images of areas with high density.

[0056] As another example of the acquisition condition, in order to avoid rainy days and icy conditions, captured images before and after the slip is observed are avoided based on vehicle information such as the operation of the ABS (Anti-lock Brake System).

[0057] Another example of the acquisition condition is to upload a captured image of the vehicle traveling in the right lane and making a lane change while avoiding traffic on the left. In other words, the captured image is uploaded of the vehicle traveling in a lane as close to the sidewalk as possible without making a lane change.

[0058] The upload decision by the acquisition condition management unit 50 using the acquisition conditions described above can be performed offline, for example. In this case, the upload decision is made based on the score of the combination of the above-mentioned multiple conditions. The score of the combination of multiple conditions is calculated using, for example, a weighted sum.

[0059] Specifically, only vehicle information having a smaller data volume than a captured image is uploaded in advance, and appropriate captured images are extracted and uploaded from a plurality of vehicles that have traveled at a target point within a certain period of time.

[0060] Since the storage of the vehicle 14 is limited, it is necessary to make a judgment on uploading in a certain period (update threshold: for example, 1 week, etc.) that is shorter than the update frequency of the common image DB 38 (for example, 1 month). Therefore, the threshold value of the appropriate number of uploads is determined based on the situation so far and an upload instruction is given. That is, the threshold value can be changed to give an upload instruction in such a way that a predetermined number of acquisitions are made within a predetermined period. In this way, the required number of captured images can be acquired within a predetermined period. When changing the threshold value, the threshold value is changed based on the actual driving performance in the past. For example, when it is necessary to acquire 1 captured image in 1 month, if it is known that 4 captured images with scores of (1, 2, 4, 8) were acquired in the previous period, the threshold value is set to about 6 to make the judgment on uploading. In addition, if the upload is performed in excess of the threshold value during the current month, the threshold value is increased during the next updated threshold value period (next week). If no images are collected during the period, the threshold value is lowered. Furthermore, since the disorder of vehicles 14 and pedestrians varies depending on the road or region, the update threshold value may be designed to be different for each road or each region.

[0061] Next, specific processing performed by each unit of the information processing system 10 of this embodiment configured as described above will be described.

[0062] First, a specific process when the imaging unit 24 captures an image of the vehicle surroundings in the vehicle-mounted device 16 will be described. Figure 5 This is a flowchart showing an example of the flow of the imaging process performed by the vehicle-mounted device 16 of the information processing system 10 according to this embodiment. Figure 5 The processing is started when, for example, an ignition switch (not shown) of the vehicle 14 is turned on and the vehicle-mounted device 16 is started.

[0063] In step 100 , the CPU 20A starts imaging the surroundings of the vehicle and proceeds to step 102 . That is, the imaging unit 24 starts imaging the surroundings of the vehicle.

[0064] In step 102, the CPU 20A obtains necessary vehicle information as a profile of the captured image and proceeds to step 104. The vehicle information is obtained by obtaining the detection results of the vehicle information detection unit 22. Furthermore, information such as weather information at the time of capture, capture conditions, and congestion information can also be obtained from an external server.

[0065] In step 104 , the CPU 20A adds the acquired attribute information to the captured image and then moves to step 106 .

[0066] In step 106, the CPU 20A stores the attributed captured image in the memory 20D and proceeds to step 108. The attributed captured image is stored in association with the attribute information, and is stored so that only the attribute information can be read.

[0067] In step 108, CPU 20A determines whether shooting has ended. This determination involves, for example, determining whether the ignition switch (not shown) has been turned off. If this determination is negative, the process returns to step 102 to continue shooting and repeat the above-described process. If the determination is positive, the series of shooting processes ends.

[0068] Next, a specific process of collecting captured images from the vehicle-mounted device 16 in the center server 12 will be described. Figure 6 This is a flowchart showing an example of the process flow when collecting captured images from the vehicle-mounted device 16 by the center server 12 in the information processing system 10 according to this embodiment. As described above, the common image 72 is updated at a shorter period (e.g., one week) than the predetermined update frequency (e.g., one month) of the common image 72. Figure 6 processing.

[0069] In step 200, the CPU 30A requests each vehicle-mounted device 16 to transmit attribute information for a predetermined period (e.g., one week), and the process proceeds to step 202. Specifically, the acquisition condition management unit 50 requests each vehicle-mounted device 16 to acquire attribute information for a predetermined period from the attribute information stored in the memory 20D of the vehicle-mounted device 16.

[0070] In step 202 , CPU 30A determines whether the attribute information has been received. This determination determines whether the requested attribute information has been received, and the process waits until the determination is affirmative before moving on to step 204 .

[0071] In step 204, CPU 30A scores the plurality of acquisition conditions for acquiring a captured image, calculates a score, and proceeds to step 206. For example, as described above, the score of each captured image corresponding to the attribute information is calculated using weighted averaging or the like.

[0072] In step 206, CPU 30A determines a captured image to be uploaded based on the score and moves to step 208. For example, a captured image having a score equal to or greater than a predetermined threshold is selected as an upload target.

[0073] In step 208, CPU 30A requests the vehicle-mounted device 16 to transmit the captured image to be uploaded and the process proceeds to step 210. For example, acquisition condition management unit 50 requests the vehicle-mounted device 16 to transmit the captured image having a calculated score equal to or greater than a predetermined threshold.

[0074] In step 210 , CPU 30A determines whether a captured image of the target has been received, and waits until the determination is affirmative before proceeding to step 212 .

[0075] In step 212 , the CPU 30A sequentially accumulates the received captured images in the DB 38 and ends a series of evaluation image collection processes.

[0076] Next, a description will be given of specific processing performed by the vehicle-mounted device 16 when transmitting a captured image in response to a request from the center server 12 . Figure 7 This is a flowchart showing an example of the process flow when the vehicle-mounted device 16 in the information processing system 10 according to the present embodiment transmits a captured image in response to a request from the center server 12. Figure 7 processing.

[0077] In step 300 , the CPU 20D extracts attribute information of images captured during a predetermined period from the memory 20D and the process proceeds to step 302 .

[0078] In step 302 , the CPU 20D transmits the extracted attribute information to the center server 12 and the process proceeds to step 304 .

[0079] In step 304, the CPU 20D determines whether a request to transmit a captured image has been issued from the center server 12. This determination is made by determining whether a request to transmit a captured image has been issued in step 208. The process waits until this determination is affirmative and then proceeds to step 306.

[0080] In step 306 , the CPU 20D extracts the requested captured image from the memory 20D and the process proceeds to step 308 .

[0081] In step 308 , the CPU 20D transmits the requested captured image to the center server 12 and terminates a series of captured image transmission processes.

[0082] Next, a specific process of generating the common image 72 in the center server 12 will be described. Figure 8 This is a flowchart showing an example of the process flow when generating a common image by the common image generating unit 60 of the center server 12 in the information processing system 10 according to this embodiment. Figure 8 processing.

[0083] In step 400 , CPU 30A reads out the captured image 62 of interest from the captured images accumulated in DB 38 during a predetermined period, and the process proceeds to step 402 .

[0084] In step 402, CPU 30A performs dynamic image frame matching processing and moves to step 404. Dynamic image frame matching processing, for example, extracts captured images of a specified range (such as 10 meters in front and behind) of the captured images of the comparison object from captured images of vehicles 14 traveling at the same location, calculates local feature quantities for each, grasps the matching of the local feature quantities of the tracking area, and selects the captured image with the highest similarity as the selected captured image 68 based on the matching results. In this way, by grasping the matching of the local feature quantities of the tracking area, it is possible to select an appropriate captured image while reducing the processing load. In addition, the dynamic image frame matching processing corresponds to the selection unit, and the detailed processing will be described later. In addition, as described above, the tracking area is an area outside the area where the vehicle 14 is reflected, such as the hood, but is not limited to this. For example, an area outside the area in the captured image where at least one of the vehicle 14 and the surrounding moving objects is captured may be used as a predetermined tracking area.

[0085] In step 404, CPU 30A identifies moving objects in the captured images and proceeds to step 406. For example, using deep learning-based techniques such as Semantic Segmentation and YoloV4, moving objects such as pedestrians 64 and vehicles 14 are identified. This identification is performed separately for the captured image 62 of interest and the selected captured image 68 extracted through dynamic image frame matching. The processing in step 404 corresponds to the detection unit.

[0086] In step 406 , the CPU 30A removes the moving objects in each of the attention captured image 62 and the selected captured image 68 and proceeds to step 408 .

[0087] In step 408, CPU 30A extracts the area to be removed from selected captured image 68 selected by the dynamic image frame matching process and moves to step 410. Although the description is simplified here, a selected image 68 is selected in which no moving object exists or exists at a position different from that of the moving object in captured image 62 of interest, and the selected captured image 68 is used to fill the area of captured image 62 of interest from which the moving object has been removed.

[0088] In step 410, the CPU 30A combines the region extracted from the selected captured image 68 with the region of the target captured image 62 from which the moving object has been removed, and the process proceeds to step 412. The processes of steps 406 to 410 correspond to a combining unit.

[0089] At step 412 , CPU 30A stores the synthesized image as the common image 72 in DB 38 and then moves to step 414 .

[0090] In step 414, CPU 30A determines whether the generation of common image 72 has ended. This determination determines whether the above-described processing has been completed for the images captured during the predetermined period. If this determination is negative, the process returns to step 400 and the above-described processing is repeated using another captured image as the focus captured image 62. If the determination in step 414 is positive, the series of processing by common image generation unit 60 ends.

[0091] By generating the common image 72 in this manner, it is possible to generate a captured image without a moving object using the captured image acquired from the vehicle 14 .

[0092] Next, the above-mentioned moving image frame matching process will be described. Figure 9 This is a flowchart showing an example of a specific processing flow of the moving image frame matching process.

[0093] In step 500, the CPU 30A extracts the vehicles 14 that have traveled in the same area and moves to step 502. For example, the vehicles 14 that have traveled in the same area are extracted based on the position information included in the vehicle information.

[0094] In step 502 , the CPU 30A extracts captured images of vehicles near the comparison target vehicle and proceeds to step 504 . Specifically, the CPU 30A extracts captured images of vehicles 14 near the vehicle 14 that captured the target captured image as a candidate group for the selected captured image 68 .

[0095] In step 504, CPU 30A calculates the feature value of the captured image of the comparison target vehicle and moves to step 506. Here, the feature value is a set of a plurality of local feature value vectors, and the local feature values at a plurality of positions are calculated.

[0096] In step 506, CPU 30A calculates the feature value of the extracted image group and moves to step 508. That is, the local feature value of each candidate group of the selected captured image 68 is calculated.

[0097] In step 508, the CPU 30A determines a non-tracking area and moves to step 510. As a non-tracking area, for example, Figure 10 As shown, at least one of the vehicle region 74 in the captured image, where the hood of the vehicle 14 is reflected, and the adjacent vehicle region 76, such as a vehicle 14 traveling next to the vehicle 14, is defined as a non-tracking region. For example, the non-tracking region is determined by semantic segmentation.

[0098] In step 510, CPU 30A finds a match for the feature values of the tracking area other than the non-tracking area and proceeds to step 512. By setting a non-tracking area to find a match for the feature values (specifically, a set of multiple local feature value vectors, or local feature values at multiple locations), the processing load can be reduced compared to not setting a non-tracking area. Alternatively, step 508 can be omitted and the matching of the local feature values can be found without setting a non-tracking area.

[0099] In step 512 , the CPU 30A selects an image with a high similarity as the selected captured image 68 based on the matching result of the feature amounts and ends the series of processing.

[0100] By performing the moving image frame matching process in this manner, the selected captured images 68 required to complete the region from which the moving object is removed in the captured image 62 of interest can be selected, and the common image 72 in which the moving object does not exist can be generated.

[0101] In addition, in the above embodiment, the functions of the captured image acquisition unit 40, the acquisition condition management unit 50, and the common image generation unit 60 are described as functions of a single center server 12, but this is not limited to this, and the functions may be distributed across multiple servers.

[0102] In the above embodiment, the center server 12 requests and obtains images to be uploaded, but the present invention is not limited thereto. The vehicle-mounted device 16 may calculate scores and upload captured images with scores exceeding a threshold to the center server 12 .

[0103] Furthermore, in the above embodiment, when removing a moving object from a captured image and filling the area where the moving object was removed using other captured images, an example is described in which the removal target area is extracted from a single captured image and synthesized. However, the present invention is not limited to this. For example, an image of the removal target area may be generated using multiple captured images, and the generated image may be synthesized with the area where the moving object was removed from the captured image.

[0104] In the above-described embodiment, the processing performed by the center server 12 and the vehicle-mounted device 16, respectively, is described as software processing performed by executing programs, but the present invention is not limited to this. For example, processing may be performed using hardware such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array). Alternatively, processing may be a combination of software and hardware. In the case of software processing, the program may be stored in various non-transitory storage media for distribution. Furthermore, while the CPU 20A or the CPU 30A has been described as an example of a processor performing software processing, the present invention is not limited to this. For example, a GPU, an ASIC, an FPGA, or a PLD (Programmable Logic Device) may also be employed.

[0105] Furthermore, the present disclosure is not limited to the above contents, and can of course be implemented with various modifications other than the above contents without departing from the gist of the present disclosure.

Claims

1. An information processing device, wherein: include: an acquisition unit for acquiring, from among images captured by a plurality of vehicles, a captured image satisfying a plurality of predetermined conditions, including a condition regarding the recency of the image, a capturing condition, and a moving object condition related to the moving object in the captured image, and vehicle information including position information corresponding to the captured image; a detection unit that detects the moving object present in the captured image acquired by the acquisition unit; a selecting unit that selects, based on the captured image acquired by the acquiring unit and the vehicle information, the captured image having a predetermined degree of similarity or greater from among the other captured images acquired by the acquiring unit and corresponding to the capturing position of the captured image in which the moving object is detected by the detecting unit; as well as a synthesis unit that removes the moving object detected by the detection unit from the captured image, extracts an image corresponding to the removed area from the captured image selected by the selection unit, and synthesizes the extracted image; The selecting unit preferentially selects at least one of the captured images of the same or similar vehicle type and the captured images taken at the same or similar time. The captured image is selected through dynamic image frame matching processing. When performing the dynamic image frame matching processing, matching is performed after correcting the offset of the captured image using the position of the vanishing point. If the captured image of the same viewpoint of the same lane cannot be obtained, matching is performed after performing lateral correction.

2. The information processing device according to claim 1, wherein The acquisition unit uses a shooting recency condition, a shooting condition, and a moving object condition as scores, and acquires the captured image having the score being equal to or greater than a predetermined threshold value.

3. The information processing device according to claim 2, wherein: The score is calculated in such a way that the newer the shooting date, the higher the score of the shooting recency condition; the closer the brightness is to the predetermined brightness suitable for the conditions at the time of shooting; the slower the vehicle speed, the higher the score of the shooting condition; and the fewer the number of pixels occupied by the moving object in the captured image, the higher the score of the moving object condition.

4. The information processing device according to any one of claims 1 to 3, wherein: The acquisition unit performs acquisition a predetermined number of times within a predetermined period.

5. The information processing device according to claim 2 or 3, wherein: The acquisition unit acquires the captured image by changing the threshold value so as to perform acquisition a predetermined number of times within a predetermined period.

6. The information processing device according to any one of claims 1 to 3, wherein: The selection unit preferentially selects the captured image in which the position of the vanishing point in the captured image is within a predetermined range.

7. The information processing device according to any one of claims 1 to 3, wherein: The selection unit extracts a predetermined tracking area from the captured image and selects the captured image having a feature amount of the tracking area having a predetermined similarity or higher.

8. The information processing apparatus according to claim 7, wherein: The tracking area is an area other than an area in which at least one of the host vehicle and surrounding moving objects is captured in the captured image.

9. An information processing system, wherein: include: The information processing device according to any one of claims 1 to 8; and The vehicle-mounted device includes an imaging unit mounted on a vehicle and capturing an image of the vehicle's surroundings to generate the captured image, and a detection unit detecting vehicle information including position information of the vehicle at the time of capturing the image.

10. An information processing method, wherein: Acquire, by a first processor, captured images of a plurality of vehicles, wherein the captured images satisfy a plurality of predetermined conditions, including a condition regarding the state of the captured images, a capturing condition, and a moving object condition related to the moving object in the captured images, and vehicle information including position information corresponding to the captured images; detecting the moving object in the captured image using a first processor, selecting, by a first processor, the captured image having a predetermined degree of similarity or greater from among the other captured images obtained and corresponding to the capturing position of the captured image in which the moving object is detected, based on the captured image and the vehicle information; removing the detected moving object from the captured image using a first processor, extracting an image corresponding to the removed area from the selected captured image and synthesizing the extracted image, Prioritizing at least one of the captured images of the same or similar vehicle model and the captured images taken at the same or similar time, The captured image is selected through dynamic image frame matching processing. When performing the dynamic image frame matching processing, matching is performed after correcting the offset of the captured image using the position of the vanishing point. If the captured image of the same viewpoint of the same lane cannot be obtained, matching is performed after performing lateral correction.

11. The information processing method according to claim 10, wherein: The first processor uses a condition of imaging recency, an imaging condition, and a moving object condition as scores, and acquires the captured image having the score being equal to or greater than a predetermined threshold.

12. The information processing method according to claim 11, wherein: The score is calculated using the first processor in such a manner that the newer the shooting date, the higher the score of the shooting condition; the closer the brightness is to the predetermined brightness suitable for the conditions at the time of shooting; the slower the vehicle speed, the higher the score of the shooting condition; and the fewer the number of pixels occupied by the moving object in the captured image, the higher the score of the moving object condition.

13. The information processing method according to any one of claims 10 to 12, wherein: The first processor performs acquisition a predetermined number of times within a predetermined period.

14. The information processing method according to claim 11 or 12, wherein: The captured image is acquired by the first processor by changing the threshold value so that acquisition is performed a predetermined number of times within a predetermined period.

15. A non-transitory storage medium, wherein: A program for causing the first processor to execute the following information processing is stored: Acquire captured images, from among captured images captured by a plurality of vehicles, that satisfy a plurality of predetermined conditions, including a capture condition, a capture condition, and a moving object condition related to a moving object in the captured image, and vehicle information including position information corresponding to the captured images; detecting the moving object present in the captured image, Based on the captured image and the vehicle information, the captured image having a predetermined similarity or higher is selected from among the other captured images obtained and corresponding to the capturing position of the captured image in which the moving object is detected. The detected moving object is removed from the captured image, and an image corresponding to the removed area is extracted from the selected captured image and synthesized. Prioritizing at least one of the captured images of the same or similar vehicle model and the captured images taken at the same or similar time, The captured image is selected through dynamic image frame matching processing. When performing the dynamic image frame matching processing, matching is performed after correcting the offset of the captured image using the position of the vanishing point. If the captured image of the same viewpoint of the same lane cannot be obtained, matching is performed after performing lateral correction. The non-transitory storage medium according to claim 15 , wherein: The method includes a process of using a shooting recency condition, a shooting condition, and a moving object condition as scores and acquiring the captured image having the score being equal to or greater than a predetermined threshold value.

17. The non-transitory storage medium according to claim 16, wherein: The processing includes calculating the score in such a manner that the newer the shooting date, the higher the score of the shooting condition; the closer the brightness is to the predetermined brightness suitable for the conditions at the time of shooting; the slower the vehicle speed, the higher the score of the shooting condition; and the smaller the number of pixels occupied by the moving object in the captured image, the higher the score of the moving object condition.

18. The non-transitory storage medium according to any one of claims 15 to 17, wherein: This includes a process of acquiring data a predetermined number of times within a predetermined period.

19. The non-transitory storage medium according to claim 16 or 17, wherein: The method includes a process of acquiring the captured image by changing the threshold value so as to acquire the captured image a predetermined number of times within a predetermined period.

Images