Information processing device, system, method, and program

Abstract

To provide an information processor which can specify a moving object in vehicle periphery from an image imaged by an image sensor in a moving vehicle.SOLUTION: An information processor mounted on a vehicle having a sensor includes: an acquisition part for acquiring information on the peripheral image of the vehicle and the behavior of the vehicle, from a sensor; and a specification part for specifying a moving object in the peripheral of the vehicle, on the basis of the information on the change with time of the peripheral image of the vehicle and the behavior of the vehicle.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an information processing device mounted on a vehicle equipped with a sensor. [Background technology]

[0002] Patent Document 1 discloses a moving object detection device that can distinguish whether a target is a moving object or a stationary object even when the vehicle is traveling. This moving object detection device detects targets using a differential frequency spectrum obtained by a sensor that transmits a signal into the space ahead and receives a signal reflected by a target in front, thereby making it possible to distinguish whether a detected target is a moving object or a stationary object even when the vehicle is traveling. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2006-058135 A Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, the number of imaging devices such as cameras that can acquire images has been increasing as sensors mounted on vehicles. In the future, it will be desirable to be able to identify moving objects present in images based on images of the surroundings of a vehicle acquired by the vehicle while moving using image sensors such as on-board cameras.

[0005] The present disclosure has been made in consideration of the above-mentioned problems, and aims to provide an information processing device and the like that can identify moving objects around a moving vehicle from images captured by an image sensor in the vehicle. [Means for solving the problem]

[0006] In order to solve the above problems, one aspect of the disclosed technology is an information processing device mounted on a vehicle equipped with a sensor, the information processing device including: an acquisition unit that acquires an image of the vehicle's surroundings and information regarding the vehicle's behavior from the sensor; and an identification unit that identifies moving objects in the image of the vehicle's surroundings based on changes over time in the image of the vehicle's surroundings and the information regarding the vehicle's behavior. Effect of the Invention

[0007] According to the information processing device disclosed herein, even when a vehicle equipped with various sensors including image sensors such as cameras is moving, a moving object can be identified based on changes over time in images obtained from the sensors and information regarding the behavior of the vehicle. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a schematic configuration diagram of a system including an information processing device according to an embodiment of the present disclosure; [Figure 2A] Flowchart of moving object identification processing executed by the information processing device [Figure 2B] Flowchart of moving object identification processing executed by the information processing device [Diagram 3] Flowchart of the data integration process performed by the server [Figure 4A] An example of an image of the vehicle's surroundings [Figure 4B] An example of an image of the vehicle's surroundings [Figure 4C] An example of an image of the vehicle's surroundings [Figure 5A] An example of an estimated image of the vehicle's surroundings [Figure 5B] An example of an estimated image of the vehicle's surroundings [Figure 6A] Example of difference data between surrounding image and estimated surrounding image [Figure 6B] Example of difference data between surrounding image and estimated surrounding image DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The information processing device etc. disclosed herein identifies a moving object captured in an image of the surroundings of a vehicle based on changes over time in an image of the surroundings of the vehicle acquired from an image sensor and information on the behavior of the vehicle acquired from a vehicle speed sensor, a steering angle sensor etc. Since the movement of the image sensor taking into account the behavior of the vehicle is grasped and a moving object is identified based on changes over time in the surrounding image, it becomes possible to identify a moving object with high accuracy even from an image of the surroundings captured while the vehicle is moving. Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0010] <Embodiment> [composition] Fig. 1 is a block diagram showing an example of a schematic configuration of a system including an information processing device 150 according to an embodiment of the present disclosure. The system illustrated in Fig. 1 is a system (e.g., a digital twin system) that includes a vehicle 100 equipped with the information processing device 150 and a server 200 and can provide predetermined data obtained in the vehicle 100 to the server 200.

[0011] The vehicle 100 is communicably connected to the server 200 via a communication line such as the Internet. The number of vehicles 100 that can communicate with the server 200 is not limited to one, and may be multiple. The vehicle 100 includes various sensors, such as an image sensor 110, a vehicle speed sensor 120, and a steering angle sensor 130, and an information processing device 150.

[0012] The image sensor 110 is an imaging device such as a camera installed on the windshield glass at the front of the vehicle 100 to capture an image of the front of the vehicle 100, or a camera installed on the trunk lid or tailgate at the rear of the vehicle to capture an image of the rear of the vehicle. The image captured by the image sensor 110 is output to the information processing device 150.

[0013] The vehicle speed sensor 120 is a detection device for detecting the speed of the vehicle 100. For example, a wheel speed sensor for detecting the rotation speed (or amount of rotation) of each wheel installed on the vehicle 100 can be used as the vehicle speed sensor 120. The speed of the vehicle 100 detected by the vehicle speed sensor 120 is output to the information processing device 150 as information on the behavior of the vehicle 100 (hereinafter referred to as "vehicle behavior information").

[0014] The steering angle sensor 130 is a detection device for detecting a steering angle (a rotation angle from a neutral position of a steering wheel) corresponding to a steering operation by a driver of the vehicle 100. The steering angle sensor 130 is attached, for example, to a steering shaft that rotates together with a steering wheel (not shown). The steering angle detected by the steering angle sensor 130 is output to the information processing device 150 as vehicle behavior information.

[0015] As a sensor for detecting vehicle behavior information, an acceleration sensor for detecting the magnitude of acceleration G acting on the vehicle 100 may be provided. For the acceleration sensor, for example, a three-axis accelerometer for detecting acceleration G in the front-rear direction, the width direction, and the up-down direction of the vehicle 100 may be used.

[0016] The information processing device 150 is configured to identify moving objects existing around the vehicle 100 in which the information processing device 150 is mounted, and provide the information to the server 200. This information processing device 150 is typically configured to include a processor, a memory, an input / output interface, and the like, and the processor reads and executes a program stored in the memory to realize all or a part of the functions performed by a transmission unit 151, an acquisition unit 152, and an identification unit 153 described below.

[0017] The acquisition unit 152 sequentially acquires images of the surroundings of the vehicle 100 captured by the image sensor 110 from the image sensor 110. The acquisition unit 152 also acquires the speed of the vehicle 100 detected by the vehicle speed sensor 120 as vehicle behavior information from the vehicle speed sensor 120. The acquisition unit 152 also acquires the steering angle of the vehicle 100 detected by the steering angle sensor 130 as vehicle behavior information from the steering angle sensor 130. An in-vehicle network such as CAN or LIN is used to acquire the surrounding images and vehicle behavior information.

[0018] The identification unit 153 identifies a moving object present in the image of the surroundings of the vehicle 100 based on a change over time in the image of the surroundings of the vehicle 100 acquired by the acquisition unit 152 and the speed and / or steering angle of the vehicle 100. Examples of the moving object include objects that can move, such as automobiles, motorcycles, bicycles, and pedestrians. The method of identifying this moving object will be described later.

[0019] When the identifying unit 153 identifies a moving object, the transmitting unit 151 transmits data related to the moving object among the data of the surrounding image to the server 200. The data related to the moving object includes data necessary for the integration process performed by the server 200, which will be described later, such as the position information (latitude, longitude) and traveling direction of the moving object.

[0020] The server 200 is provided outside the vehicle 100 so as to be able to communicate with the vehicle 100, and can collect various data transmitted from the vehicle 100. For example, the server 200 can be a cloud server configured on a cloud. This server 200 is typically configured to include a processor, a memory, an input / output interface, and the like, and the processor reads and executes a program stored in the memory to realize all or a part of the functions performed by the receiving unit 210, the storage unit 220, and the integration unit 230 described below.

[0021] The receiving unit 210 receives data transmitted from the vehicle 100, which in this embodiment is data related to a moving object. The data related to the moving object received by the receiving unit 210 is output to the integrating unit 230.

[0022] The storage unit 220 stores data on various structures (static objects) in real space. Typically, the storage unit 220 stores data on structures necessary for generating a digital twin in a virtual space that is time-synchronized with the real space. The data on the structures may be stored in advance by an input operation by an operator or the like, or data provided from a plurality of vehicles 100 connected to the server 200 by a method different from that of the present embodiment may be stored as needed. The structures stored in the storage unit 220 desirably include at least structures that are likely to be captured by the image sensor 110 of the vehicle 100 while traveling. Examples of structures include static objects that are unlikely to move, such as buildings (such as condominiums and pedestrian bridges), white lines (such as driving lines and crosswalks), road signs, and traffic lights.

[0023] The integration unit 230 performs a process of integrating the data on moving objects transmitted (provided) from the vehicle 100 and the data on structures stored in the storage unit 220. Note that the method of data integration is not particularly limited, but as an example, it is possible to overlay data on moving objects on data on structures so that information equivalent to the entire data of the surrounding image captured by the image sensor 110 of the vehicle 100 in real space can be managed on the server 200.

[0024] [control] Next, the process performed by the information processing device 150 and the server 200 according to this embodiment will be described with further reference to Figs. 2A-B, 3, 4A-4C, 5A-B, and 6A-B. Figs. 2A and 2B are flowcharts for explaining the procedure of the moving object identification process executed by the information processing device 150. The process of Fig. 2A and the process of Fig. 2B are connected by connectors X and Y, respectively. Fig. 3 is a flowchart for explaining the procedure of the data integration process executed by the server 200. Figs. 4A, 4B, and 4C are diagrams showing an example of an actual image of the surroundings of the vehicle 100. Figs. 5A and 5B are diagrams showing an example of an estimated surroundings image (estimated surroundings image) of the vehicle 100. Figs. 6A and 6B are examples of differential data images between the surroundings image and the estimated surroundings image.

[0025] (1) Processing of information processing device The moving object identification process illustrated in FIGS. 2A and 2B is started, for example, when the ignition switch of the vehicle 100 is turned on (IG-ON), and is repeatedly executed until the ignition switch is turned off (IG-OFF).

[0026] (Step S201) After this process is started, the acquisition unit 152 of the information processing device 150 acquires, from the image sensor 110, an image IT of the surroundings of the vehicle 100 at time T and an image ITt of the surroundings of the vehicle 100 at time [T+Δt] after a predetermined time Δt has elapsed since time T. The time T is not particularly limited, but can be any time after the operation of the image sensor 110 has stabilized, for example. The predetermined time Δt can be set arbitrarily based on the performance of the image sensor 110, the processing capacity of the information processing device 150, the frequency and amount of data requested by the server 200, and the like.

[0027] 4A is an example of a surrounding image IT at time T. In the surrounding image IT at time T, there are a leading vehicle 301 traveling ahead of the vehicle 100 and an oncoming vehicle 302 traveling in the oncoming lane as moving objects, and there are trees 401-406, driving lanes 407-409, and a mountainous area 410 as stationary objects.

[0028] 4B is an example of a surrounding image ITt at time [T+Δt]. In this surrounding image ITt at time [T+Δt], compared to the surrounding image IT (FIG. 4A) at time T, the positions of the preceding vehicle 301 traveling at approximately the same speed as the vehicle 100, the straight driving lines 407-409, and the distant mountainous area 410 have hardly changed on the image, whereas the position of the oncoming vehicle 302 traveling at a higher relative speed than the vehicle 100 has changed significantly on the image.

[0029] When the acquisition unit 152 acquires the peripheral image IT at time T and the peripheral image ITt at time [T+Δt], the process proceeds to step S202.

[0030] (Step S202) The acquisition unit 152 of the information processing device 150 acquires an image IT2t of the surroundings of the vehicle 100 at time [T+2Δt] after a predetermined time 2Δt has elapsed from time T, from the image sensor 110. The time 2Δt is twice the time Δt (2Δt=Δt+Δt).

[0031] 4C is an example of a surrounding image IT2t at time [T+2Δt]. In this surrounding image IT2t at time [T+2Δt], compared with the surrounding image IT at time T (FIG. 4A), the positions of the preceding vehicle 301 traveling at approximately the same speed as the vehicle 100, the straight driving lines 407-409, and the distant mountain area 410 have hardly changed on the image, as in the surrounding image ITt at time [T+Δt] (FIG. 4B), whereas the position on the image of the oncoming vehicle 302 traveling at a higher relative speed than the vehicle 100 has changed even more significantly from the surrounding image ITt at time [T+Δt] (FIG. 4B).

[0032] When the acquisition unit 152 acquires the peripheral image IT2t at the time [T+2Δt], the process proceeds to step S203.

[0033] (Step S203) The acquisition unit 152 of the information processing device 150 acquires vehicle behavior information (vehicle speed, steering angle) from the vehicle speed sensor 120 and / or the steering angle sensor 130. The vehicle behavior information acquired here is information on the behavior of the vehicle 100 from time T to time [T+Δt] and from time [T+Δt] to time [T+2Δt]. For example, the information is that the vehicle 100 ran at 20 km / h for the time 2Δt, or that the steering wheel was tilted 10 degrees to the right for the time 2Δt. The vehicle behavior information acquired by the acquisition unit 152 may be at least one of the speed and steering angle of the vehicle 100. When the vehicle behavior information is acquired by the acquisition unit 152, the process proceeds to step S204.

[0034] (Step S204) The identification unit 153 of the information processing device 150 estimates a surrounding image ITt' of the vehicle 100 after a time Δt has elapsed from the surrounding image IT of the vehicle 100 at time T, based on the vehicle behavior information acquired by the acquisition unit 152. More specifically, assuming that all of the targets captured in the surrounding image IT are structures (stationary objects), the estimated surrounding image ITt' estimates the positions to which the targets are predicted to move based on the behavior of the vehicle 100, since the image sensor 110 also moves in the same manner if the vehicle 100 continues to be in the same state (traveling at 20 km / h or turning right with a 10-degree steering wheel) for the entire time Δt. Note that structures that can be clearly determined from the surrounding image to be stationary, such as the driving lines 407-409 and the mountainous area 410, are excluded from the target targets.

[0035] 5A is an example of an estimated surroundings image ITt' after a time Δt. In this estimated surroundings image ITt', the leading vehicle 301 and the oncoming vehicle 302 are also processed as stationary objects, and therefore their positions move in the same manner as the trees 401-406.

[0036] When the specification unit 153 estimates the surrounding image ITt' after the time Δt from the surrounding image IT, the process proceeds to step S205.

[0037] (Step S205) Based on the vehicle behavior information acquired by the acquisition unit 152, the identification unit 153 of the information processing device 150 estimates a surrounding image IT2t' after a further time Δt has passed from the surrounding image ITt of the vehicle 100 at time [T+Δt], in other words, after a time 2Δt has passed from the surrounding image IT of the vehicle 100 at time T. More specifically, assuming that all of the targets captured in the surrounding image ITt are structures (stationary objects), the estimated surrounding image IT2t' estimates the positions where the multiple targets are predicted to be moving based on the behavior of the vehicle 100, since the image sensor 110 also moves in the same manner if the vehicle 100 continues to be in the same state (driving at 20 km / h or turning right with a 10-degree steering) for the entire time Δt. Note that structures that can be clearly determined from the surrounding image to be stationary, such as the driving lines 407-409 and the mountainous area 410, are excluded from the target targets.

[0038] 5B is an example of an estimated surroundings image IT2t' after a time 2Δt. In this estimated surroundings image IT2t', the positions of the leading vehicle 301, the oncoming vehicle 302, and the trees 401 to 406, which are processed as stationary objects, have further moved compared to the estimated surroundings image ITt' (FIG. 5A) after a time Δt.

[0039] When the specification unit 153 estimates a surrounding image IT2t' after a time Δt from the surrounding image ITt (that is, after a time 2Δt from the surrounding image IT), the process proceeds to step S206.

[0040] (Step S206) The determination unit 153 of the information processing device 150 performs a predetermined difference process between the surrounding image ITt of the vehicle 100 at time [T+Δt] and the estimated surrounding image ITt' after a time Δt from the surrounding image IT. In this difference process, for example, the surrounding image ITt and the estimated surrounding image ITt' are compared to determine the positional deviation of each target.

[0041] Fig. 6A is an image diagram of the result of performing a difference process between the surrounding image ITt shown in Fig. 4B and the estimated surrounding image ITt' after a time Δt shown in Fig. 5A. As shown in Fig. 6A, by comparing the surrounding image ITt with the estimated surrounding image ITt', a difference result is obtained only for the moving preceding vehicle 301 and the oncoming vehicle 302.

[0042] When the specification unit 153 executes the difference process between the peripheral image ITt and the estimated peripheral image ITt', the process proceeds to step S207.

[0043] (Step S207) The determination unit 153 of the information processing device 150 performs a predetermined difference process between a surrounding image IT2t of the vehicle 100 at time [T+2Δt] and an estimated surrounding image IT2t' taken 2Δt after the surrounding image IT (time Δt after the surrounding image ITt). In this difference process, for example, the surrounding image IT2t and the estimated surrounding image IT2t' are compared to determine the positional deviation of each target.

[0044] Fig. 6B is an image diagram of the result of performing a difference process between the surrounding image IT2t shown in Fig. 4C and the estimated surrounding image IT2t' after a time 2Δt shown in Fig. 5B. As shown in Fig. 6B, by comparing the surrounding image IT2t with the estimated surrounding image IT2t', a difference result is obtained only for the moving preceding vehicle 301 and oncoming vehicle 302.

[0045] When the specification unit 153 executes a difference process between the surrounding image IT2t and the estimated surrounding image IT2t', the process proceeds to step S208.

[0046] (Step S208) The identification unit 153 of the information processing device 150 extracts a moving target based on the result of the difference processing between the peripheral image ITt and the estimated peripheral image ITt' and the result of the difference processing between the peripheral image IT2t and the estimated peripheral image IT2t', and identifies the extracted target as a moving object. For example, a target that is a structure (stationary object) has a movement amount that can be obtained by the difference processing within a predetermined range (including zero). In contrast, a target that is a moving object has a movement amount that can be obtained by the difference processing that is extremely small (such as the preceding vehicle 301) or an extremely large movement amount that can be obtained by the difference processing (such as the oncoming vehicle 302). Therefore, it is possible to identify a target whose position is moving outside the predetermined range as a moving object. The predetermined range can be freely set based on the vehicle behavior information, the performance of the information processing device 150, and the like. When the identification unit 153 identifies a moving object, the process proceeds to step S209.

[0047] (Step S209) The transmission unit 151 of the information processing device 150 transmits data on the moving object identified by the identification unit 153 from among the data on the peripheral images (such as differential data between the peripheral image IT and the peripheral image IT2t taking into account the movement of the moving object) to the server 200. In this embodiment, only data on the moving object, excluding data on stationary objects, is transmitted from the information processing device 150 (vehicle 100) to the server 200. When the data on the moving object is transmitted to the server 200 by the transmission unit 151, the process proceeds to step S210.

[0048] (Step S210) The acquisition unit 152 of the information processing device 150 replaces (substitutes) the time [T+Δt] with the time T and replaces (substitutes) the time [T+2Δt] with the time T[T+Δt] in order to identify a moving object in the peripheral image at the next time, and proceeds to step S202. As a result of this process, the difference process between the peripheral image ITt and the estimated peripheral image ITt' and the difference process between the peripheral image IT2t and the estimated peripheral image IT2t' are repeatedly performed at intervals of time Δt thereafter.

[0049] (2) Server processing The data integration process illustrated in FIG. 3 is started, for example, by the server 200 establishing communication with the vehicle 100, and is repeatedly executed until the communication with the vehicle 100 ends (is interrupted).

[0050] (Step S301) The receiving unit 210 of the server 200 determines whether or not data related to a moving object has been received from the vehicle 100 (information processing device 150). If the receiving unit 210 has received data related to a moving object from the vehicle 100 (step S301, Yes), the process proceeds to step S302.

[0051] (Step S302) The integrating unit 230 of the server 200 acquires data on structures from the storage unit 220. The data on structures acquired by the integrating unit 230 does not need to be all data stored in the storage unit 220, and may only be data on structures related to the periphery of the vehicle 100 from which the receiving unit 210 received data on the moving object in the above step S301. In this case, the vehicle 100 may transmit to the server 200 data on the moving object as well as data indicating the location (latitude, longitude) where the moving object was identified. When the data on the structures is acquired by the integrating unit 230, the process proceeds to step S303.

[0052] (Step S303) The integrating unit 230 of the server 200 integrates the data related to the moving object received from the vehicle 100 and the data related to the structure acquired from the storage unit 220. When the data related to the moving object is integrated with the data related to the structure by the integrating unit 230, the process proceeds to step S301 to integrate the data related to the moving object received from the vehicle 100 (information processing device 150).

[0053] <Actions, Effects, etc.> As described above, according to the information processing device 150 according to an embodiment of the present disclosure, a peripheral image of the vehicle 100 is acquired from the image sensor 110, information on the behavior of the vehicle 100 (vehicle speed, steering angle) is acquired from the vehicle speed sensor 120 and the steering angle sensor 130, and a moving object in the peripheral image of the vehicle 100 is identified based on the change over time in the peripheral image of the vehicle 100 and the information on the behavior of the vehicle 100. In this way, the information processing device 150 identifies a moving object based on the change over time in the peripheral image of the vehicle 100 after grasping the movement of the image sensor 110, and therefore it is possible to identify a moving object with high accuracy even from a peripheral image captured while the vehicle 100 is moving.

[0054] Moreover, according to the information processing device 150 of this embodiment, only data related to moving objects among data of images of the surroundings of the vehicle 100 is transmitted to the server 200. This transmission process can reduce the amount of communication data from the vehicle 100 to the server 200, thereby realizing suppression of communication costs and prevention of delays in communication time.

[0055] Furthermore, the server 200 of this embodiment is able to generate a digital twin on the server 200 by integrating data about moving objects received from the vehicle 100 with data about structures stored in the server 200.

[0056] The above describes one embodiment of the present disclosure, but the present disclosure can be viewed as a system including an information processing device, a method executed by an information processing device having a processor and a memory, a program for executing the method, a computer-readable non-transitory storage medium storing the program, and a vehicle equipped with an information processing device and a server. [Industrial Applicability]

[0057] The information processing device of the present disclosure is equipped with various sensors including an image sensor and can be used in vehicles and the like that are communicatively connected to a server. [Explanation of symbols]

[0058] 100 vehicles 110 Image Sensor 120 Vehicle speed sensor 130 Steering Angle Sensor 150 Information processing equipment 151 Transmitter 152 Acquisition Department 153 Specific part 200 Servers 210 Receiving unit 220 Storage Unit 230 Integration Department 301 Leading vehicle 302 Oncoming vehicle 401~406 Trees 407~409 Line 410 Mountains

Claims

1. An information processing device mounted on a vehicle equipped with a sensor, an acquisition unit that acquires an image of the surroundings of the vehicle and information regarding a behavior of the vehicle from the sensor; an identification unit that identifies a moving object in the surrounding image of the vehicle based on a time-dependent change in the surrounding image of the vehicle and information regarding a behavior of the vehicle; The acquisition unit is acquiring, as peripheral images of the vehicle, a reference peripheral image at a reference time, a first peripheral image at a first time after a predetermined time has elapsed from the reference time, and a second peripheral image at a second time after the predetermined time has elapsed from the first time; Acquire vehicle behavior information from the reference time to the first time and from the first time to the second time as information regarding the behavior of the vehicle, The identification unit is based on the vehicle behavior information, estimating a first estimated surrounding image after the predetermined time has elapsed since the reference surrounding image, and a second estimated surrounding image after the predetermined time has elapsed since the first surrounding image; An information processing device that identifies the moving object in the surrounding image of the vehicle based on a difference between the first surrounding image and the first estimated surrounding image and a difference between the second surrounding image and the second estimated surrounding image.

2. The information processing device according to claim 1 , wherein the information on the behavior of the vehicle is at least one of a speed of the vehicle and a steering angle of the vehicle.

3. The information processing device according to claim 1 , further comprising a transmission unit configured to transmit data relating to the moving object from among the data of the image of the surroundings of the vehicle to an outside.

4. A system including an information processing device mounted on a vehicle equipped with a sensor and a server capable of communicating with the information processing device, The information processing device includes: an acquisition unit that acquires an image of the surroundings of the vehicle and information regarding a behavior of the vehicle from the sensor; an identification unit that identifies a moving object in the surrounding image of the vehicle based on a time-dependent change in the surrounding image of the vehicle and information regarding a behavior of the vehicle; a transmission unit that transmits data related to the moving object among data of the surrounding image of the vehicle to the server, The acquisition unit is acquiring, as peripheral images of the vehicle, a reference peripheral image at a reference time, a first peripheral image at a first time after a predetermined time has elapsed from the reference time, and a second peripheral image at a second time after the predetermined time has elapsed from the first time; Acquire vehicle behavior information from the reference time to the first time and from the first time to the second time as information regarding the behavior of the vehicle, The identification unit is based on the vehicle behavior information, estimating a first estimated surrounding image after the predetermined time has elapsed since the reference surrounding image, and a second estimated surrounding image after the predetermined time has elapsed since the first surrounding image; identifying the moving object in the surroundings image of the vehicle based on a difference between the first surroundings image and the first estimated surroundings image and a difference between the second surroundings image and the second estimated surroundings image; The server, a receiving unit that receives data related to the moving object from the information processing device; A storage unit for storing data relating to structures included in the surrounding image of the vehicle; and an integration unit that integrates the data regarding the moving object and the data regarding the structure.

5. The system according to claim 4 , wherein the information about the behavior of the vehicle is at least one of a speed of the vehicle and a steering angle of the vehicle.

6. The system according to claim 4 or 5, which forms a digital twin in a virtual space that is time-synchronized with a real space using data that is an integrated combination of data regarding the moving object and data regarding the structure.

7. A method executed by a computer of an information processing device mounted on a vehicle equipped with a sensor, comprising: acquiring, from the sensor, a reference surrounding image at a reference time, a first surrounding image at a first time after a predetermined time has elapsed from the reference time, and a second surrounding image at a second time after the predetermined time has elapsed from the first time, as surrounding images of the vehicle; acquiring vehicle behavior information from the sensor as information regarding a behavior of the vehicle, the vehicle behavior information being from the reference time to the first time and from the first time to the second time; A step of estimating a first estimated surrounding image after the predetermined time has elapsed since the reference surrounding image and a second estimated surrounding image after the predetermined time has elapsed since the first surrounding image based on the vehicle behavior information; and identifying a moving object in the surroundings image of the vehicle based on a difference between the first surroundings image and the first estimated surroundings image and a difference between the second surroundings image and the second estimated surroundings image.

8. A program to be executed by a computer of an information processing device mounted on a vehicle equipped with a sensor, acquiring, from the sensor, a reference surrounding image at a reference time, a first surrounding image at a first time after a predetermined time has elapsed from the reference time, and a second surrounding image at a second time after the predetermined time has elapsed from the first time, as surrounding images of the vehicle; acquiring vehicle behavior information from the sensor as information regarding a behavior of the vehicle, the vehicle behavior information being from the reference time to the first time and from the first time to the second time; A step of estimating a first estimated surrounding image after the predetermined time has elapsed since the reference surrounding image and a second estimated surrounding image after the predetermined time has elapsed since the first surrounding image based on the vehicle behavior information; and identifying a moving object in the surroundings image of the vehicle based on a difference between the first surroundings image and the first estimated surroundings image and a difference between the second surroundings image and the second estimated surroundings image.

Images