IMAGE GENERATION DEVICE, IMAGE GENERATION METHOD, AND PROGRAM

Abstract

An image generation device (20) is provided with an acquisition unit (210), a data processing unit (220), and a display (230). The acquisition unit (210) repeatedly acquires analysis data from at least one transmission device (10). Analysis data include at least type data and relative position data. Each time analysis data are acquired, the data processing unit (220) generates a reconfigured image using the analysis data, and displays the reconfigured image on the display (230). The reconfigured image has an indication that is based on the type data at a position corresponding to the relative position data. This indication may be a mark resembling an outer shape of the type indicated by the type data, or may be an abstract mark.

Description

[Technical field]

[0001] The present invention relates to an image generating device, an image generating method, and a program. [Background technology]

[0002] In recent years, vehicles such as automobiles are increasingly being equipped with various sensors such as cameras. For example, Patent Document 1 describes a method in which a server acquires the detection results of these sensors from multiple vehicles including the vehicle itself, the server predicts the behavior of the vehicle itself and other vehicles, performs risk analysis using the prediction results, and visualizes the possibility of a collision using augmented reality. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-9428 A Summary of the Invention [Problem to be solved by the invention]

[0004] If the images generated by the imaging unit installed in the vehicle could be viewed at a monitoring center, the risk of traffic accidents could be reduced because the monitor could visually check the conditions around the vehicle. On the other hand, sending the images directly to the monitoring center would increase the amount of communication traffic.

[0005] One example of an object of the present invention is to enable a monitoring center to visually check the conditions around a vehicle while suppressing communication traffic. [Means for solving the problem]

[0006] According to the present invention, there is provided an acquisition means for repeatedly acquiring analysis data, the analysis data being a result of processing an image generated by an imaging means mounted on a vehicle, the analysis data including type data indicating a type of an object located around the vehicle and relative position data indicating a relative position of the object with respect to the vehicle; a data processing means for generating a reconstructed image having a display based on the type data at a position corresponding to the relative position data and displaying the reconstructed image on a display every time the acquisition means acquires the analysis data; An image generating device is provided comprising:

[0007] According to the present invention, a computer an acquisition process for repeatedly acquiring analysis data, the analysis data being a result of processing an image generated by an imaging means mounted on the vehicle, the analysis data including type data indicating a type of an object located around the vehicle and relative position data indicating a relative position of the object with respect to the vehicle; before data processing for generating a reconstructed image having a display based on the type data at a position corresponding to the relative position data and displaying the reconstructed image on a display every time the analysis data is acquired; An image generation method is provided that performs the above steps.

[0008] According to the present invention, a computer includes: an acquisition function for repeatedly acquiring analysis data, the analysis data being a result of processing an image generated by an imaging means mounted on the vehicle, the analysis data including type data indicating a type of an object located around the vehicle and relative position data indicating a relative position of the object with respect to the vehicle; before a data processing function for generating a reconstructed image having a display based on the type data at a position corresponding to the relative position data and displaying the reconstructed image on a display every time the analysis data is acquired; A program will be provided to help students develop these skills. Effect of the Invention

[0009] According to the present invention, the conditions around the vehicle can be visually confirmed at the monitoring center while suppressing communication traffic. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating a usage environment of an image generating device according to an embodiment. [Diagram 2] FIG. 2 illustrates an example of a functional configuration of a transmission device. [Diagram 3] FIG. 2 illustrates an example of a functional configuration of an image generating apparatus. [Figure 4] FIG. 2 is a diagram illustrating an example of a hardware configuration of a main part of a transmitting device. [Diagram 5] 4 is a flowchart showing a first example of processing performed by the image generating device. [Figure 6] FIG. 11 is a diagram showing a first example of a reconstructed image displayed on the display in step S50. [Figure 7] FIG. 11 is a diagram showing a second example of a reconstructed image displayed on the display in step S50. [Figure 8] 13 is a flowchart showing a second example of the process performed by the image generating device. [Figure 9] 13 is a flowchart showing a third example of a process performed by the image generating device. [Figure 10] 13 is a flowchart showing a fourth example of the process performed by the image generating device. [Figure 11] 13 is a flowchart showing a fifth example of a process performed by the image generating device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, the same components are given the same reference numerals and the description will be omitted as appropriate.

[0012] 1 is a diagram illustrating a usage environment of an image generating device 20 according to an embodiment. The image generating device 20 is an example of a data processing device, and is used together with a plurality of transmitting devices 10. The image generating device 20 is installed in a monitoring center. At the monitoring center, a monitor monitors, for example, a road and a vehicle 30. Here, the vehicle 30 may be an autonomous vehicle.

[0013] The transmitting device 10 is mounted on the vehicle 30, generates an image (photographed image) of the surroundings of the vehicle 30, for example, the front, and transmits the result of processing the image (hereinafter, described as analysis data) to the image generating device 20. The analysis data includes at least type data indicating the type of object located around the vehicle 30 (hereinafter, described as the first vehicle 30) on which the transmitting device 10 is mounted, and relative position data indicating the relative position of the object with respect to the first vehicle 30. Here, the object may be another vehicle 30 (hereinafter, described as the second vehicle 30), a pedestrian 40, a fallen object 50 on the road, or a traffic sign placed around the road or a road sign painted on the road. The image generating device 20 then generates a reconstructed image using the analysis data and displays it on a display. The position of the object in this reconstructed image corresponds to the position where the object exists in the real space. Therefore, a monitor can visually grasp the environment around the first vehicle 30 by looking at the reconstructed image.

[0014] Furthermore, the image generating device 20 requests the image itself from the transmitting device 10 as necessary. As an example, when a predetermined input is received from a user of the image generating device 20 (e.g., a monitor), the image generating device 20 requests an image from the transmitting device 10. The transmitting device 10 then transmits the image to the image generating device 20. In this case, the image generating device 20 displays the image generated by the transmitting device 10 on a display. This allows the user of the image generating device 20 to directly check the image generated by the transmitting device 10.

[0015] FIG. 2 is a diagram showing an example of the functional configuration of the transmission device 10. As described above, the transmission device 10 is mounted on a vehicle. In the example shown in this figure, the transmission device 10 includes an imaging unit 12, an image processing unit 14, and a communication unit 16. The imaging unit 12 is, for example, an in-vehicle camera, and repeatedly captures the surroundings of the first vehicle 30, for example, the surroundings of the first vehicle 30 (for example, at least one of the front, side, and rear). The imaging unit 12 may be a monocular camera or a stereo camera. The frame rate at this time is, for example, 10 frames / second or more, but is not limited to this. Every time the imaging unit 12 generates an image, the image processing unit 14 processes the image to generate the above-mentioned analysis data. Every time the image processing unit 14 generates analysis data, the communication unit 16 transmits the analysis data to the image generating device 20. In addition, when an image is requested by the image generating device 20, the communication unit 16 transmits the image generated by the imaging unit 12 to the image generating device 20.

[0016] As described above, the analysis data generated by the image processing unit 14 includes type data indicating the type of object located around the first vehicle 30, and relative position data indicating the relative position of the object with respect to the first vehicle 30. The analysis data may include other data as necessary.

[0017] For example, the analysis data may include data (hereinafter, referred to as road data) indicating the condition of the road located around (for example, at least one of the front, side, and rear) the first vehicle 30. Examples of the road condition include, but are not limited to, the width, the extension, and signs painted on the road.

[0018] Furthermore, when the object is a second vehicle 30, the analysis data may include relative speed data. The relative speed data indicates the relative speed between the first vehicle 30 and the second vehicle 30. The relative speed data is calculated, for example, using a change in the position of the second vehicle 30 in the image, but may be generated using a sensor (not shown).

[0019] The analysis data may be data indicating a difference from previously transmitted analysis data, for example, a difference from type data and relative position data indicated by previously transmitted analysis data. Here, the "previously transmitted analysis data" may be analysis data transmitted immediately before, or analysis data transmitted at a predetermined timing.

[0020] The communication unit 16 may also transmit, together with the analysis data, information for identifying the first vehicle 30 from other vehicles 30. The communication unit 16 may also transmit, together with the analysis data, other data related to the first vehicle 30. The other data includes, for example, at least one of data indicating the position of the first vehicle 30 (hereinafter referred to as vehicle position data) and data indicating the speed of the first vehicle 30 (hereinafter referred to as vehicle speed data). Here, the vehicle position data is generated, for example, using a GPS, and the vehicle speed data is generated using a speedometer mounted on the first vehicle 30.

[0021] FIG. 3 is a diagram showing an example of the functional configuration of the image generating device 20. In the example shown in this figure, the image generating device 20 includes an acquisition unit 210, a data processing unit 220, and a display 230. The acquisition unit 210 repeatedly acquires analysis data from at least one transmission device 10. As described above, the analysis data includes at least type data and relative position data. Each time the data processing unit 220 acquires analysis data, it generates a reconstructed image using the analysis data and displays the reconstructed image on the display 230. Here, the display 230 may be located outside the image generating device 20. In this case, the image generating device 20 can be realized by a cloud server, and the display 230 can be placed in a monitoring center. The reconstructed image has a display based on the type data at a position corresponding to the relative position data. This display may be a mark imitating the outer shape of the type indicated by the type data, or may be an abstracted mark.

[0022] Furthermore, if the analysis data includes the road data described above, the data processing unit 220 may include a display of the road according to the road data in the reconstructed image. In this case, the data processing unit 220 reproduces the road on which the first vehicle 30 is traveling in the reconstructed image, and also reproduces objects located around the first vehicle 30 in the reconstructed image. In other words, the reconstructed image is an image that reproduces the surroundings of the first vehicle 30.

[0023] Furthermore, when the analysis data includes the above-mentioned relative speed data and the acquisition unit 210 acquires vehicle speed data together with the analysis data, the data processing unit 220 may estimate the speed of the second vehicle 30 using the vehicle speed data and the relative speed data, and may include a display showing the estimation result in the reconstructed image or may display the same together with the reconstructed image. This estimation result may be displayed, for example, near the second vehicle 30 that is the estimation target, or may be displayed in a list.

[0024] The data processing unit 220 may use information stored in the map data storage unit 222 when generating the reconstructed image. The map data storage unit 222 stores map data in association with position information. The acquisition unit 210 acquires the vehicle position data described above together with the analysis data. The data processing unit 220 then acquires map data including points corresponding to the vehicle position data from the map data storage unit 222. The map data includes at least the width and shape of roads. The data processing unit 220 then includes roads based on the map data in the reconstructed image. These roads are at least a reproduction of the roads on which the vehicle 30 is traveling.

[0025] The map data storage unit 222 may be a part of the image generation device 20 or may be located outside the image generation device 20 .

[0026] Furthermore, when the criteria are met, the data processing unit 220 may request the image generated by the imaging unit 12 from the transmitting device 10. In this case, the data processing unit 220 causes the image acquired from the transmitting device 10 to be displayed on the display 230. Note that this criterion may be determined, for example, with respect to the analysis data, or may be determined with respect to the input to the image generating device 20 by the user (surveillance officer). Specific examples of this criterion will be described later with reference to other figures.

[0027] 4 is a diagram showing an example of a hardware configuration of the main parts of the transmission device 10. The transmission device 10 includes a bus 1010, a processor 1020, a memory 1030, a storage device 1040, an input / output interface 1050, and a network interface 1060.

[0028] The bus 1010 is a data transmission path for transmitting and receiving data among the processor 1020, memory 1030, storage device 1040, input / output interface 1050, and network interface 1060. However, the method of connecting the processor 1020 and the like to each other is not limited to bus connection.

[0029] The processor 1020 is implemented by a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like.

[0030] The memory 1030 is a main storage device realized by a RAM (Random Access Memory) or the like.

[0031] The storage device 1040 is an auxiliary storage device realized by a hard disk drive (HDD), a solid state drive (SSD), a memory card, a read only memory (ROM), etc. The storage device 1040 stores program modules that realize each function of the transmission device 10 (e.g., the image processing unit 14 and the communication unit 16). The processor 1020 loads each of these program modules onto the memory 1030 and executes them, thereby realizing each function corresponding to the program module.

[0032] The input / output interface 1050 is an interface for connecting the main part of the transmission device 10 to various input / output devices. For example, the main part of the transmission device 10 communicates with the imaging unit 12 via the input / output interface 1050.

[0033] The network interface 1060 is an interface for connecting the transmission device 10 to a network. This network is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network). The network interface 1060 may connect to the network wirelessly or by wire. The transmission device 10 communicates with the image generation device 20 via the network interface 1060.

[0034] An example of the hardware configuration of the image generating device 20 is also as shown in Fig. 4. In this case, the storage device 1040 stores a program module that realizes the functions of the image generating device 20 (e.g., the acquisition unit 210 and the data processing unit 220). The storage device 1040 also functions as a map data storage unit 222.

[0035] 5 is a flowchart showing a first example of processing performed by the image generating device 20 together with processing performed by the transmitting device 10. The transmitting device 10 and the image generating device 20 perform the processing shown in this figure every time the imaging unit 12 of the transmitting device 10 generates an image.

[0036] When the imaging unit 12 generates an image (step S10), the image processing unit 14 of the transmitting device 10 processes the image to generate analysis data (step S20). Next, the communication unit 16 of the transmitting device 10 transmits the analysis data generated in step S20 to the image generating device 20. At this time, the communication unit 16 transmits the relative speed data and the vehicle speed data of the first vehicle 30 together with the analysis data (step S30).

[0037] The acquisition unit 210 of the image generating device 20 acquires the data transmitted from the transmission device 10. Then, the data processing unit 220 of the image generating device 20 generates a reconstructed image using the data acquired by the acquisition unit 210 (step S40). Re The constituent images are displayed on the display 230 (step S50).

[0038] The processing shown after step S20 may be performed only on a part of the image generated by the imaging unit 12. For example, in step S10, the imaging unit 12 may capture images at a normal video frame rate (e.g., 24 frames / second or higher), and the processing after step S20 may be performed at a frame rate lower than that of the imaging unit 12 (e.g., 12 frames / second).

[0039] Furthermore, the frequency with which the processes shown after step S20 are performed may vary according to the speed of the first vehicle 30. As an example, the frequency increases as the speed of the first vehicle 30 increases. In this way, the load on the transmitting device 10 and the image generating device 20 decreases when the first vehicle 30 is traveling at a low speed.

[0040] In step S30, the communication unit 16 may transmit only a part of the analysis data generated in step S20 to the image generating device 20. For example, the communication unit 16 may transmit only data related to the second vehicle 30 and traffic signs to the image generating device 20. In this case, the data processing unit 220 of the image generating device 20 requests all the analysis data from the transmitting device 10 as necessary. One example is a case where a specified input is received from a monitor. Then, the communication unit 16 of the transmitting device 10 thereafter transmits all the analysis data (for example, data related to fallen objects 50 on the road) to the image generating device 20. In this way, the amount of communication between the transmitting device 10 and the image generating device 20 is reduced.

[0041] FIG. 6 shows a first example of a reconstructed image displayed on the display 230 in step S50. In the example shown in this figure, the data processing unit 220 generates an image of the first vehicle 30 viewed from the first vehicle 30 as the reconstructed image. As an example, the reconstructed image is an image viewed from the driver's seat of the first vehicle 30. For example, if the first vehicle 30 is traveling on a road with multiple lanes in one direction, the reconstructed image displays the second vehicle 30 and a traffic sign located in front of the first vehicle 30 (including diagonally in front). In addition, if there is a fallen object 50 on the road, the fallen object 50 is also displayed in the reconstructed image. In addition, if there is a pedestrian around the first vehicle 30, the pedestrian is also displayed in the reconstructed image.

[0042] In the example shown in the figure, the reconstructed image also displays the speed of the second vehicle 30. This speed is calculated using the relative speed data and the vehicle speed data.

[0043] Fig. 7 shows a second example of the reconstructed image displayed on the display 230 in step S50. In the example shown in this figure, the reconstructed image includes a bird's-eye view in addition to the image shown in Fig. 6. The reconstructed image may be only the bird's-eye view. When the analysis data relates to a plurality of directions of the first vehicle 30, it becomes easier to generate a bird's-eye view as the reconstructed image.

[0044] Fig. 8 is a flowchart showing a second example of the process performed by the image generating device 20 together with the process performed by the transmitting device 10. The process shown in this figure is similar to the process shown in Fig. 5, except that map data is used when generating a reconstructed image. In the example shown in this figure, the analysis data does not have to include road data.

[0045] After step S20, the communication unit 16 of the transmission device 10 transmits the vehicle position data together with the analysis data to the image generation device 20. At this time, the communication unit 16 transmits the relative speed data and the vehicle speed data of the first vehicle 30 together with the analysis data (step S32).

[0046] When the acquisition unit 210 of the image generating device 20 acquires the data transmitted from the transmitting device 10, the data processing unit 220 reads out map data including the point indicated by the vehicle position data from the map data storage unit 222 (step S34), generates a reconstructed image using this map data (step S40), and displays the generated reconstructed image on the display 230 (step S50).

[0047] When map data is used, it becomes unnecessary for the image processing unit 14 of the transmission device 10 to generate road data, and therefore the processing load on the image processing unit 14 is reduced.

[0048] The data processing unit 220 may generate the first reconstructed image by the method shown in Fig. 5 and may generate the second reconstructed image by the method shown in this figure. In this case, the data processing unit 220 may display the first reconstructed image and the second reconstructed image on the display 230 in a comparable state. For example, the data processing unit 220 may display the first reconstructed image and the second reconstructed image side by side on the display 230, or may display the first reconstructed image and the second reconstructed image superimposed on the display 230. In this way, the observer can visually recognize, for example, a difference between the map data and the road data generated by the image processing unit 14 of the transmitting device 10 (for example, an abnormality occurring on the road).

[0049] Fig. 9 is a flowchart showing a third example of the process performed by the image generating device 20. In the process shown in this figure, the transmitting device 10 is mounted on a plurality of vehicles 30. The transmitting device 10 performs the process shown in Fig. 5 or Fig. 8.

[0050] The acquisition unit 210 of the image generating device 20 acquires the analysis data, the vehicle position data, the vehicle speed data, and the relative speed data from the multiple transmitting devices 10 (step S110).

[0051] Next, the data processing unit 220 acquires information for identifying a target vehicle 30 (corresponding to the above-mentioned first vehicle 30) from the multiple vehicles 30. This acquisition may be performed, for example, by input from an observer. Then, the data processing unit 220 uses the vehicle position data to identify a vehicle 30 located near the first vehicle 30 as a second vehicle 30. As an example, the data processing unit 220 acquires vehicle position data corresponding to the first vehicle 30, identifies at least one other vehicle position data whose relevance to the vehicle position data (for example, direction and distance) satisfies a criterion, and determines the vehicle 30 corresponding to this vehicle position data as the second vehicle 30. Here, when multiple vehicles 30 are identified, the data processing unit 220 determines these multiple vehicles 30 as the second vehicle 30 (step S120).

[0052] Next, the data processing unit 220 acquires analysis data corresponding to the first vehicle 30 (hereinafter referred to as the first analysis data) and selects analysis data corresponding to the second vehicle 30 (hereinafter referred to as the second analysis data) (step S130). Next, the data processing unit 220 determines whether there is a mismatch between the first analysis data and the second analysis data. As an example, the data processing unit 220 determines whether there is a mismatch between the type and position of the object indicated by the first analysis data and the type and position of the object indicated by the second analysis data (step S140).

[0053] For example, the data processing unit 220 identifies the position of each object using the position information of the first vehicle 30 and the first analysis data. Similarly, the data processing unit 220 identifies the position of each object using the position information of the second vehicle 30 and the second analysis data. The data processing unit 220 then determines whether or not there is a mismatch between these object-specific positions. One example of a mismatch is when an object exists in one analysis result but does not exist in the other analysis result. Another example of a mismatch is when the position of an object indicated by one analysis result differs from the position of an object indicated by the other analysis result by a reference value or more.

[0054] If there is a mismatch (step S140: Yes), the data processing unit 220 requests at least one of the transmitting device 10 of the first vehicle 30 and the transmitting device 10 of the second vehicle 30 to transmit an image (step S150). The transmitting device 10 then transmits the image generated by the imaging unit 12 to the image generating device 20 together with the analysis data or instead of the analysis data. The data processing unit 220 then causes the image to be displayed on the display 230. The data processing unit 220 may display this image alongside the reconstructed image.

[0055] Furthermore, data processing unit 220 identifies the position (i.e., the point) of the object where the mismatch occurs (step S160), generates a reconstructed image so as to include an indication of the identified point (step S170), and causes display 230 to display the generated reconstructed image (step S180). Other indications included in the reconstructed image are as shown in Fig. 6 or Fig. 7. Here, data processing unit 220 may perform a predetermined output, such as an alarm display, if a mismatch occurs.

[0056] On the other hand, if there is no mismatch in step S140 (step S140: No), data processing unit 220 generates a reconstructed image (step S170) and displays the generated reconstructed image on display 230 (step S180). The reconstructed image generated here is the same as the reconstructed image described above, except that it does not include an indication of the point where the mismatch occurs.

[0057] In either case, the data processing unit 220 generates a reconstructed image using the first analysis data and at least one of the second analysis data in step S170. For example, the data processing unit 220 identifies the position of each object using the position information of the first vehicle 30 and the first analysis data. Similarly, the data processing unit 220 identifies the position of each object using the position information of the second vehicle 30 and the second analysis data. The data processing unit 220 then generates a bird's-eye view using these identification results. In this way, the presence of an object in a range that cannot be covered by the first analysis data can be identified using the second analysis data, and a display showing the object can be included in the reconstructed image together with a display showing the object identified by the first analysis data.

[0058] Fig. 10 is a flowchart showing a fourth example of the process performed by the image generating device 20. The process shown in this figure is performed in parallel with the process shown in Fig. 5, Fig. 8, or Fig. 9 each time the image generating device 20 acquires analysis data.

[0059] First, the data processing unit 220 identifies the movement of each detected object using the analysis data transmitted from the transmitting device 10 of the first vehicle 30. For example, the data processing unit 220 identifies the movement of the object using the difference between the analysis data acquired this time and the analysis data acquired a little before (step S210). Then, the data processing unit 220 determines whether the movement of the object identified in step S210 satisfies a criterion set for each object (step S220).

[0060] For example, if the object is a pedestrian, the criterion is that the pedestrian is moving toward the roadway. If the object is a second vehicle 30, the criterion is that the relative position of the second vehicle 30 with respect to the first vehicle 30 or its change is judged to be abnormal. As a specific example, when the second vehicle 30 is an oncoming vehicle, it may be judged that the oncoming vehicle is moving at an impossible speed. The judgment of whether or not there is an abnormality is performed using, for example, a model generated by machine learning. If an error is included in the analysis data, the above-mentioned change in relative position may show a physically impossible behavior. In this case, the data processing unit 220 also judges there to be an abnormality.

[0061] If the criteria are met in step S220 (step S220: Yes), the data processing unit 220 requests the transmitting device 10 of the first vehicle 30 to transmit an image (step S230). The transmitting device 10 then transmits the image generated by the imaging unit 12 to the image generating device 20 together with or instead of the analysis data. The data processing unit 220 then causes the image to be displayed on the display 230. Note that the data processing unit 220 may display this image alongside the reconstructed image.

[0062] Note that the communication unit 16 of the transmission device 10 may determine whether the analysis data satisfies the criteria instead of the image generating device 20. An example of this determination is the process shown in steps S210 and S220 of FIG. 10. Another example of this determination is when the reliability of the analysis data (e.g., the score when an object is detected) does not satisfy the criteria. Then, when the analysis data satisfies the criteria, the communication unit 16 transmits the image generated by the imaging unit 12 to the image generating device 20 together with the analysis data or instead of the analysis data.

[0063] Fig. 11 is a flowchart showing a fifth example of the process performed by the image generating device 20. The process shown in this figure is performed in parallel with the process shown in Fig. 5, Fig. 8, or Fig. 9, and the process shown in Fig. 10.

[0064] In the example shown in this figure, the observer using the image generating device 20 is checking the reconstructed image. When the observer judges that it would be better to directly check the image generated by the imaging unit 12 of the transmitting device 10, he performs a predetermined input to the image generating device 20 (step S310: Yes). Such a case may be, for example, when an abnormality occurs in the movement of any of the vehicles 30, or when vehicles 30 in various orientations are included in the reconstructed image in a bird's-eye view. Examples of the former case include the presence of a fallen object 50 that cannot be captured by image processing, or when road construction is being carried out. Then, the data processing unit 220 requests the transmitting device 10 of the first vehicle 30 to transmit an image (step S320). The process performed thereafter is as described with reference to FIG. 10.

[0065] As described above, according to this embodiment, the transmitting device 10 transmits, instead of an image, analysis data, which is the result of analyzing the image, to the image generating device 20. The analysis data includes at least type data indicating the type of object located around the vehicle 30, and relative position data indicating the relative position of the object with respect to the vehicle 30. The data processing unit 220 of the image generating device 20 then generates a reconstructed image using this analysis data, and displays it on the display 230. This allows the monitor to confirm the objects present around the vehicle 30. Furthermore, the amount of communication between the transmitting device 10 and the image generating device 20 is less than when the transmitting device 10 transmits an image to the image generating device 20.

[0066] Furthermore, when the criteria are met, the image generating device 20 requests an image from the transmitting device 10. Then, the transmitting device 10 transmits the image to the image generating device 20. The data processing unit 220 of the image generating device 20 causes the image acquired from the transmitting device 10 to be displayed on the display 230. In this way, the image generating device 20 causes the image generated by the imaging unit 12 to be displayed on the display 230 when necessary. This can improve the quality of monitoring by the monitor.

[0067] Although the embodiments of the present invention have been described above with reference to the drawings, these are merely examples of the present invention, and various configurations other than those described above can also be adopted.

[0068] In addition, in the flowcharts used in the above description, multiple steps (processes) are described in order, but the order of execution of the steps performed in each embodiment is not limited to the order described. In each embodiment, the order of the steps shown in the figures can be changed to the extent that the content is not affected. In addition, the above-mentioned embodiments can be combined to the extent that the content is not contradictory.

[0069] A part or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes. 1. An acquisition means for repeatedly acquiring analysis data obtained as a result of processing an image generated by an imaging means mounted on a vehicle, the analysis data including type data indicating a type of an object located around the vehicle and relative position data indicating a relative position of the object with respect to the vehicle; a data processing means for generating a reconstructed image having a display based on the type data at a position corresponding to the relative position data and displaying the reconstructed image on a display every time the acquisition means acquires the analysis data; An image generating device comprising: 2. The image generating device according to claim 1, The analysis data further includes road data indicating a state of a road located around the vehicle, The data processing means is an image generating device for including in the reconstructed image a representation of the road in accordance with the road data. 3. In the image generating device according to the above 1 or 2, The acquisition means acquires vehicle position data indicating a position of the vehicle together with the analysis data, The data processing means acquires map data including points corresponding to the vehicle position data, and the image generating device includes roads based on the map data in the reconstructed image. 4. In the image generating device according to any one of the above items 1 to 3, The acquisition means acquires vehicle position data indicating a position of each of the plurality of vehicles together with the analysis data, When the data processing means acquires information specifying vehicle position data, Obtaining a first analysis data corresponding to one of the vehicle position data; Obtain at least one other vehicle location data whose association with the one vehicle location data satisfies a criterion, and obtain second analysis data corresponding to the at least one other vehicle location data; an image generating device that generates the reconstructed image using the first analysis data and at least one of the second analysis data; 5. The image generating device according to claim 4, The data processing means includes: An image generating device that performs a predetermined output when a mismatch occurs between the type and position of the object indicated by the first analysis data and the type and position of the object indicated by the second analysis data. 6. The image generating device according to claim 5, The image generating device, wherein the data processing means includes in the reconstructed image information indicating the position of the object where the mismatch occurs. 7. In the image generating device according to any one of claims 1 to 6, the object is another vehicle located around the vehicle, The analysis data further includes and Relative speed data indicating a relative speed of the other vehicle, The acquiring means further acquires vehicle speed data indicating a speed of the vehicle together with the analysis data, The data processing means further estimates the speed of the other vehicle using the vehicle speed data and the relative speed data, and displays an indication of the estimation result together with or within the reconstructed image. 8. In the image generating device according to any one of 1 to 7 above, The data processing means is an image generating device that generates, as the reconstructed image, an image viewed from the vehicle outside the vehicle. 9. The image generating device according to any one of claims 1 to 7, The data processing means is an image generating device that generates a bird's-eye view as the reconstructed image. 10. In the image generating device according to any one of claims 1 to 9, An image generating device in which the frequency with which the acquisition means acquires the analysis data varies according to the speed of the vehicle. 11. In the image generating device according to any one of claims 1 to 10, The analysis data indicates a difference between the type data and the relative position data indicated by the analysis data transmitted in the past and the type data and the relative position data indicated by the analysis data transmitted in the past. 12. In the image generating device according to any one of claims 1 to 11, the vehicle is an autonomous vehicle; The display is an image generating device located at a monitoring center that monitors the vehicle. 13. The computer: an acquisition process for repeatedly acquiring analysis data, the analysis data being a result of processing an image generated by an imaging means mounted on the vehicle, the analysis data including type data indicating a type of an object located around the vehicle and relative position data indicating a relative position of the object with respect to the vehicle; data processing for generating a reconstructed image having a display based on the type data at a position corresponding to the relative position data and displaying the reconstructed image on a display every time the analysis data is acquired; An image generation method that performs the above. 1 4. The image generating method according to claim 13, The analysis data further includes road data indicating a state of a road located around the vehicle, The computer teeth an image generating method for including in the reconstructed image a representation of the road according to the road data. 15. The image generating method according to claim 13 or 14, The computer includes: acquiring vehicle position data indicative of a position of the vehicle together with the analysis data; The image generating method includes acquiring map data including points corresponding to the vehicle position data, and including roads based on the map data in the reconstructed image. 16. In the image generating method according to any one of the above 13 to 15, The computer includes: acquiring vehicle position data indicating a position of each of the plurality of vehicles together with the analysis data; Upon obtaining information identifying vehicle location data, Obtaining a first analysis data corresponding to one of the vehicle position data; Obtain at least one other vehicle location data whose association with the one vehicle location data satisfies a criterion, and obtain second analysis data corresponding to the at least one other vehicle location data; An image generating method for generating the reconstructed image using the first analysis data and at least one of the second analysis data. 17. The image generating method according to claim 16, The computer includes: An image generating method that performs a specified output when there is a mismatch between the type and position of the object indicated by the first analysis data and the type and position of the object indicated by the second analysis data. 18. The image generating method according to claim 17, The image generating method, in which the computer includes in the reconstructed image information indicating the position of the object where the mismatch occurs. 19. In the image generating method according to any one of the above 13 to 18, the object is another vehicle located around the vehicle, The analysis data further includes and Relative speed data indicating a relative speed of the other vehicle, The computer includes: acquiring vehicle speed data indicative of a speed of the vehicle together with the analysis data; an image generating method for estimating a speed of the other vehicle using the vehicle speed data and the relative speed data, and displaying an indication of the estimation result together with or within the reconstructed image; 20. In the image generating method according to any one of the above 13 to 19, An image generating method in which the computer generates an image of the outside of the vehicle viewed from the vehicle as the reconstructed image. 21. In the image generating method according to any one of the above items 13 to 19, The image generating method in which the computer generates a bird's-eye view as the reconstructed image. 22. In the image generating method according to any one of the above 13 to 21, A method of generating an image, wherein the frequency with which the computer acquires the analysis data varies according to the speed of the vehicle. 23. In the image generating method according to any one of the above 13 to 22, The image generating method, wherein the analysis data indicates a difference between the type data and the relative position data indicated by the analysis data transmitted in the past. 24. In the image generating method according to any one of the above 13 to 23, the vehicle is an autonomous vehicle; The image generating method, wherein the display is located at a monitoring center that monitors the vehicle. 25. To the computer, an acquisition function for repeatedly acquiring analysis data, the analysis data being a result of processing an image generated by an imaging means mounted on the vehicle, the analysis data including type data indicating a type of an object located around the vehicle and relative position data indicating a relative position of the object with respect to the vehicle; a data processing function that generates a reconstructed image having a display based on the type data at a position corresponding to the relative position data and displays the reconstructed image on a display every time the acquisition function acquires the analysis data; A program that allows you to have 26. In the program according to 25 above, The analysis data further includes road data indicating a state of a road located around the vehicle, The data processing function is a program for including in the reconstructed image a representation of the road according to the road data. 27. In the program according to 25 or 26 above, The acquisition function acquires vehicle position data indicating a position of the vehicle together with the analysis data, The data processing function is a program for acquiring map data including points corresponding to the vehicle position data, and including roads based on the map data in the reconstructed image. 28. In the program according to any one of the above 25 to 27, The acquisition function acquires vehicle position data indicating a position of each of the plurality of vehicles together with the analysis data, When the data processing function obtains information identifying vehicle position data, Obtaining a first analysis data corresponding to one of the vehicle position data; Obtain at least one other vehicle location data whose association with the one vehicle location data satisfies a criterion, and obtain second analysis data corresponding to the at least one other vehicle location data; A program for generating the reconstructed image using the first analysis data and at least one of the second analysis data. 29. In the program according to 28 above, The data processing function includes: A program that performs a predetermined output when a mismatch occurs between the type and position of the object indicated by the first analysis data and the type and position of the object indicated by the second analysis data. 30. In the program according to claim 29, The data processing function is a program for including in the reconstructed image information indicating the position of the object where the mismatch occurs. 31. In the program according to any one of the above 25 to 30, the object is another vehicle located around the vehicle, The analysis data further includes and Relative speed data indicating a relative speed of the other vehicle, The acquisition function further acquires vehicle speed data indicating a speed of the vehicle together with the analysis data; The data processing function further includes a program for estimating the speed of the other vehicle using the vehicle speed data and the relative speed data, and displaying an indication of the estimation result together with or within the reconstructed image. 32. In the program according to any one of the above 25 to 31, The data processing function is a program that generates an image of the vehicle viewed from the vehicle outside as the reconstructed image. 33. In the program according to any one of the above 25 to 31, The data processing function is a program for generating a bird's-eye view as the reconstructed image. 34. In the program according to any one of the above 25 to 34, A program in which the frequency at which the acquisition function acquires the analysis data varies according to a speed of the vehicle. 35. In the program according to any one of the above 25 to 34, The analysis data indicates a difference between the type data and the relative position data indicated by the analysis data transmitted in the past.

[0070] This application claims priority based on Japanese Patent Application No. 2020-049868, filed on March 19, 2020, the disclosure of which is incorporated herein in its entirety. [Explanation of symbols]

[0071] 10 Transmitting device 12 Imaging unit 14 Image processing section 16 Communications Department 20 Image generating device 210 Acquisition Department 220 Data Processing Unit 222 Map data storage unit 230 Display

Claims

1. an acquisition means for repeatedly acquiring analysis data obtained as a result of processing an image generated by an imaging means mounted on the vehicle, the analysis data including type data indicating a type of an object located around the vehicle and relative position data indicating a relative position of the object with respect to the vehicle; a data processing means for generating a reconstructed image which reproduces the captured image by arranging a display based on the type data at a position corresponding to the relative position data each time the acquisition means acquires the analysis data, the reconstructed image including an image showing a state where the outside of the vehicle is seen from a driver's seat of the vehicle, and displaying the reconstructed image on a display; Equipped with The acquisition means includes: acquiring vehicle position data indicating a position of each of the plurality of vehicles together with the analysis data; The data processing means includes: Upon obtaining information identifying vehicle location data, acquiring a first analysis data corresponding to one of the vehicle position data; Obtaining at least one other vehicle position data whose association with the one vehicle position data satisfies a criterion, and obtaining second analysis data corresponding to the at least one other vehicle position data; generating the reconstructed image using the first analysis data and at least one of the second analysis data; performing a predetermined output when a mismatch occurs between the type and position of the object indicated by the first analysis data and the type and position of the object indicated by the second analysis data; Image generating device.

2. 2. The image generating device according to claim 1, The data processing means generates the reconstructed image, which further includes a bird's-eye view of the vehicle from above, and displays the reconstructed image on a display. Image generating device.

3. 3. The image generating device according to claim 1, The analysis data further includes road data indicating a state of a road located around the vehicle, The data processing means is an image generating device for including in the reconstructed image a representation of the road in accordance with the road data.

4. The image generating device according to any one of claims 1 to 3, The acquisition means acquires vehicle position data indicating a position of the vehicle together with the analysis data, The data processing means acquires map data including points corresponding to the vehicle position data, and the image generating device includes roads based on the map data in the reconstructed image.

5. The image generating device according to any one of claims 1 to 4, The image generating device, wherein the data processing means includes in the reconstructed image information indicating the position of the object where the mismatch occurs.

6. The image generating device according to any one of claims 1 to 5, the object is another vehicle located around the vehicle, The analysis data further includes relative speed data indicating a relative speed between the vehicle and the other vehicle, The acquiring means further acquires vehicle speed data indicating a speed of the vehicle together with the analysis data, The data processing means further estimates the speed of the other vehicle using the vehicle speed data and the relative speed data, and displays an indication of the estimation result together with or within the reconstructed image.

7. The image generating device according to any one of claims 1 to 6, The data processing means is an image generating device that generates a bird's-eye view as the reconstructed image.

8. The computer an acquisition process for repeatedly acquiring analysis data, the analysis data being a result of processing an image generated by an imaging means mounted on the vehicle, the analysis data including type data indicating a type of an object located around the vehicle and relative position data indicating a relative position of the object with respect to the vehicle; a data processing for generating a reconstructed image, which is a pseudo-reproduction of the captured image by arranging a display based on the type data at a position corresponding to the relative position data each time the analysis data is acquired, the reconstructed image including an image showing a state of looking outside the vehicle from a driver's seat of the vehicle, and displaying the reconstructed image on a display; Do the following: The computer, In the acquisition process, vehicle position data indicating a position of each of the plurality of vehicles is acquired together with the analysis data, In the data processing, when information for identifying vehicle position data is acquired, acquiring a first analysis data corresponding to one of the vehicle position data; Obtaining at least one other vehicle position data whose association with the one vehicle position data satisfies a criterion, and obtaining second analysis data corresponding to the at least one other vehicle position data; generating the reconstructed image using the first analysis data and at least one of the second analysis data; performing a predetermined output when a mismatch occurs between the type and position of the object indicated by the first analysis data and the type and position of the object indicated by the second analysis data; The image generating method includes:

9. On the computer, an acquisition function for repeatedly acquiring analysis data, the analysis data being a result of processing an image generated by an imaging means mounted on the vehicle, the analysis data including type data indicating a type of an object located around the vehicle and relative position data indicating a relative position of the object with respect to the vehicle; a data processing function for generating a reconstructed image that reproduces the captured image by arranging a display based on the type data at a position corresponding to the relative position data each time the acquisition function acquires the analysis data, the reconstructed image including an image showing a state of looking outside the vehicle from a driver's seat of the vehicle, and displaying the reconstructed image on a display; Let them have The acquisition function includes: acquiring vehicle position data indicating a position of each of the plurality of vehicles together with the analysis data; The data processing function includes: Upon obtaining information identifying vehicle location data, acquiring a first analysis data corresponding to one of the vehicle position data; Obtaining at least one other vehicle position data whose association with the one vehicle position data satisfies a criterion, and obtaining second analysis data corresponding to the at least one other vehicle position data; generating the reconstructed image using the first analysis data and at least one of the second analysis data; performing a predetermined output when a mismatch occurs between the type and position of the object indicated by the first analysis data and the type and position of the object indicated by the second analysis data; program.

Images