Information processing device, information processing method, and program

Abstract

In order to accurately determine whether or not equipment included as a subject in an image is blurred, an information processing device according to the present invention comprises: an acquisition unit that acquires a first vehicle surrounding image captured by a first camera and a second vehicle surrounding image captured by a second camera; an identification unit that identifies the same equipment included as a subject in the two images; a calculation unit that calculates respective distances from the two cameras to the same equipment; and a determination unit that determines whether or not a partial image including the same equipment is blurred.

Description

Information processing device, information processing method, and program

[0001] The present invention relates to an information processing device, an information processing method, and a program that use images of facilities around a vehicle.

[0002] There is a demand for technology that can analyze the condition of equipment around a vehicle using images of the equipment.

[0003] Patent Document 1 discloses a crossover facility monitoring device that extracts overhead wires and overhead contact line fittings by performing edge detection processing on first image data and second image data captured by two line cameras installed on a railway vehicle. The crossover facility monitoring device determines whether overhead contact line fittings are located within a fitting attachment prohibited area based on the positions of the first and second trolley wires identified based on the extracted overhead wires and overhead contact line fittings.

[0004] Japanese Patent Application Publication No. 2016-168880

[0005] When analyzing the condition of equipment using an image of the equipment, the accuracy of analyzing the condition decreases if the equipment is blurred in the image. Therefore, when analyzing the condition of equipment using an image of the equipment, it is preferable to use an image in which the equipment is not blurred.

[0006] However, the crossover equipment monitoring device described in Patent Document 1 determines whether an overhead contact wire fitting is located within the fitting prohibition zone regardless of whether the overhead contact wire and overhead contact wire fitting are blurred in the captured image. Therefore, when the overhead contact wire and overhead contact wire fitting are blurred in the captured image, the crossover equipment monitoring device has a problem in that the accuracy of determining whether the overhead contact wire fitting is located within the fitting prohibition zone decreases. To solve this problem, a technology is needed that can accurately determine whether equipment included as a subject in an image is blurred. Furthermore, if such a technology were to be realized, it would contribute to improving the analysis accuracy not only of the crossover equipment monitoring device described in Patent Document 1, but also of general devices that analyze the condition of equipment using captured images of the equipment.

[0007] One aspect of the present invention has been made in consideration of the above-mentioned problems, and one example of its purpose is to provide a technology for accurately determining whether or not equipment included as a subject in an image is blurred.

[0008] An information processing device according to one aspect of the present invention comprises an acquisition means for acquiring a first vehicle surroundings image taken by a first camera and a second vehicle surroundings image taken by a second camera, the image capturing ranges of which partially overlap each other; an identification means for identifying the same equipment that is included as a subject in both the first vehicle surroundings image and the second vehicle surroundings image; a calculation means for calculating the distance between the first camera and the second camera, and the distance from the first camera and the second camera to the same equipment, respectively, by referring to the first vehicle surroundings image and the second vehicle surroundings image; and a determination means for determining whether a partial image including the same equipment in the first vehicle surroundings image is blurred by referring to the distance from the first camera to the same equipment and focus information of the first camera, and for determining whether a partial image including the same equipment in the second vehicle surroundings image is blurred by referring to the distance from the second camera to the same equipment and focus information of the second camera.

[0009] An information processing method according to one aspect of the present invention includes an information processing device acquiring a first vehicle surroundings image taken by a first camera and a second vehicle surroundings image taken by a second camera, the image capturing ranges of which partially overlap each other; identifying the same equipment that is included as a subject in both the first vehicle surroundings image and the second vehicle surroundings image; calculating the distance between the first camera and the second camera, and the distance from the first camera and the second camera to the same equipment, respectively, by referring to the distance from the first camera to the same equipment and focus information of the first camera; determining whether a partial image including the same equipment in the first vehicle surroundings image is blurred by referring to the distance from the first camera to the same equipment and focus information of the first camera; and determining whether a partial image including the same equipment in the second vehicle surroundings image is blurred by referring to the distance from the second camera to the same equipment and focus information of the second camera.

[0010] A program according to one aspect of the present invention is a program for causing a computer to function as an information processing device, and causes the computer to function as an acquisition means for acquiring a first vehicle surroundings image taken by a first camera and a second vehicle surroundings image taken by a second camera, the image capturing ranges of which partially overlap each other; an identification means for identifying the same equipment that is included as a subject in both the first vehicle surroundings image and the second vehicle surroundings image; a calculation means for calculating the distance between the first camera and the second camera, and the distances from the first camera and the second camera to the same equipment, respectively, by referring to the first vehicle surroundings image and the second vehicle surroundings image; and a determination means for determining whether a partial image including the same equipment in the first vehicle surroundings image is blurred by referring to the distance from the first camera to the same equipment and focus information of the first camera, and for determining whether a partial image including the same equipment in the second vehicle surroundings image is blurred by referring to the distance from the second camera to the same equipment and focus information of the second camera.

[0011] According to one aspect of the present invention, it is possible to determine with high accuracy whether or not equipment included as a subject in an image is blurred.

[0012] FIG. 1 is a block diagram showing the configuration of an information processing device according to a first exemplary embodiment of the present invention. FIG. 2 is a flow diagram showing the flow of an information processing method according to the first exemplary embodiment of the present invention. FIG. 3 is a table showing an example of equipment and states in a second exemplary embodiment of the present invention. FIG. 4 is a schematic diagram showing an example of a railway vehicle and equipment in a second exemplary embodiment of the present invention. FIG. 5 is a block diagram showing the configuration of an information processing device according to a second exemplary embodiment of the present invention. FIG. 6 is a schematic diagram of a plane including two base points and a reference point in the principle of triangulation in the second exemplary embodiment of the present invention. FIG. 7 is a flow diagram showing the flow of an information processing method according to the second exemplary embodiment of the present invention. FIG. 8 is a diagram showing an example of processing performed by a determination unit according to the second exemplary embodiment of the present invention. FIG. 9 is a diagram showing another example of processing performed by a determination unit according to the second exemplary embodiment of the present invention. FIG. 10 is a diagram showing yet another example of processing performed by a determination unit according to the second exemplary embodiment of the present invention. FIG. 11 is a block diagram showing an example of the hardware configuration of an information processing device according to each exemplary embodiment of the present invention.

[0013] [First Exemplary Embodiment] A first exemplary embodiment of the present invention will be described in detail with reference to the drawings. This exemplary embodiment is a basic form of the exemplary embodiments described below.

[0014] (Configuration of Information Processing Apparatus 1) The configuration of the information processing apparatus 1 according to this exemplary embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the information processing apparatus 1 according to this exemplary embodiment.

[0015] 1, the information processing device 1 according to this exemplary embodiment includes an acquisition unit 11, an identification unit 12, a calculation unit 13, and a determination unit 14. The acquisition unit 11, the identification unit 12, the calculation unit 13, and the determination unit 14 are components that respectively realize an acquisition means, an identification means, a calculation means, and a determination means in this exemplary embodiment.

[0016] The acquisition unit 11 acquires a first vehicle surroundings image captured by a first camera and a second vehicle surroundings image captured by a second camera, the capturing ranges of which partially overlap each other, and supplies the acquired first vehicle surroundings image and second vehicle surroundings image to the identification unit 12, the calculation unit 13, and the determination unit 14.

[0017] The identification unit 12 identifies the same facility that is included as a subject in both the first vehicle surroundings image and the second vehicle surroundings image supplied from the acquisition unit 11. The identification unit 12 supplies information indicating the identified same facility to the calculation unit 13 and the determination unit 14.

[0018] The calculation unit 13 refers to the distance between the first camera and the second camera, and the first vehicle surroundings image and the second vehicle surroundings image supplied from the acquisition unit 11, and calculates the distance from the first camera and the second camera to the same facility identified by the identification unit 12. The calculation unit 13 supplies information indicating the calculated distances to the determination unit 14.

[0019] The determination unit 14 determines whether or not a partial image including the same equipment in the first vehicle surroundings image supplied from the acquisition unit 11 is blurred by referring to the distance calculated by the calculation unit 13, which is the distance from the first camera to the same equipment identified by the identification unit 12, and the focus information of the first camera, and also determines whether or not a partial image including the same equipment in the second vehicle surroundings image supplied from the acquisition unit 11 is blurred by referring to the distance calculated by the calculation unit 13, which is the distance from the second camera to the same equipment identified by the identification unit 12, and the focus information of the second camera.

[0020] As described above, the information processing device 1 according to this exemplary embodiment includes the acquisition unit 11 that acquires the first vehicle surroundings image captured by the first camera and the second vehicle surroundings image captured by the second camera, the image capturing ranges of which partially overlap each other, the identification unit 12 that identifies the same facility that is included as a subject in both the first vehicle surroundings image and the second vehicle surroundings image supplied from the acquisition unit 11, and the device that calculates the distances from the first camera and the second camera to the same facility identified by the identification unit 12, with reference to the distance between the first camera and the second camera and the first vehicle surroundings image and the second vehicle surroundings image supplied from the acquisition unit 11. The configuration includes a calculation unit 13, and a determination unit 14 that determines whether a partial image including the same equipment in the first vehicle surroundings image supplied from the acquisition unit 11 is blurred by referring to the distance calculated by the calculation unit 13, which is the distance from the first camera to the same equipment identified by the identification unit 12, and the focus information of the first camera, and also determines whether a partial image including the same equipment in the second vehicle surroundings image supplied from the acquisition unit 11 is blurred by referring to the distance calculated by the calculation unit 13, which is the distance from the second camera to the same equipment identified by the identification unit 12, and the focus information of the second camera.

[0021] Therefore, the information processing device 1 according to this exemplary embodiment has the effect of being able to determine with high accuracy whether or not the equipment included as a subject in the image is blurred.

[0022] (Flow of Information Processing Method S1) The flow of the information processing method S1 according to this exemplary embodiment will be described with reference to Fig. 2. Fig. 2 is a flow chart showing the flow of the information processing method S1 according to this exemplary embodiment.

[0023] In step S11, the acquisition unit 11 acquires a first vehicle surroundings image captured by the first camera and a second vehicle surroundings image captured by the second camera, the capturing ranges of which partially overlap each other. The acquisition unit 11 supplies the acquired first vehicle surroundings image and second vehicle surroundings image to the identification unit 12, the calculation unit 13, and the determination unit 14.

[0024] (Step S12) In step S12, the identification unit 12 identifies the same facility that is included as a subject in both the first vehicle surroundings image and the second vehicle surroundings image supplied from the acquisition unit 11. The identification unit 12 supplies information indicating the identified same facility to the calculation unit 13 and the determination unit 14.

[0025] (Step S13) In step S13, the calculation unit 13 refers to the distance between the first camera and the second camera, and the first vehicle surroundings image and the second vehicle surroundings image supplied from the acquisition unit 11, and calculates the distance from the first camera and the distance from the second camera to the same facility identified by the identification unit 12. The calculation unit 13 supplies information indicating the calculated distances to the determination unit 14.

[0026] (Step S14) In step S14, the determination unit 14 determines whether or not a partial image including the same equipment in the first vehicle surroundings image supplied from the acquisition unit 11 is blurred, by referring to the distance calculated by the calculation unit 13, which is the distance from the first camera to the same equipment identified by the identification unit 12, and the focus information of the first camera, and also determines whether or not a partial image including the same equipment in the second vehicle surroundings image supplied from the acquisition unit 11 is blurred, by referring to the distance calculated by the calculation unit 13, which is the distance from the second camera to the same equipment identified by the identification unit 12, and the focus information of the second camera.

[0027] As described above, in the information processing method S1 according to this exemplary embodiment, the acquisition unit 11 acquires a first vehicle surroundings image taken by a first camera and a second vehicle surroundings image taken by a second camera, the image capturing ranges of which partially overlap each other, in step S11; the identification unit 12 identifies the same equipment that is included as a subject in both the first vehicle surroundings image and the second vehicle surroundings image supplied from the acquisition unit 11, in step S12; and the calculation unit 13 calculates the distances from the first camera and the second camera to the same equipment identified by the identification unit 12, by referring to the distance between the first camera and the second camera and the first vehicle surroundings image and the second vehicle surroundings image supplied from the acquisition unit 11. and a step S14 in which the determination unit 14 determines whether or not a partial image including the same equipment in the first vehicle surroundings image supplied from the acquisition unit 11 is blurred, by referring to the distance calculated by the calculation unit 13, which is the distance from the first camera to the same equipment identified by the identification unit 12, and the focus information of the first camera, and determines whether or not a partial image including the same equipment in the second vehicle surroundings image supplied from the acquisition unit 11 is blurred, by referring to the distance calculated by the calculation unit 13, which is the distance from the second camera to the same equipment identified by the identification unit 12, and the focus information of the second camera.

[0028] Therefore, according to the information processing method S1 according to this exemplary embodiment, the same effects as those of the information processing device 1 described above can be obtained.

[0029]

[0033] A second exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first exemplary embodiment are denoted by the same reference numerals, and their description will be omitted as appropriate.

[0030] (Overview of Information Processing Device 2) The information processing device 2 according to this exemplary embodiment is a device that analyzes the state of facilities around a vehicle. Although the vehicle is not particularly limited, this exemplary embodiment will describe a case where the information processing device 2 analyzes the state of facilities around a railway vehicle.

[0031] The term "surrounding equipment for railway vehicles" refers to equipment installed around railway vehicles to allow the vehicles to run. For example, the equipment is equipment for supplying power to railway vehicles. Examples of such equipment include wires, contact wires, hangers, ears, connectors, bolts, and insulators.

[0032] The state of the equipment is a state that can be determined from the appearance of the equipment. For example, the state of the equipment may be either a good state or a bad state. Furthermore, the good state and the bad state may differ depending on the equipment, and in this case, the equipment and the state may be defined in association with each other.

[0033] An example of a case where equipment and status are defined in association with each other will be described with reference to Fig. 3. Fig. 3 is a table showing an example of equipment and status in this exemplary embodiment.

[0034] 3, equipment and states are associated with each other. For example, equipment "W Year" is associated with the state "CC Misalignment" and the state "Bolt Dropped," and equipment "Protector" is associated with the state "Bolt Dropped."

[0035] The information processing device 2 analyzes whether the state of the equipment is in a state associated with the equipment based on the table of Fig. 3. As an example, the information processing device 2 analyzes whether the equipment "W Year" is in the state "CC Deviation" and whether the equipment "W Year" is in the state "Bolt Missing". As another example, the information processing device 2 analyzes whether the equipment "Protector" is in the state "Bolt Missing".

[0036] 3 indicates a state in which a malfunction has occurred in the equipment, so it can be said that the information processing device 2 analyzes whether or not the equipment is in a defective state.

[0037] Furthermore, in order to analyze the condition of the equipment, the information processing device 2 acquires a vehicle surroundings image, which is an image captured by a camera installed on an externally exposed surface of the railway vehicle and includes multiple pieces of equipment as subjects. The vehicle surroundings image may be an image captured while the vehicle is in motion. The externally exposed surface may include, for example, the top surface, bottom surface, left and right sides, and some or all of the front and rear sides of the railway vehicle. In this exemplary embodiment, the surface on which the camera is installed will be described mainly as the top surface of the railway vehicle, but is not limited to this. Hereinafter, "on the top surface of the railway vehicle" will also be simply referred to as "on the railway vehicle." The camera installed on the railway vehicle and the equipment as the subject will be described with reference to FIG. 4. FIG. 4 is a schematic diagram showing an example of a railway vehicle TR and equipment in this exemplary embodiment.

[0038] As shown in Fig. 4, a plurality of cameras (cameras CA1 to CA6) are installed on the railway vehicle TR, each capturing an image of equipment included in its angle of view and outputting the captured image. It is desirable that at least a portion of the ranges included in the angles of view of the plurality of cameras are different from each other. In other words, the range included in the angle of view of each camera (hereinafter also referred to as "capture range") may partially overlap with the range included in the angle of view of at least one other camera, but it is desirable that at least a portion of the ranges are different.

[0039] Although the installation manner of the multiple cameras is not particularly limited, one example is a configuration in which three cameras CA1 to CA3 are installed at different heights on the right side of the vehicle TR when the front of the vehicle TR is viewed from the direction of travel of the vehicle TR, as shown in Figure 4. Similarly, three cameras CA4 to CA6 are installed at different heights on the left side of the vehicle TR when the front of the vehicle TR is viewed from the direction of travel of the vehicle TR.

[0040] 4, the three cameras CA1 to CA3 are positioned to photograph the equipment, such as the hanger HA, ear EA, and trolley wire TW, from the left side of the vehicle's traveling direction. Similarly, the three cameras CA4 to CA6 are positioned to photograph the equipment, such as the hanger HA, ear EA, and trolley wire TW, from the right side of the vehicle's traveling direction.

[0041] Furthermore, the timing at which cameras CA1 to CA6 photograph the equipment is not particularly limited. One example is a configuration in which cameras CA1 to CA6 each photograph the equipment at a predetermined interval. Another example is a configuration in which multiple cameras photograph the equipment synchronously at a predetermined interval. One example of such a configuration is a configuration in which cameras CA1 and CA4 photograph the equipment synchronously, cameras CA2 and CA5 photograph the equipment synchronously, and cameras CA3 and CA6 photograph the equipment synchronously. However, due to processing delays and the like, images photographed by multiple cameras synchronously are not necessarily photographed at exactly the same time. Furthermore, as one example, as shown in FIG. 4, cameras CA1 to CA6 photograph the equipment, including a hanger HA, an ear EA, and a trolley wire TW, as their subjects.

[0042] The information processing device 2 acquires vehicle surroundings images captured by the cameras CA1 to CA6. The configuration in which the information processing device 2 acquires the vehicle surroundings images is not particularly limited. One example is a configuration in which the vehicle surroundings images captured by the cameras CA1 to CA6 are stored in a recording medium, and the information processing device 2 acquires the vehicle surroundings images from the recording medium. Another configuration is a configuration in which the information processing device 2 and the cameras CA1 to CA6 are communicably connected via a network, and the information processing device 2 acquires the vehicle surroundings images via the network.

[0043] The following description will focus on an example in which vehicle surroundings images captured by camera CA1 (first camera) and camera CA4 (second camera), among cameras CA1 to CA6, whose capturing ranges partially overlap each other, are referenced. As an example, the description will focus on a configuration in which cameras CA1 and CA4 are controlled to capture images of a subject in synchronization with each other. In the following description, the vehicle surroundings image captured by camera CA1 will be referred to as the "first vehicle surroundings image," and the vehicle surroundings image captured by camera CA4 will be referred to as the "second vehicle surroundings image." When no particular distinction is required, the vehicle surroundings images will simply be referred to as the "vehicle surroundings image."

[0044] (Configuration of Information Processing Device 2) The configuration of the information processing device 2 will be described with reference to Fig. 5. Fig. 5 is a block diagram showing the configuration of the information processing device 2 according to this exemplary embodiment.

[0045] As shown in FIG. 5, the information processing device 2 includes a control unit 20, an input / output unit 27, a communication unit 28, and a storage unit 29.

[0046] The input / output unit 27 is an interface that receives user input and outputs data. For example, the input / output unit 27 supplies information indicating the received user input to the control unit 20 and outputs information supplied from the control unit 20. Examples of the input / output unit 27 include, but are not limited to, a keyboard, a mouse, a touchpad, a microphone, and a liquid crystal display.

[0047] The communication unit 28 is an interface that transmits and receives data via a network. For example, the communication unit 28 transmits data supplied from the control unit 20 to other devices, and supplies data received from other devices to the control unit 20. Examples of the communication unit 28 include, but are not limited to, communication chips for various communication standards such as Ethernet (registered trademark), Wi-Fi (registered trademark), and wireless communication standards for mobile data communication networks, and USB-compliant connectors.

[0048] The storage unit 29 stores data referenced by the control unit 20. Examples of data stored in the storage unit 29 include, but are not limited to, vehicle surroundings images, partial images, and analysis results. Examples of the storage unit 29 include, but are not limited to, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0049] (Functions of control unit 20) The control unit 20 controls each component included in the information processing device 2. As shown in Fig. 5 , the control unit 20 also includes an acquisition unit 11, an identification unit 12, a calculation unit 13, a determination unit 14, an image connection unit 21, and an analysis unit 22. In this exemplary embodiment, the acquisition unit 11, the identification unit 12, the calculation unit 13, the determination unit 14, and the analysis unit 22 are components that respectively realize an acquisition means, an identification means, a calculation means, a determination means, and an analysis means.

[0050] (Acquisition Unit 11) The acquisition unit 11 acquires data supplied from the input / output unit 27 or the communication unit 28. The acquisition unit 11 also acquires data stored in the storage unit 29. As an example, the acquisition unit 11 acquires vehicle surroundings images from a plurality of cameras, cameras CA1 to CA6, via the input / output unit 27 or the communication unit 28. As another example, the acquisition unit 11 may acquire vehicle surroundings images captured by cameras CA1 to CA6 and stored in the storage unit 29.

[0051] (Identification unit 12) The identification unit 12 identifies that the subjects included in the multiple images are the same. As an example, the identification unit 12 first detects one or more pieces of equipment included as subjects in the first vehicle surroundings image (hereinafter referred to as "first equipment") and one or more pieces of equipment included as subjects in the second vehicle surroundings image (hereinafter referred to as "second equipment"). Next, the identification unit 12 identifies that the first equipment and the second equipment that satisfy a predetermined condition are the same equipment. Then, the identification unit 12 stores in the storage unit 29 a partial image obtained by extracting a portion of the first vehicle surroundings image that includes the same equipment. Hereinafter, the same equipment identified by the identification unit 12 will also be referred to as equipment EQ.

[0052] The method by which the identification unit 12 detects the first facility and the second facility is not limited. As an example, the identification unit 12 detects the facility using a region extraction model that is trained to receive an image of the vehicle's surroundings as input and output a region of the facility detected by region extraction (e.g., PWC). As another example, the identification unit 12 detects the facility using an object detection model that is trained to receive an image of the vehicle's surroundings as input and output a facility detected by object detection (e.g., SSD (Single Shot Multibox Detector)). As an example, the object detection model outputs an image of the vehicle's surroundings on which a detection frame that encloses the detected facility is superimposed. The object detection model also outputs facility information indicating the detected facility. In this exemplary embodiment, an example using the object detection model will be mainly described.

[0053] An example of the process in which the identification unit 12 identifies that the first equipment and the second equipment are the same equipment EQ will be described later.

[0054] (Calculation unit 13) The calculation unit 13 calculates the distance between the camera and the subject included in the image captured by the camera. As an example, the calculation unit 13 refers to the distance between the camera CA1 and the camera CA4, as well as the first vehicle surroundings image and the second vehicle surroundings image. Then, the calculation unit 13 calculates the distance from the camera CA1 and the camera CA4 to the same equipment EQ identified by the identification unit 12 and included as a subject in the first vehicle surroundings image and the second vehicle surroundings image. Hereinafter, the distance from the camera CA1 to the equipment EQ will be referred to as the first distance, and the distance from the camera CA4 to the equipment EQ will be referred to as the second distance. The first distance and the second distance are distances in a real three-dimensional space (hereinafter, referred to as real space).

[0055] For example, the calculation unit 13 calculates the first distance and the second distance using the principle of triangulation. As an example, the calculation unit 13 may calculate the first distance and the second distance based on the positions of the cameras CA1 and CA4 in real space, the positions of the areas including the equipment EQ in each of the first vehicle surroundings image and the second vehicle surroundings image, etc. As the positions of the cameras CA1 and CA4 in real space, for example, the center of the lens of the camera CA1 and the center of the lens of the camera CA4 may be used, but are not limited to this. Furthermore, as the position of the area including the equipment EQ in each of the first vehicle surroundings image and the second vehicle surroundings image, a point at the upper end (e.g., a predetermined point, a midpoint, etc.) or a point at the lower end (e.g., a predetermined point, a midpoint, etc.) of the detection frame of the equipment EQ may be used, but are not limited to this.

[0056] An example of a method in which the calculation unit 13 calculates the first distance and the second distance using the principle of triangulation will be described with reference to Fig. 6. Fig. 6 is a schematic diagram of a plane including two base points and a reference point in the principle of triangulation in this exemplary embodiment. Here, the position of camera CA1 in real space is set as the first base point, the position of camera CA4 is set as the second base point, and the position of equipment EQ is set as the reference point.

[0057] In Figure 6, distance AB is the distance between the first reference point and the second reference point. In other words, distance AB is the length of the line connecting the first reference point and the second reference point. Hereinafter, the line connecting the first reference point and the second reference point will also be referred to as the baseline.

[0058] In addition, the angle θ shown in FIG. １ The angle θ is the angle between the base line and the line connecting the first reference point and the reference point. ２ The angle θ is the angle between the base line and the line connecting the second base point and the reference point. １ °, and angle θ ２ Similarly, the calculation unit 13 calculates the first distance d1 from the first base point to the reference point by referring to the distance AB and the angle θ based on the principle of triangulation. １ °, and angle θ ２By referring to the angle θ, a second distance d2 (not shown) from the second base point to the reference point is calculated. １ °, and angle θ ２ A known technique can be applied to the calculation of the angle θ １ °, and angle θ ２ The degrees can be calculated by referring to the difference in coordinates of the reference points projected onto the first vehicle surroundings image and the second vehicle surroundings image, but is not limited to this.

[0059] (Determination unit 14) The determination unit 14 determines whether or not a subject included in an image is blurred. As an example, the determination unit 14 determines whether or not a partial image extracted from the vehicle surroundings image and including the equipment EQ identified by the identification unit 12 is blurred. The determination unit 14 stores the determination result in the storage unit 29. An example of the processing executed by the determination unit 14 will be described again with reference to FIG. 6 .

[0060] The determination unit 14 determines whether the partial image including the equipment EQ in the first vehicle surroundings image is blurred by referring to the first distance d1 and the focus information of the camera CA1. The focus information of the camera CA1 is information related to the focus set for the camera CA1. An example of the focus information of the camera CA1 is the distance from the camera CA1 at which the camera CA1 is in focus, but is not limited to this.

[0061] For example, as shown in Fig. 6 , the range of focal lengths indicating the range in which the camera CA1 is in focus is defined as a focal range f_r. In this case, the determination unit 14 determines whether the first distance d1 is included within the focal range f_r, as shown in Fig. 6 . If it is determined that the first distance d1 is included within the focal range f_r, the determination unit 14 determines that the partial image including the equipment EQ in the first vehicle surroundings image is not blurred. On the other hand, if it is determined that the first distance d1 is not included within the focal range f_r, the determination unit 14 determines that the partial image including the equipment EQ in the first vehicle surroundings image is blurred.

[0062] Similarly, the determination unit 14 determines whether or not a partial image including the equipment EQ in the second vehicle surroundings image is blurred by referring to the second distance d2 and the focus information of the camera CA4.

[0063] (Image connection unit 21) The image connection unit 21 connects images. As an example, the image connection unit 21 connects a plurality of vehicle surroundings images stored in the storage unit 29. The image connection unit 21 stores the connected vehicle surroundings images in the storage unit 29.

[0064] For example, the image connection unit 21 acquires from the storage unit 29 a plurality of first vehicle surroundings images captured by the camera CA1. Next, the image connection unit 21 sorts the acquired plurality of first vehicle surroundings images according to the date and time of capture. The image connection unit 21 then connects the rearranged plurality of first vehicle surroundings images so that equipment included as subjects in each of the plurality of first vehicle surroundings images are connected, thereby generating a connected first vehicle surroundings image. The image connection unit 21 performs similar processing on the plurality of vehicle surroundings images captured by the other cameras CA2 to CA6, respectively, to generate a connected vehicle surroundings image.

[0065] (Analysis unit 22) The analysis unit 22 analyzes the state of the equipment. As an example, the analysis unit 22 analyzes the state of the equipment EQ by referring to the partial image determined by the determination unit 14 to be non-blurred. As described above, the analysis unit 22 may determine whether the equipment EQ is poor. An example of the process by which the analysis unit 22 analyzes the state of the equipment EQ will be described later.

[0066] In this way, the analysis unit 22 analyzes the state of the equipment EQ included as a subject in the non-blurred partial image, and therefore can analyze the state of the equipment EQ with high accuracy.

[0067] (Flow of Information Processing Method S2) The flow of the information processing method S2 according to this exemplary embodiment will be described with reference to Fig. 7. Fig. 7 is a flow chart showing the flow of the information processing method S2 according to this exemplary embodiment.

[0068] (Step S21) In step S21, the acquisition unit 11 acquires the first vehicle surroundings image and the second vehicle surroundings image captured by the camera CA1 and the camera CA4, respectively. The acquisition unit 11 stores the acquired first vehicle surroundings image and the acquired second vehicle surroundings image in the storage unit 29.

[0069] (Step S22) In step S22, the image connection unit 21 acquires the plurality of first vehicle surroundings images and the plurality of second vehicle surroundings images stored in the storage unit 29. The image connection unit 21 also connects the acquired plurality of first vehicle surroundings images. The image connection unit 21 stores the connected first vehicle surroundings image in the storage unit 29. Similarly, the image connection unit 21 connects the acquired plurality of second vehicle surroundings images. The image connection unit 21 stores the connected second vehicle surroundings image in the storage unit 29.

[0070] (Step S23) In step S23, the identification unit 12 acquires the first-vehicle surroundings image after coupling and the second-vehicle surroundings image after coupling, which are stored in the storage unit 29. Then, the identification unit 12 identifies that the first equipment included as a subject in both of the first-vehicle surroundings images after coupling and the second equipment included as a subject in the first-vehicle surroundings image after coupling are the same equipment EQ. The identification unit 12 stores in the storage unit 29 a partial image obtained by extracting a portion including the first equipment identified as the same equipment EQ, a partial image obtained by extracting a portion including the second equipment, and equipment information indicating the equipment EQ.

[0071] An example of the process executed by the identification unit 12 in step S23 will be described later. Note that, in step S23, if the identification unit 12 cannot identify that the first equipment and the second equipment are the same equipment EQ, the information processing device 2 ends the execution of the information processing method S2.

[0072] (Step S24) In step S24, the calculation unit 13 calculates the distances (first distance and second distance) from the camera CA1 and the camera CA4 to the equipment EQ, respectively, by referring to the distance between the camera CA1 and the camera CA4, and the first vehicle surroundings image and the second vehicle surroundings image stored in the storage unit 29. The calculation unit 13 stores the calculated first distance and second distance in the storage unit 29 in association with the equipment EQ.

[0073] In step S25, the determination unit 14 determines whether a partial image including the equipment EQ is blurred in the first vehicle surroundings image, and determines whether a partial image including the equipment EQ is blurred in the second vehicle surroundings image. The determination unit 14 stores the determination results in the storage unit 29 in association with the partial images.

[0074] (Step S26) In step S26, the analysis unit 22 refers to the determination result stored in the storage unit 29 and determines whether or not the partial image has been determined to be blurred.

[0075] (Step S27) If it is determined in step S26 that the partial image is not blurred (step S26: NO), in step S27, the analysis unit 22 refers to the partial image and analyzes the state of the equipment EQ. An example of the process executed by the analysis unit 22 in step S27 will be described later.

[0076] In step S26, if it is determined that the partial image is blurred (step S26: YES), or after executing step S27, the information processing device 2 ends the execution of the information processing method S2.

[0077] (Processing Example 1 Executed by the Identifying Unit 12 in Step S23) An example of processing executed by the identifying unit 12 in step S23 will be described with reference to Fig. 8. Fig. 8 is a diagram illustrating an example of processing executed by the identifying unit 12 according to this exemplary embodiment.

[0078] In FIG. 8, the first vehicle surroundings image c_vp1 after linking includes the first equipment eq1 as a subject, and the second vehicle surroundings image c_vp2 after linking includes the second equipment eq2 as a subject.

[0079] The identification unit 12 detects the first equipment eq1 included as a subject in the first vehicle surroundings image c_vp1 after coupling, and the second equipment eq2 included as a subject in the second vehicle surroundings image c_vp2 after coupling.

[0080] The identification unit 12 also identifies a predetermined direction along the vehicle's traveling direction in each of the combined first vehicle surroundings image c_vp1 and the combined second vehicle surroundings image c_vp2. In Fig. 8 , the identification unit 12 sets the direction along the identified predetermined direction as the x-axis. The identification unit 12 also sets the direction perpendicular to the x-axis as the y-axis.

[0081] Next, the identification unit 12 calculates the positions in the x-axis direction of the region including the first equipment eq1 and the region including the second equipment eq2. As an example of the position of the region including the first equipment eq1, any point at the top end (e.g., the midpoint, the upper left end, the upper right end, etc.) or any point at the bottom end (e.g., the midpoint, the upper left end, the upper right end, etc.) of the detection frame of the first equipment eq1 in the first vehicle surroundings image may be used. The same applies to an example of the position of the region of the second equipment eq2.

[0082] As another example, the identifying unit 12 may set any corner of the detection frame to the position of the area including the first equipment eq1. As yet another example, the identifying unit 12 may set the center of rotational symmetry of the detection frame of the first equipment eq1 to the position of the area including the first equipment eq1. The same applies to other examples of the position of the area of ​​the second equipment eq2.

[0083] Next, the identification unit 12 determines whether the difference between the calculated position of the area including the first equipment eq1 and the position of the area including the second equipment eq2 is within a predetermined range. In other words, as shown in Fig. 8 , the identification unit 12 determines whether the position of the area including the second equipment eq2 is included within a predetermined range sr from the position of the area including the first equipment eq1 in the x-axis direction.

[0084] If it is determined that the difference between the location of the area including the first equipment eq1 and the location of the area including the second equipment eq2 is within a predetermined range, the identification unit 12 identifies the first equipment eq1 and the second equipment eq2 as the same equipment EQ. On the other hand, if it is determined that the difference between the location of the area including the first equipment eq1 and the location of the area including the second equipment eq2 is not within the predetermined range, the identification unit 12 identifies the first equipment eq1 and the second equipment eq2 as not being the same equipment EQ.

[0085] In this way, the identification unit 12 identifies the first equipment eq1 and the second equipment eq2 as the same equipment EQ when the difference in the position in the x-axis direction between the area including the first equipment eq1 and the area including the second equipment eq2 is within a predetermined range. For example, in the first vehicle surroundings image and the second vehicle surroundings image taken while the vehicle is traveling around a curve, the position of the area including the first equipment eq1 and the position of the area including the second equipment eq2 in the x-axis direction along the vehicle's traveling direction may not match. Even in such a case, the identification unit 12 can identify the first equipment eq1 and the second equipment eq2 as the same equipment EQ.

[0086] (Processing Example 2 Executed by the Identifying Unit 12 in Step S23) Another example of the processing executed by the identifying unit 12 in step S23 will be described with reference to Fig. 9. Fig. 9 is a diagram illustrating another example of the processing executed by the identifying unit 12 according to this exemplary embodiment.

[0087] 9 , the first vehicle surroundings image c_vp1 after linking includes the first equipment eq1_1 as a subject, and the second vehicle surroundings image c_vp2 after linking includes multiple second equipment (the second equipment eq2_1 and the second equipment eq2_2) as subjects. The process by which the identification unit 12 detects the first equipment eq1_1, the second equipment eq2_1, and the second equipment eq2_2 is the same as the process described above.

[0088] In this example, the identification unit 12 first identifies a predetermined direction along the vehicle's traveling direction in each of the combined first vehicle surroundings image c_vp1 and the combined second vehicle surroundings image c_vp2. In the figure, the identification unit 12 sets the direction along the identified predetermined direction as the x-axis. The identification unit 12 also sets the direction perpendicular to the x-axis as the y-axis.

[0089] Next, the identifying unit 12 calculates the positions in the x-axis direction of the regions that each include the first equipment eq1_1, the second equipment eq2_1, and the second equipment eq2_2. The process by which the identifying unit 12 calculates the positions in the x-axis direction of the regions that each include the first equipment eq1_1, the second equipment eq2_1, and the second equipment eq2_2 is the same as the process described above.

[0090] Next, the identification unit 12 identifies either the second equipment eq2_1 or the second equipment eq2_2, the second equipment eq2_1 whose position in the x-axis direction of the area including the second equipment eq2_1 and the second equipment eq2_2 is closest to the position in the x-axis direction of the area including the first equipment eq1_1, as the same equipment EQ.

[0091] For example, in Fig. 9, the difference in the x-axis direction between the position of the region including the first equipment eq1_1 and the position of the region including the second equipment eq2_1 is x1. Also in Fig. 9, the difference in the x-axis direction between the position of the region including the first equipment eq1_1 and the position of the region including the second equipment eq2_2 is x2. The identifying unit 12 identifies the smaller value of x1 and x2 (in this example, x1). Then, the identifying unit 12 identifies the second equipment eq2_1 corresponding to x1 and the first equipment eq1_1 as the same equipment EQ.

[0092] In this way, the identification unit 12 identifies, among the plurality of second facilities, the second facility eq2_1 that is closest to the position in the x-axis direction of the area including the first facility eq1_1, and the first facility eq1_1, as the same facility EQ. Therefore, even if a plurality of second facilities are included as subjects in the second vehicle surroundings image, the same facility EQ can be suitably identified.

[0093] (Processing Example 3 Executed by the Identifying Unit 12 in Step S23) Another example of the processing executed by the identifying unit 12 in step S23 will be described with reference to Fig. 10. Fig. 10 is a diagram illustrating yet another example of the processing executed by the identifying unit 12 according to this exemplary embodiment.

[0094] 10 , the first vehicle surroundings image c_vp1 after coupling includes the first equipment eq1_1 and the first equipment eq1_2 as subjects, and the second vehicle surroundings image c_vp2 after coupling includes the second equipment eq2_1, the second equipment eq2_2, and the second equipment eq2_3 as subjects. The identification unit 12 detects the first equipment eq1_1 and the first equipment eq1_2 included as subjects in the first vehicle surroundings image c_vp1 after coupling, and the second equipment eq2_1, the second equipment eq2_2, and the second equipment eq2_3 included as subjects in the second vehicle surroundings image c_vp2 after coupling.

[0095] In the processing example 3, the identification unit 12 selects any first facility in the first vehicle surroundings image as an identification target and performs a process of identifying the corresponding second facility, and then continues the identification process for unidentified first facilities and second facilities other than the identified first facilities and second facilities. In addition, the identification unit 12 performs a process of identifying second facilities that are the same as the first facility among a plurality of first facilities in the first vehicle surroundings image, in order from the smallest x-coordinate.

[0096] For example, in Figure 10, the identification unit 12 identifies the first equipment eq1_1 and the second equipment eq2_1 as the same equipment EQ1, and then performs a process of identifying the next first equipment eq1_2 and the second equipment eq2_3, which is the same equipment EQ2 as the first equipment eq1_2.

[0097] With this configuration, the identifying unit 12 can prevent different first equipment and second equipment from being identified as the same equipment EQ. Also, the identifying unit 12 can reduce the processing load.

[0098] Furthermore, in this configuration, the identifying unit 12 may identify one or more pieces of second equipment, among the plurality of pieces of second equipment, whose area including the second equipment is located within a predetermined range sr from the area including the first equipment eq1_1. In this case, the identifying unit 12 does not execute a process of identifying one or more pieces of second equipment whose area including the second equipment is not located within the predetermined range sr from the area including the first equipment eq1_1 as having the same equipment EQ as the first equipment eq1_1. In other words, the identifying unit 12 may execute a process of identifying one or more pieces of second equipment whose area including the second equipment is located within the predetermined range sr from the area including the first equipment eq1_1 as having the same equipment EQ as the first equipment eq1_1.

[0099] (Processing Example 4 Executed by Identifying Unit 12 in Step S23) Another example of the processing executed by the identifying unit 12 in step S23 will be described with reference to FIG. 11. FIG. 11 is a diagram illustrating another example of the processing executed by the identifying unit 12 according to this exemplary embodiment. In addition to Processing Example 3, Processing Example 4 includes processing that takes into consideration a mismatch between the positional relationship in the y-axis direction of the plurality of first facilities in the first vehicle surroundings image and the positional relationship in the second vehicle surroundings image of the plurality of second facilities.

[0100] For example, as shown in the upper part of Fig. 11, a case will be described in which equipment EQ3, equipment EQ4, and equipment EQ5 are installed above railway vehicle TR. In the upper part of Fig. 11, equipment EQ3, equipment EQ4, and equipment EQ5 are installed side by side and substantially parallel to the baselines of cameras CA1 and CA4.

[0101] 11 , among equipment EQ3, equipment EQ4, and equipment EQ5, the equipment closer to the camera is shown at the top of the image. For example, in the first vehicle surroundings image captured by camera CA1, equipment EQ3 is shown at the top, equipment EQ4 is shown below equipment EQ3, and equipment EQ5 is shown at the bottom. On the other hand, in the second vehicle surroundings image captured by camera CA4, equipment EQ5 is shown at the top, equipment EQ4 is shown below equipment EQ5, and equipment EQ3 is shown at the bottom.

[0102] The process by which the identification unit 12 identifies the same equipment EQ when multiple pieces of equipment are installed side by side substantially parallel to the baselines of the cameras CA1 and CA4 will be described below.

[0103] First, assume that the first vehicle surroundings image c_vp1 after linking includes one or more first facilities as subjects, and the second vehicle surroundings image c_vp2 after linking includes one or more second facilities as subjects. In this case, when the number of one or more first facilities is the same as the number of one or more second facilities, the identification unit 12 identifies a correspondence between the one or more first facilities and the one or more second facilities, and thereby identifies the first facilities and the second facilities identified as having a correspondence relationship as the same facility EQ.

[0104] With this configuration, the identification unit 12 can prevent different first equipment and second equipment from being identified as the same equipment EQ.

[0105] For example, as shown in the lower part of Fig. 11 , a case is assumed in which the first vehicle surroundings image c_vp1 after concatenation includes multiple first facilities (first facilities eq1_1, eq1_2, and eq1_3) as subjects. Similarly, a case is assumed in which the second vehicle surroundings image c_vp2 after concatenation includes multiple second facilities (second facilities eq2_1, eq2_2, and eq2_3) as subjects.

[0106] 11 , the number of first facilities included as subjects in the first vehicle surroundings image c_vp1 after linking is 3, and the number of second facilities included as subjects in the second vehicle surroundings image c_vp2 after linking is also 3, so the number of first facilities is the same as the number of second facilities. In this case, the identification unit 12 identifies a correspondence relationship between the first facilities eq1_1, eq1_2, and eq1_3 and the second facilities eq2_1, eq2_2, and eq2_3.

[0107] For example, the identification unit 12 first identifies a vertical direction perpendicular to the vehicle's traveling direction in each of the first concatenated vehicle surroundings image c_vp1 and the second concatenated vehicle surroundings image c_vp2. In the lower part of Fig. 11 , the direction along the identified vertical direction is set as the y-axis. In the lower part of Fig. 11 , the identification unit 12 also sets a direction perpendicular to the y-axis as the x-axis.

[0108] Next, the identification unit 12 identifies the correspondence between each of the multiple first facilities and each of the multiple second facilities, taking into account the vertical positional relationship in the y-axis direction, and thereby identifies the first facilities and second facilities that have been identified as having a correspondence as the same facilities.

[0109] For example, as described above, the first equipment eq1_1 having the largest y coordinate (appearing at the top) in the first vehicle surroundings image c_vp1 after concatenation is the same as the second equipment eq2_1 having the smallest y coordinate (appearing at the bottom) in the second vehicle surroundings image c_vp2 after concatenation. Therefore, the identification unit 12 identifies the first equipment eq1_1 and the second equipment eq2_1 as the same equipment EQ, taking into account their vertical positional relationship in the y-axis direction. Similarly, the identification unit 12 identifies the first equipment eq1_2 and the second equipment eq2_2 as the same equipment EQ, and identifies the first equipment eq1_3 and the second equipment eq2_3 as the same equipment EQ.

[0110] In this way, even if the vertical (y-axis direction) positional relationship of multiple pieces of equipment included as subjects in the first vehicle surroundings image and the second vehicle surroundings image is different in the first vehicle surroundings image and the second vehicle surroundings image, the identification unit 12 can identify the same equipment EQ.

[0111] The identification unit 12 may identify identical equipment according to the type of equipment in addition to the vertical positional relationship in the y-axis direction. For example, for equipment of a predetermined type, the identification unit 12 may identify identical equipment by matching the vertical positional relationship in the y-axis direction with the vertical relationship in the first vehicle surroundings image and the second vehicle surroundings image. On the other hand, for equipment other than the predetermined type, the identification unit 12 may identify identical equipment by reversing the vertical positional relationship in the y-axis direction with the vertical relationship in the first vehicle surroundings image and the second vehicle surroundings image as described above.

[0112] (Processing Example 1 Executed by the Analysis Unit 22 in Step S27) As an example, in step S27, the analysis unit 22 may analyze the state of the equipment using a quality determination model that receives a partial image as input and outputs information indicating whether the state of the equipment EQ included as a subject in the partial image is poor. The quality determination model is trained using a training dataset that includes information that pairs the partial image with the state of the equipment EQ included as a subject in the partial image. With this configuration, the analysis unit 22 can suitably analyze the state of the equipment.

[0113] (Processing Example 2 Executed by Analysis Unit 22 in Step S27) As another example, in step S27, the analysis unit 22 may determine the parameters used to analyze the state of the equipment EQ according to the distance from camera CA1 or camera CA4 to the equipment EQ. For example, when the partial image to be analyzed is included in the first vehicle surroundings image, the distance from camera CA1 is applied as the distance to the equipment EQ. Furthermore, for example, when the partial image to be analyzed is included in the second vehicle surroundings image, the distance from camera CA4 is applied as the distance to the equipment EQ.

[0114] In this configuration, the parameter may be a criterion for determining that the equipment EQ is poor. For example, the criterion for determining that the equipment EQ is poor may be a threshold for determining that the equipment EQ is poor. In the following, an example will be described in which a threshold is used as the criterion for determining that the equipment EQ is poor. In addition, in the following, the first threshold is a value smaller than the second threshold.

[0115] For example, assume that in a first vehicle surroundings image captured by camera CA1, equipment EQ1, which is determined to be defective if it is further away from equipment EQ2 by a predetermined distance or more, and equipment EQ2, the equipment EQ1 and the equipment EQ2 are captured at a distance D. Note that if the number of pixels in the first vehicle surroundings image is fixed, the number of pixels may be used instead of the distance D.

[0116] In this case, the analysis unit 22 refers to the first distance from the camera CA1 to the equipment EQ1 calculated by the calculation unit 13. Next, the analysis unit 22 determines whether the referred first distance is equal to or greater than a predetermined value. In other words, the analysis unit 22 determines whether the distance between the camera CA1 and the equipment EQ1 is equal to or greater than a predetermined distance.

[0117] If it is determined that the referenced first distance is equal to or greater than a predetermined value (if the camera CA1 and the equipment EQ1 are separated by a distance equal to or greater than the predetermined value), the analysis unit 22 sets the first threshold as the threshold. Then, the analysis unit 22 determines whether the distance D is equal to or greater than the first threshold. If it is determined that the distance D is equal to or greater than the first threshold, the analysis unit 22 determines that the equipment EQ1 is defective. On the other hand, if it is determined that the distance D is not equal to or greater than the first threshold, the analysis unit 22 determines that the equipment EQ1 is not defective.

[0118] On the other hand, if it is determined that the referenced first distance is not equal to or greater than the predetermined value (if the camera CA1 and the equipment EQ1 are closer than the predetermined distance), the analysis unit 22 sets a second threshold value that is greater than the first threshold value as the threshold value. Then, the analysis unit 22 determines whether the distance D is equal to or greater than the second threshold value. If it is determined that the distance D is equal to or greater than the second threshold value, the analysis unit 22 determines that the equipment EQ1 is defective. On the other hand, if it is determined that the distance D is not equal to or greater than the second threshold value, the analysis unit 22 determines that the equipment EQ1 is not defective.

[0119] In this way, the analysis unit 22 does not use the distance on the first vehicle surroundings image as it is, but performs processing to convert the distance on the first vehicle surroundings image into an actual distance to analyze the state of the equipment, thereby enabling the analysis unit 22 to analyze the state of the equipment with high accuracy.

[0120] (Processing Example 3 Executed by the Analysis Unit 22 in Step S27) As yet another example, in step S27, the analysis unit 22 may determine the parameters used to analyze the state of the equipment EQ based on the angle between a line connecting the position of the camera CA1 or camera CA4 and the position of the equipment EQ and a baseline connecting the positions of the cameras CA1 and CA4. For example, when the partial image to be analyzed is included in the first vehicle surroundings image, the angle formed by the line connecting the position of the camera CA1 and the position of the equipment EQ and the baseline is applied. Furthermore, when the partial image to be analyzed is included in the second vehicle surroundings image, the angle formed by the line connecting the position of the camera CA4 and the position of the equipment EQ and the baseline is applied.

[0121] In this configuration, the parameter may also be a criterion for determining that the equipment EQ is defective. For example, the criterion for determining that the equipment EQ is defective may be a threshold for determining that the equipment EQ is defective. In the following, an example will be described in which a threshold is used as the criterion for determining that the equipment EQ is defective. In addition, in the following, the third threshold is a value smaller than the fourth threshold.

[0122] As in the above example, assume that in the first vehicle surroundings image captured by camera CA1, equipment EQ1, which is determined to be defective if it is further away from equipment EQ2 by a predetermined distance or more, and equipment EQ2, the equipment EQ1 and the equipment EQ2 are captured at a distance D. Note that if the number of pixels in the first vehicle surroundings image is fixed, the number of pixels may be used instead of the distance D.

[0123] First, the analysis unit 22 calculates the angle θ between the base line and the line connecting the position of the camera CA1 and the equipment EQ1. ＥＱ１ The analysis unit 22 calculates the angle θ calculated by the calculation unit 13. ＥＱ１ The configuration may be such that the angle .theta.

[0124] Next, the analysis unit 22 calculates the calculated angle θ ＥＱ１ In other words, the analysis unit 22 determines whether the angle formed by the baseline, the camera CA1, and the equipment EQ1 is equal to or greater than a predetermined angle.

[0125] Calculated angle θ ＥＱ１ If it is determined that ° is equal to or greater than a predetermined value, the analysis unit 22 sets a third threshold as the threshold. Then, the analysis unit 22 determines whether the distance D is equal to or greater than the third threshold. If it is determined that the distance D is equal to or greater than the third threshold, the analysis unit 22 determines that the equipment EQ1 is defective. On the other hand, if it is determined that the distance D is not equal to or greater than the third threshold, the analysis unit 22 determines that the equipment EQ1 is not defective.

[0126] On the other hand, the calculated angle θ ＥＱ１ If it is determined that ° is not equal to or greater than the predetermined value, the analysis unit 22 sets a fourth threshold value that is greater than the third threshold value as the threshold value. Then, the analysis unit 22 determines whether the distance D is equal to or greater than the fourth threshold value. If it is determined that the distance D is equal to or greater than the fourth threshold value, the analysis unit 22 determines that the equipment EQ1 is defective. On the other hand, if it is determined that the distance D is not equal to or greater than the fourth threshold value, the analysis unit 22 determines that the equipment EQ1 is not defective.

[0127] In this way, the analysis unit 22 does not use the distance on the first vehicle surroundings image as it is, but performs processing to convert the distance on the first vehicle surroundings image into an actual distance according to the angle, thereby analyzing the state of the equipment, which allows the analysis unit 22 to analyze the state of the equipment with high accuracy.

[0128] (Processing Example 4 Executed by the Analysis Unit 22 in Step S27) As yet another example, in step S27, the analysis unit 22 may refer to the position of the equipment EQ in real space in order to analyze the state of the equipment EQ.

[0129] For example, assume that in a first vehicle surroundings image captured by camera CA1, equipment EQ1, which would be determined to be defective if it were not in contact with equipment EQ2, and equipment EQ2, are in contact with each other in the first vehicle surroundings image. In this case, even if equipment EQ1 and equipment EQ2 are in a defective state of not being in contact in real space, they may appear to be in contact depending on the viewing direction. Therefore, even if a first area including equipment EQ1 and a second area including equipment EQ2 are in contact in the first vehicle surroundings image (or the second vehicle surroundings image), this does not necessarily mean that equipment EQ1 and equipment EQ2 are in a good state of being in contact in real space.

[0130] Here, the analysis unit 22 calculates the positions of equipment EQ1 and equipment EQ2 in real space based on the principle of triangulation by referring to the positions of cameras CA1 and CA2 in real space, the position of the first region in the first vehicle surroundings image, and the position of the second region in the second vehicle surroundings image. For example, the analysis unit 22 may calculate the positions in real space of the endpoints of equipment EQ1 and equipment EQ2 that should be in contact with each other. The analysis unit 22 also calculates the distance between the calculated position in real space of the endpoint of equipment EQ1 and the position in real space of the endpoint of equipment EQ2. If the calculated distance is less than a threshold, the analysis unit 22 determines that equipment EQ1 and equipment EQ2 are in a good state of contact. On the other hand, if the calculated distance is equal to or greater than the threshold, the analysis unit 22 determines that equipment EQ1 and equipment EQ2 are in a poor state of not contacting each other.

[0131] In this way, the analysis unit 22 can analyze the states of the multiple pieces of equipment by referring to the distances between the multiple pieces of equipment in real space, rather than the apparent distances on the first vehicle surroundings image, thereby enabling the states of the multiple pieces of equipment to be analyzed with high accuracy.

[0132] (Effects of Information Processing Device 2) In this manner, the information processing device 2 according to this exemplary embodiment identifies that the first facility included as a subject in the first vehicle surroundings image captured by camera CA1 and the second facility included as a subject in the second vehicle surroundings image captured by camera CA4 are the same facility EQ. Furthermore, the information processing device 2 calculates the distances from each of cameras CA1 and CA4 to the identified facility EQ. Therefore, the information processing device 2 can accurately calculate the distances from each of cameras CA1 and CA4 to the identified facility EQ.

[0133] Furthermore, the information processing device 2 determines whether a partial image including the same equipment EQ in the first vehicle surroundings image is blurred by referring to the calculated distance from camera CA1 to the same equipment EQ and the focus information of camera CA1. Furthermore, the information processing device 2 determines whether a partial image including the same equipment EQ in the second vehicle surroundings image is blurred by referring to the calculated distance from camera CA4 to the same equipment EQ and the focus information of camera CA4. Therefore, the information processing device 2 can accurately determine whether the equipment EQ included as a subject in the vehicle surroundings image is blurred.

[0134] [Example of Software Implementation] Some or all of the functions of the information processing devices 1 and 2 may be implemented by hardware such as an integrated circuit (IC chip), or may be implemented by software.

[0135] In the latter case, the information processing devices 1 and 2 are realized, for example, by a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in FIG. 12. The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for operating the computer C as the information processing devices 1 and 2. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing each function of the information processing devices 1 and 2.

[0136] The processor C1 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a tensor processing unit (TPU), a quantum processor, a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0137] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may also include a communication interface for transmitting and receiving data to and from other devices. The computer C may also include an input / output interface for connecting input / output devices such as a keyboard, a mouse, a display, and a printer.

[0138] The program P can also be recorded on a non-transitory, tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.

[0139] [Additional Note 1] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means disclosed in the above-described embodiments are also included in the technical scope of the present invention.

[0140] [Additional Note 2] Part or all of the above-described embodiment can also be described as follows: However, the present invention is not limited to the following described aspects.

[0141] (Supplementary Note 1) An information processing device comprising: an acquisition means for acquiring a first vehicle surroundings image taken by a first camera and a second vehicle surroundings image taken by a second camera, the image capturing ranges of which partially overlap; an identification means for identifying an identical facility included as a subject in both the first vehicle surroundings image and the second vehicle surroundings image; a calculation means for calculating the distance between the first camera and the second camera and the distance from the first camera and the second camera to the identical facility by referring to the first vehicle surroundings image and the second vehicle surroundings image; and a determination means for determining whether a partial image including the identical facility in the first vehicle surroundings image is blurred by referring to the distance from the first camera to the identical facility and focus information of the first camera, and for determining whether a partial image including the identical facility in the second vehicle surroundings image is blurred by referring to the distance from the second camera to the identical facility and focus information of the second camera.

[0142] (Supplementary Note 2) The information processing device according to Supplementary Note 1, further comprising an analysis unit that analyzes a state of the same equipment by referring to the partial image that is determined not to be blurred by the determination unit.

[0143] (Supplementary Note 3) The information processing device according to Supplementary Note 2, wherein the analysis means determines a parameter used to analyze a state of the identical equipment depending on a distance from the first camera or the second camera to the identical equipment.

[0144] (Supplementary Note 4) The information processing device according to Supplementary Note 2 or 3, wherein the analysis means determines parameters used to analyze the state of the identical equipment according to an angle between a line connecting the position of the first camera or the second camera and the position of the identical equipment and a baseline connecting the positions of the first camera and the second camera.

[0145] (Supplementary Note 5) The information processing device described in any one of Supplementary Notes 1 to 4, wherein the first camera is positioned to photograph a first facility included as a subject in the first vehicle surroundings image from the left side of the vehicle's traveling direction, the second camera is positioned to photograph a second facility included as a subject in the second vehicle surroundings image from the right side of the vehicle's traveling direction, the first camera and the second camera are controlled to photograph the subjects in synchronization with each other, and the identification means identifies a predetermined direction along the traveling direction in each of the first vehicle surroundings image and the second vehicle surroundings image, and identifies the first facility and the second facility as the same facility when a difference in position in the predetermined direction between an area including the first facility and an area including the second facility is within a predetermined range.

[0146] (Supplementary Note 6) The information processing device described in any one of Supplementary Notes 1 to 5, wherein the second vehicle surroundings image includes a plurality of second facilities as subjects, and the identification means identifies a predetermined direction along the vehicle's direction of travel in each of the first vehicle surroundings image and the second vehicle surroundings image, and identifies any one of the plurality of second facilities whose position in the predetermined direction of an area including the second facility is closest to the position in the predetermined direction of an area including first facilities included as subjects in the first vehicle surroundings image, as the same facility as the first facility.

[0147] (Supplementary Note 7) An information processing device according to any one of Supplementary Notes 1 to 6, wherein the first vehicle surroundings image includes one or more first facilities as subjects, and the second vehicle surroundings image includes one or more second facilities as subjects, and the identification means, when the number of the one or more first facilities and the number of the one or more second facilities are the same, identifies the first facilities and the second facilities identified as having a correspondence relationship as the same facility by identifying a correspondence relationship between the one or more first facilities and the one or more second facilities.

[0148] (Supplementary Note 8) An information processing device according to any one of Supplementary Notes 1 to 7, wherein the first vehicle surroundings image includes a plurality of first facilities as subjects, and the second vehicle surroundings image includes a plurality of second facilities as subjects, and the identification means identifies a vertical direction perpendicular to the direction of travel of the vehicle in each of the first vehicle surroundings image and the second vehicle surroundings image, and identifies the correspondence between each of the plurality of first facilities and each of the plurality of second facilities by taking into account their vertical positional relationship in the vertical direction, thereby identifying the first facilities and the second facilities that have been identified as having a correspondence relationship as the same facility.

[0149] (Supplementary Note 9) An information processing method including: an information processing device acquiring a first vehicle surroundings image taken by a first camera and a second vehicle surroundings image taken by a second camera, the image capturing ranges of which partially overlap; identifying identical equipment included as a subject in both the first vehicle surroundings image and the second vehicle surroundings image; calculating the distance between the first camera and the second camera and the distance from the first camera and the second camera to the identical equipment by referring to the first vehicle surroundings image and the second vehicle surroundings image; determining whether a partial image including the identical equipment in the first vehicle surroundings image is blurred by referring to the distance from the first camera to the identical equipment and focus information of the first camera, and determining whether a partial image including the identical equipment in the second vehicle surroundings image is blurred by referring to the distance from the second camera to the identical equipment and focus information of the second camera.

[0150] (Supplementary Note 10) A program for causing a computer to function as an information processing device, the program causing the computer to function as: an acquisition means for acquiring a first vehicle surroundings image taken by a first camera and a second vehicle surroundings image taken by a second camera, the image capturing ranges of which partially overlap; an identification means for identifying the same equipment included as a subject in both the first vehicle surroundings image and the second vehicle surroundings image; a calculation means for calculating the distance between the first camera and the second camera and the distance from the first camera and the second camera to the same equipment, by referring to the first vehicle surroundings image and the second vehicle surroundings image; and a determination means for determining whether a partial image including the same equipment in the first vehicle surroundings image is blurred by referring to the distance from the first camera to the same equipment and focus information of the first camera, and for determining whether a partial image including the same equipment in the second vehicle surroundings image is blurred by referring to the distance from the second camera to the same equipment and focus information of the second camera.

[0151] [Additional Note 3] Part or all of the above-described embodiment can also be expressed as follows.

[0152] An information processing device having at least one processor, which executes the following processes: an acquisition process for acquiring a first vehicle surroundings image taken by a first camera and a second vehicle surroundings image taken by a second camera, the image capturing ranges of which partially overlap; an identification process for identifying the same equipment that is included as a subject in both the first vehicle surroundings image and the second vehicle surroundings image; a calculation process for calculating the distance between the first camera and the second camera, and the distance from the first camera and the second camera to the same equipment, by referring to the first vehicle surroundings image and the second vehicle surroundings image; and a determination process for determining whether a partial image including the same equipment in the first vehicle surroundings image is blurred by referring to the distance from the first camera to the same equipment and focus information of the first camera, and determining whether a partial image including the same equipment in the second vehicle surroundings image is blurred by referring to the distance from the second camera to the same equipment and focus information of the second camera.

[0153] The information processing device may further include a memory that stores a program for causing the processor to execute the acquisition process, the identification process, the calculation process, and the determination process. The program may be recorded on a computer-readable, non-transitory, tangible recording medium.

[0154] REFERENCE SIGNS LIST 1, 2 Information processing device 11 Acquisition unit 12 Identification unit 13 Calculation unit 14 Determination unit 22 Analysis unit

Claims

1. An information processing device comprising: an acquisition means for acquiring a first vehicle surroundings image taken by a first camera and a second vehicle surroundings image taken by a second camera, the image capture ranges of which partially overlap; an identification means for identifying the same equipment included as a subject in both the first vehicle surroundings image and the second vehicle surroundings image; a calculation means for calculating the distance between the first camera and the second camera, and the distance from the first camera and the second camera to the same equipment, by referring to the first vehicle surroundings image and the second vehicle surroundings image; and a determination means for determining whether a partial image including the same equipment in the first vehicle surroundings image is blurred by referring to the distance from the first camera to the same equipment and focus information of the first camera, and for determining whether a partial image including the same equipment in the second vehicle surroundings image is blurred by referring to the distance from the second camera to the same equipment and focus information of the second camera.

2. The information processing device according to claim 1, further comprising an analysis means for analyzing the state of the same equipment by referring to the partial image determined by the determination means to be non-blurred.

3. The information processing device according to claim 2, wherein the analysis means determines the parameters used to analyze the state of the identical equipment according to the distance from the first camera or the second camera to the identical equipment.

4. An information processing device as described in claim 2 or 3, wherein the analysis means determines the parameters used to analyze the state of the identical equipment according to the angle between the position of the first camera or the second camera and a line connecting the identical equipment, and a baseline connecting the positions of the first camera and the second camera.

5. An information processing device as described in any one of claims 1 to 4, wherein the first camera is positioned to photograph a first facility included as a subject in the first vehicle surroundings image from the left side of the vehicle's direction of travel, the second camera is positioned to photograph a second facility included as a subject in the second vehicle surroundings image from the right side of the vehicle's direction of travel, the first camera and the second camera are controlled to photograph the subjects in synchronization with each other, and the identification means identifies a predetermined direction along the direction of travel in each of the first vehicle surroundings image and the second vehicle surroundings image, and identifies the first facility and the second facility as the same facility when the difference in position in the predetermined direction between the area including the first facility and the area including the second facility is within a predetermined range.

6. An information processing device according to any one of claims 1 to 5, wherein the second vehicle surroundings image includes a plurality of second facilities as subjects, and the identification means identifies a predetermined direction along the vehicle's traveling direction in each of the first vehicle surroundings image and the second vehicle surroundings image, and identifies any one of the plurality of second facilities whose position in the predetermined direction of an area including the second facility is closest to the position in the predetermined direction of an area including the first facility included as a subject in the first vehicle surroundings image, and the first facility as the same facility.

7. An information processing device according to any one of claims 1 to 6, wherein the first vehicle surroundings image includes one or more first facilities as subjects, and the second vehicle surroundings image includes one or more second facilities as subjects, and the identification means, when the number of the one or more first facilities and the number of the one or more second facilities are the same, identifies the first facilities and the second facilities identified as having a correspondence relationship as the same facilities by identifying a correspondence relationship between the one or more first facilities and the one or more second facilities.

8. An information processing device according to any one of claims 1 to 7, wherein the first vehicle surroundings image includes a plurality of first facilities as subjects, and the second vehicle surroundings image includes a plurality of second facilities as subjects, and the identification means identifies a vertical direction perpendicular to the vehicle's direction of travel in each of the first vehicle surroundings image and the second vehicle surroundings image, and identifies the correspondence between each of the plurality of first facilities and each of the plurality of second facilities by taking into account the vertical positional relationship in the vertical direction, thereby identifying the first facilities and the second facilities that have been identified as having a correspondence relationship as the same facility.

9. An information processing method including: an information processing device acquiring a first vehicle surroundings image taken by a first camera and a second vehicle surroundings image taken by a second camera, the image capture ranges of which partially overlap; identifying the same equipment that appears as a subject in both the first vehicle surroundings image and the second vehicle surroundings image; calculating the distance between the first camera and the second camera, and the distance from the first camera and the second camera to the same equipment, by referring to the first vehicle surroundings image and the second vehicle surroundings image; determining whether a partial image including the same equipment in the first vehicle surroundings image is blurred by referring to the distance from the first camera to the same equipment and focus information of the first camera, and determining whether a partial image including the same equipment in the second vehicle surroundings image is blurred by referring to the distance from the second camera to the same equipment and focus information of the second camera.

10. A program for causing a computer to function as an information processing device, the program causing the computer to function as: an acquisition means for acquiring a first vehicle surroundings image taken by a first camera and a second vehicle surroundings image taken by a second camera, the image capture ranges of which partially overlap; an identification means for identifying the same equipment included as a subject in both the first vehicle surroundings image and the second vehicle surroundings image; a calculation means for calculating the distance between the first camera and the second camera, and the distance from the first camera and the second camera to the same equipment, by referring to the first vehicle surroundings image and the second vehicle surroundings image; and a determination means for determining whether a partial image including the same equipment in the first vehicle surroundings image is blurred by referring to the distance from the first camera to the same equipment and focus information of the first camera, and for determining whether a partial image including the same equipment in the second vehicle surroundings image is blurred by referring to the distance from the second camera to the same equipment and focus information of the second camera.

Images