Information processing device, information processing method, and program

Abstract

In order to improve the accuracy of detecting a facility from an image obtained by imaging facilities around a vehicle in various environments, an information processing device comprises: an acquisition unit that acquires an image of the surrounding of the vehicle; an estimation unit that refers to the image of the surrounding of the vehicle and estimates an environment at the time of imaging; and a detection unit that, as a reference for detecting a facility included as a subject from the image of the surroundings of the vehicle, detects the facility using a progressively lower reference as the difficulty level of detection that corresponds to the estimated environment increases.

Description

Information processing device, information processing method, and program

[0001] The present invention relates to an information processing device, an information processing method, and a program that use images of facilities around a vehicle.

[0002] 2. Description of the Related Art Techniques for detecting equipment around a vehicle from images of the equipment are known.

[0003] Patent Literature 1 discloses an overhead line fitting detection device that detects overhead line fittings using line sensor images output from two line sensor cameras installed on a railway vehicle. The line sensor image is generated by combining line images captured continuously in a line shape while the line sensor or an object is moving. The overhead line fitting detection device performs a day / night determination process on each line image that constitutes the line sensor image. Furthermore, the overhead line fitting detection device detects overhead wires and overhead line fittings by inverting the brightness of areas where a certain number of consecutive lines have a brightness value above a threshold for line images determined to be night.

[0004] Japanese Patent Application Publication No. 2020-149286

[0005] The overhead line fitting detection device described in Patent Document 1 can only detect overhead line fittings when the area around the overhead line fitting is bright in the line sensor image. Therefore, it is not possible to detect overhead line fittings from line sensor images captured in environments where the area around the overhead line fitting is not bright (for example, inside a tunnel, under a bridge, or under a road). In other words, the overhead line fitting detection device has a problem in that the environments in which images for detecting overhead line fittings can be captured are limited. Therefore, there is a need for technology that can detect equipment included as subjects in images captured in various environments.

[0006] One aspect of the present invention has been made in consideration of the above-mentioned problems, and one example of its purpose is to provide a technology that improves the accuracy of detecting equipment around a vehicle from images taken of the equipment in various environments.

[0007] An information processing device according to one aspect of the present invention includes an acquisition means for acquiring an image of the vehicle's surroundings, an estimation means for estimating the environment at the time of photographing by referring to the image of the vehicle's surroundings, and a detection means for detecting equipment included as a subject from the image of the vehicle's surroundings using a lower criterion as the difficulty of detection according to the estimated environment increases.

[0008] An information processing method according to one aspect of the present invention includes an information processing device acquiring a vehicle surroundings image, estimating the environment at the time of photographing by referring to the vehicle surroundings image, and detecting equipment included as a subject from the vehicle surroundings image using lower criteria as the difficulty of detection according to the estimated environment increases.

[0009] A program according to one aspect of the present invention is a program that causes a computer to function as an information processing device, and causes the computer to function as an acquisition means that acquires an image of the vehicle's surroundings, an estimation means that estimates the environment at the time of shooting by referring to the image of the vehicle's surroundings, and a detection means that detects equipment included as a subject from the image of the vehicle's surroundings using lower criteria as the difficulty of detection according to the estimated environment increases.

[0010] According to one aspect of the present invention, it is possible to improve the accuracy of detecting facilities around a vehicle from images of the facilities taken in various environments.

[0011] FIG. 1 is a block diagram showing the configuration of an information processing device according to exemplary embodiment 1 of the present invention. FIG. 2 is a flow diagram showing the flow of an information processing method according to exemplary embodiment 1 of the present invention. FIG. 3 is a table showing an example of equipment and states in exemplary embodiment 2 of the present invention. FIG. 4 is a schematic diagram showing an example of a railway vehicle TR and equipment in exemplary embodiment 2 of the present invention. FIG. 5 is a block diagram showing the configuration of an information processing device 2 according to exemplary embodiment 2 of the present invention. FIG. 6 is a diagram showing a process for training an estimation model in exemplary embodiment 2 of the present invention. FIG. 7 is a flow diagram showing the flow of an information processing method according to exemplary embodiment 2 of the present invention. FIG. 8 is a diagram showing an example of the process of an estimation unit according to exemplary embodiment 2 of the present invention. FIG. 9 is a block diagram showing an example of the hardware configuration of an information processing device according to each exemplary embodiment of the present invention.

[0012] [First Exemplary Embodiment] A first exemplary embodiment of the present invention will be described in detail with reference to the drawings. This exemplary embodiment is a basic form of the exemplary embodiments described below.

[0013] (Configuration of Information Processing Apparatus 1) The configuration of the information processing apparatus 1 according to this exemplary embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the information processing apparatus 1 according to this exemplary embodiment.

[0014] 1, the information processing device 1 includes an acquisition unit 11, an estimation unit 12, and a detection unit 13. In this exemplary embodiment, the acquisition unit 11, the estimation unit 12, and the detection unit 13 are components that respectively realize an acquisition means, an estimation means, and a detection means.

[0015] The acquisition unit 11 acquires a vehicle surroundings image and supplies the acquired vehicle surroundings image to the estimation unit 12.

[0016] The estimation unit 12 estimates the environment at the time of shooting by referring to the vehicle surroundings image supplied from the acquisition unit 11. The estimation unit 12 supplies the vehicle surroundings image and the estimation result to the detection unit 13.

[0017] The detection unit 13 detects equipment included as a subject from the vehicle surroundings image supplied from the estimation unit 12 using lower criteria as the difficulty of detection increases according to the environment estimated by the estimation unit 12.

[0018] As described above, the information processing device 1 according to this exemplary embodiment is configured to include an acquisition unit 11 that acquires images of the vehicle's surroundings, an estimation unit 12 that estimates the environment at the time of shooting by referring to the images of the vehicle's surroundings supplied from the acquisition unit 11, and a detection unit 13 that detects equipment included as a subject from the images of the vehicle's surroundings supplied from the estimation unit 12 using lower criteria as the difficulty of detection increases according to the environment estimated by the estimation unit 12.

[0019] Therefore, the information processing device 1 according to this exemplary embodiment has the effect of improving the accuracy of detecting facilities around a vehicle from images of the facilities taken in various environments.

[0020] (Flow of Information Processing Method S1) The flow of the information processing method S1 according to this exemplary embodiment will be described with reference to Fig. 2. Fig. 2 is a flow chart showing the flow of the information processing method S1 according to this exemplary embodiment.

[0021] (Step S11) In step S11, the acquisition unit 11 acquires a vehicle surroundings image and supplies the acquired vehicle surroundings image to the estimation unit 12.

[0022] (Step S12) In step S12, the estimation unit 12 estimates the environment at the time of shooting by referring to the vehicle surroundings image supplied from the acquisition unit 11. The estimation unit 12 supplies the vehicle surroundings image and the estimation result to the detection unit 13.

[0023] (Step S13) In step S13, the detection unit 13 detects equipment included as a subject from the vehicle surroundings image supplied from the estimation unit 12 using a lower criterion as the difficulty of detection in accordance with the environment estimated by the estimation unit 12 increases.

[0024] As described above, the information processing method S1 according to this exemplary embodiment employs a configuration including step S11 in which the acquisition unit 11 acquires a vehicle surroundings image, step S12 in which the estimation unit 12 estimates the environment at the time of image capture by referring to the vehicle surroundings image supplied from the acquisition unit 11, and step S13 in which the detection unit 13 detects equipment included as a subject from the vehicle surroundings image supplied from the estimation unit 12 using a lower criterion for detecting the equipment as the difficulty of detection increases according to the environment estimated by the estimation unit 12. Therefore, the information processing method S1 according to this exemplary embodiment can achieve the same effects as the information processing device 1 described above.

[0025]

[0033] A second exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first exemplary embodiment are denoted by the same reference numerals, and their description will be omitted as appropriate.

[0026] (Overview of Information Processing Device 2) The information processing device 2 according to this exemplary embodiment is a device that analyzes the state of facilities around a vehicle. Although the vehicle is not particularly limited, this exemplary embodiment will describe a case where the information processing device 2 analyzes the state of facilities around a railway vehicle.

[0027] The term "surrounding equipment for railway vehicles" refers to equipment installed around railway vehicles to allow the vehicles to run. For example, the equipment is equipment for supplying power to railway vehicles. Examples of such equipment include wires, contact wires, hangers, ears, connectors, bolts, and insulators.

[0028] The state of the equipment is a state that can be determined from the appearance of the equipment. For example, the state of the equipment may be either a good state or a bad state. Furthermore, the good state and the bad state may differ depending on the equipment, and in this case, the equipment and the state may be defined in association with each other.

[0029] An example of a case where equipment and status are defined in association with each other will be described with reference to Fig. 3. Fig. 3 is a table showing an example of equipment and status in this exemplary embodiment.

[0030] 3, equipment and states are associated with each other. For example, equipment "W Year" is associated with the state "CC Misalignment" and the state "Bolt Dropped," and equipment "Protector" is associated with the state "Bolt Dropped."

[0031] The information processing device 2 analyzes whether the state of the equipment is in a state associated with the equipment based on the table of Fig. 3. As an example, the information processing device 2 analyzes whether the equipment "W Year" is in the state "CC Deviation" and whether the equipment "W Year" is in the state "Bolt Missing". As another example, the information processing device 2 analyzes whether the equipment "Protector" is in the state "Bolt Missing".

[0032] 3 indicates a state in which a malfunction has occurred in the equipment, so it can be said that the information processing device 2 analyzes whether or not the equipment is in a defective state.

[0033] Furthermore, in order to analyze the condition of the equipment, the information processing device 2 acquires a vehicle surroundings image, which is an image captured by a camera installed on an externally exposed surface of the railway vehicle and includes multiple pieces of equipment as subjects. The vehicle surroundings image may be an image captured while the vehicle is in motion. The externally exposed surface may include, for example, the top surface, bottom surface, left and right sides, and some or all of the front and rear sides of the railway vehicle. In this exemplary embodiment, the surface on which the camera is installed will be described mainly as the top surface of the railway vehicle, but is not limited to this. Hereinafter, "on the top surface of the railway vehicle" will also be simply referred to as "on the railway vehicle." The camera installed on the railway vehicle and the equipment as the subject will be described with reference to FIG. 4. FIG. 4 is a schematic diagram showing an example of a railway vehicle TR and equipment in this exemplary embodiment.

[0034] As shown in Fig. 4, a plurality of cameras (cameras CA1 to CA6) are installed on the railway vehicle TR, each capturing an image of equipment included in its angle of view and outputting the captured image. It is desirable that at least a portion of the ranges included in the angles of view of the plurality of cameras are different from each other. In other words, the range included in the angle of view of each camera (hereinafter also referred to as "capture range") may partially overlap with the range included in the angle of view of at least one other camera, but it is desirable that at least a portion of the ranges are different.

[0035] Although the installation manner of the multiple cameras is not particularly limited, one example is a configuration in which three cameras CA1 to CA3 are installed at different heights on the right side of the vehicle TR when the front of the vehicle TR is viewed from the direction of travel of the vehicle TR, as shown in Figure 4. Similarly, three cameras CA4 to CA6 are installed at different heights on the left side of the vehicle TR when the front of the vehicle TR is viewed from the direction of travel of the vehicle TR.

[0036] Furthermore, the timing at which cameras CA1 to CA6 photograph the equipment is not particularly limited. One example is a configuration in which cameras CA1 to CA6 each photograph the equipment at a predetermined interval. Another example is a configuration in which multiple cameras photograph the equipment synchronously at a predetermined interval. One example of such a configuration is a configuration in which cameras CA1 and CA4 photograph the equipment synchronously, cameras CA2 and CA5 photograph the equipment synchronously, and cameras CA3 and CA6 photograph the equipment synchronously. However, due to processing delays and the like, images photographed by multiple cameras synchronously are not necessarily photographed at exactly the same time. Furthermore, as one example, as shown in FIG. 4, cameras CA1 to CA6 photograph the equipment, including a hanger HA, an ear EA, and a trolley wire TW, as their subjects.

[0037] The information processing device 2 acquires vehicle surroundings images captured by the cameras CA1 to CA6. The configuration in which the information processing device 2 acquires the vehicle surroundings images is not particularly limited. One example is a configuration in which the vehicle surroundings images captured by the cameras CA1 to CA6 are stored in a recording medium, and the information processing device 2 acquires the vehicle surroundings images from the recording medium. Another configuration is a configuration in which the information processing device 2 and the cameras CA1 to CA6 are communicably connected via a network, and the information processing device 2 acquires the vehicle surroundings images via the network.

[0038] (Configuration of Information Processing Device 2) The configuration of the information processing device 2 will be described with reference to Fig. 5. Fig. 5 is a block diagram showing the configuration of the information processing device 2 according to this exemplary embodiment.

[0039] As shown in FIG. 5, the information processing device 2 includes a control unit 20, an input / output unit 27, a communication unit 28, and a storage unit 29.

[0040] The input / output unit 27 is an interface that receives user input and outputs data. For example, the input / output unit 27 supplies information indicating the received user input to the control unit 20 and outputs information supplied from the control unit 20. Examples of the input / output unit 27 include, but are not limited to, a keyboard, a mouse, a touchpad, a microphone, and a liquid crystal display.

[0041] The communication unit 28 is an interface that transmits and receives data via a network. For example, the communication unit 28 transmits data supplied from the control unit 20 to other devices, and supplies data received from other devices to the control unit 20. Examples of the communication unit 28 include, but are not limited to, communication chips for various communication standards such as Ethernet (registered trademark), Wi-Fi (registered trademark), and wireless communication standards for mobile data communication networks, and USB-compliant connectors.

[0042] The storage unit 29 stores data referenced by the control unit 20. Examples of data stored in the storage unit 29 include, but are not limited to, vehicle surroundings images, equipment images, and analysis results. Examples of the storage unit 29 include, but are not limited to, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0043] (Functions of control unit 20) The control unit 20 controls each component included in the information processing device 2. As shown in Fig. 5 , the control unit 20 also includes an acquisition unit 11, an estimation unit 12, a detection unit 13, an image connection unit 21, and an analysis unit 22. In this exemplary embodiment, the acquisition unit 11, the estimation unit 12, the detection unit 13, and the analysis unit 22 are components that respectively realize an acquisition means, an estimation means, a detection means, and an analysis means.

[0044] (Acquisition Unit 11) The acquisition unit 11 acquires data supplied from the input / output unit 27 or the communication unit 28. The acquisition unit 11 also acquires data stored in the storage unit 29. As an example, the acquisition unit 11 acquires vehicle surroundings images from a plurality of cameras, cameras CA1 to CA6, via the input / output unit 27 or the communication unit 28. As another example, the acquisition unit 11 may acquire vehicle surroundings images captured by cameras CA1 to CA6 and stored in the storage unit 29.

[0045] (Estimation Unit 12) The estimation unit 12 estimates the environment at the time of image capture by referring to the vehicle surroundings image. The environment at the time of image capture estimated by the estimation unit 12 is not limited. As an example, the estimation unit 12 estimates whether the environment at the time of image capture is a predetermined environment. Examples of the predetermined environment include "inside a tunnel," "under a bridge," "under a road," and "raining." As another example, the estimation unit 12 is configured to estimate which of two or more environments (e.g., inside a tunnel, under a bridge, and others) the environment at the time of image capture is. In this exemplary embodiment, the estimation unit 12 estimates whether the environment at the time of image capture is inside a tunnel. The estimation unit 12 supplies the vehicle surroundings image and the estimation result to the detection unit 13.

[0046] There is no limitation on the method by which the estimation unit 12 estimates whether the environment at the time the vehicle surroundings image was captured is inside a tunnel. As an example, the estimation unit 12 may estimate whether the environment at the time the vehicle surroundings image was captured is inside a tunnel using an estimation model learned by machine learning.

[0047] A configuration for training the estimation model will be described with reference to Fig. 6 . Fig. 6 is a diagram showing a process for training the estimation model MD1 in this exemplary embodiment. The estimation model receives an image of the vehicle's surroundings as input and outputs an estimation result indicating whether the image of the vehicle's surroundings was captured inside a tunnel. The training process shown in Fig. 6 may be executed by another information processing device or by the control unit 20 of the information processing device 2.

[0048] As shown in FIG. 6 , first, a vehicle periphery image vp is associated with environmental information ei indicating whether the environment at the time of capturing the vehicle periphery image vp is a tunnel. In other words, the environmental information ei is a correct label. Next, the vehicle periphery image vp is cropped, its image size is reduced, and contrast correction is performed to match the contrast between the multiple vehicle periphery images vp (for example, by performing contrast limited adaptive histogram equalization (CLAHE)). A processed vehicle periphery image p_vp is generated. An estimation model MD1 is trained using a training dataset that combines the processed vehicle periphery image p_vp and the environmental information ei. Specifically, when the vehicle periphery image p_vp is input to the estimation model MD1, training is performed so that environmental information ei, which is a correct label and indicates whether the vehicle periphery image p_vp is an image captured inside a tunnel, is output.

[0049] Another method for the estimation unit 12 to estimate whether the environment at the time of image capture is inside a tunnel is to refer to pixel values ​​of the vehicle surroundings image. For example, if the average brightness value of each pixel in the vehicle surroundings image is higher than a predetermined value, the estimation unit 12 estimates that the vehicle surroundings image was captured outside a tunnel, and if the average brightness value of each pixel in the vehicle surroundings image is equal to or lower than a predetermined value, the estimation unit 12 estimates that the vehicle surroundings image was captured inside a tunnel. However, the method by which the estimation unit 12 estimates the environment at the time of image capture is not limited to the above-mentioned method.

[0050] The estimation unit 12 may also divide the vehicle surroundings image into a plurality of partial images along a direction corresponding to the vehicle's traveling direction, and estimate whether the environment at the time of capturing each partial image was inside a tunnel. The direction corresponding to the vehicle's traveling direction may be the vehicle's traveling direction. Specific examples of processing performed by the estimation unit 12 for this configuration will be described later.

[0051] (Detection unit 13) The detection unit 13 detects equipment as a subject from the vehicle surroundings image using a lower criterion as the detection difficulty according to the environment indicated by the estimation result supplied from the estimation unit 12 increases. The detection unit 13 supplies the estimation result and an equipment image including the detected equipment to the analysis unit 22. One equipment image includes one or more pieces of equipment as subjects.

[0052] A high level of difficulty in detection refers to a situation in which the equipment is difficult to distinguish from its surroundings in the vehicle surroundings image. An example of this is a situation in which the vehicle surroundings image is taken in a dark situation, and an object similar to the equipment is captured in the background of the equipment. For example, a vehicle surroundings image taken in a tunnel may be difficult to see the equipment due to the darkness, or the inner wall of the tunnel captured in the background of the equipment may resemble the equipment, making it difficult for the detection unit 13 to detect the equipment.

[0053] Furthermore, a low standard means that the conditions for satisfying the standard are lenient. That is, the higher the difficulty level of detecting a facility, the more lenient the detection unit 13 uses conditions that make it easier to detect the facility. On the other hand, the lower the difficulty level of detecting a facility, the more strict the detection unit 13 uses conditions that make it harder to detect the facility.

[0054] With this configuration, for equipment included as a subject in a vehicle surroundings image captured in an environment where detection difficulty is low, the detection unit 13 sets high standards for detecting the equipment, thereby reducing overdetection, in which a subject that is not equipment is detected as equipment. On the other hand, for equipment included as a subject in a vehicle surroundings image captured in an environment where detection difficulty is high, the detection unit 13 sets low standards for detecting the equipment, thereby reducing overdetection while allowing overdetection. Furthermore, by limiting the range in which overdetection is allowed to only environments where detection difficulty is high, the detection unit 13 can reduce overdetection while suppressing overdetection overall.

[0055] The detection criterion may be a threshold value for comparison with a certainty factor indicating the likelihood that the facility is included as a subject in the vehicle surroundings image. In this case, if the certainty factor is equal to or greater than the criterion, the facility is detected, and if the certainty factor is lower than the criterion, the facility is not detected. In other words, the lower the detection criterion, the higher the facility detection rate, and although there is a possibility of an increase in overdetection, there is a reduction in overdetection. The storage unit 29 may store each environment that can be estimated (e.g., inside a tunnel, not inside a tunnel) in association with the detection criterion.

[0056] The detection unit 13 uses a lower detection standard when the vehicle is inside a tunnel than when the vehicle is not inside a tunnel. In other words, when the estimation result supplied from the estimation unit 12 indicates that the vehicle surroundings image was taken inside a tunnel, the detection unit 13 detects the facility using a lower standard. On the other hand, when the estimation result supplied from the estimation unit 12 indicates that the vehicle surroundings image was taken outside the tunnel, the detection unit 13 detects the facility using a higher standard.

[0057] As an example of a method for detecting equipment, the detection unit 13 detects equipment using a region extraction model that is trained to receive an image of the vehicle's surroundings as input and output an area of ​​the equipment detected by region extraction (e.g., PWC). As another example, the detection unit 13 detects equipment using an object detection model that is trained to receive an image of the vehicle's surroundings as input and output an area of ​​the equipment detected by object detection (e.g., SSD (Single Shot Multibox Detector)). In this exemplary embodiment, an example in which an object detection model is applied as the detection model will be mainly described.

[0058] Furthermore, when detecting equipment using an object detection model, the detection unit 13 detects the equipment depending on whether the certainty factor output from the object detection model exceeds a reference value. For example, when the estimation result supplied from the estimation unit 12 indicates that the vehicle surroundings image was taken inside a tunnel, the detection unit 13 detects the equipment depending on whether the certainty factor output from the object detection model is higher than a first threshold. On the other hand, when the estimation result supplied from the estimation unit 12 indicates that the vehicle surroundings image was taken outside the tunnel, the detection unit 13 detects the equipment depending on whether the certainty factor output from the object detection model is higher than a second threshold that is higher than the first threshold.

[0059] As an example, assume that the estimation result supplied from the estimation unit 12 indicates that the vehicle surroundings image was taken inside a tunnel, and the confidence level output from the object detection model is higher than a first threshold value. In this case, the detection unit 13 detects the facility detected by the object detection model as the facility included in the vehicle surroundings image.

[0060] On the other hand, assume that the estimation result supplied from the estimation unit 12 indicates that the vehicle surroundings image was taken inside a tunnel, and the confidence level output from the object detection model is equal to or less than the first threshold. In this case, the detection unit 13 does not detect the facility detected by the object detection model as a facility included in the vehicle surroundings image.

[0061] As another example, assume that the estimation result supplied from the estimation unit 12 indicates that the vehicle surroundings image was taken outside a tunnel, and the confidence level output from the object detection model is lower than a second threshold that is higher than the first threshold. In this case, the detection unit 13 does not detect the facility detected by the object detection model as a facility included in the vehicle surroundings image.

[0062] On the other hand, it is assumed that the estimation result supplied from the estimation unit 12 indicates that the vehicle surroundings image was taken outside the tunnel, and the confidence level output from the object detection model is higher than the second threshold value. In this case, the detection unit 13 detects the facility detected by the object detection model as a facility included in the vehicle surroundings image.

[0063] (Image connection unit 21) The image connection unit 21 connects images. As an example, the image connection unit 21 connects a plurality of vehicle surroundings images stored in the storage unit 29. The image connection unit 21 stores the connected vehicle surroundings images in the storage unit 29.

[0064] For example, the image connection unit 21 acquires a plurality of vehicle surroundings images captured by the camera CA1 from the storage unit 29. Next, the image connection unit 21 sorts the acquired vehicle surroundings images according to the date and time of capture. The image connection unit 21 then connects the rearranged vehicle surroundings images so that equipment included as subjects in each of the vehicle surroundings images are connected, thereby generating a connected vehicle surroundings image. The image connection unit 21 performs similar processing on the plurality of vehicle surroundings images captured by the other cameras CA2 to CA6, respectively, to generate a connected vehicle surroundings image.

[0065] (Analysis unit 22) The analysis unit 22 analyzes the state of the equipment. As an example, the analysis unit 22 analyzes the state of the equipment using a quality determination model that receives an equipment image as input and outputs information indicating whether the state of the equipment included as a subject in the equipment image is poor and the degree of certainty of the information.

[0066] As another example, in this exemplary embodiment, the analysis unit 22 uses parameters according to the environment estimated by the estimation unit 12 to analyze the state of the equipment by referring to an equipment image obtained from a vehicle surroundings image, which includes equipment that is included as a subject in the vehicle surroundings image.

[0067] In this configuration, the parameter may be a criterion for determining that the equipment is defective. For example, the criterion for determining that the equipment is defective may be a threshold value for comparison with a certainty factor indicating the likelihood that the equipment is defective. In this case, if the certainty factor is equal to or greater than the criterion, the equipment is determined to be defective, and if the certainty factor is lower than the criterion, the equipment is determined to be not defective (i.e., good). In other words, the lower the criterion for determining that the equipment is defective, the higher the defect determination rate, which may increase the number of false determinations in which equipment that is actually good is determined to be defective, but reduces the number of overlooked defects. The storage unit 29 may store each environment that can be estimated (e.g., inside a tunnel, not inside a tunnel) and the criterion for determining that the equipment is defective, in association with each other.

[0068] In this case, the analysis unit 22 determines whether the equipment is defective using a lower standard as the difficulty of determining whether the equipment is defective depending on the environment at the time of image capture increases. For example, when analyzing equipment included as a subject in an equipment image that the estimation unit 12 has estimated to be inside a tunnel at the time of image capture, the analysis unit 22 uses the third threshold as the standard for determining whether the equipment is defective. On the other hand, when analyzing equipment included as a subject in an equipment image that the estimation unit 12 has estimated to be outside a tunnel at the time of image capture, the analysis unit 22 uses a fourth threshold, which is higher than the third threshold, as the standard for determining whether the equipment is defective.

[0069] As an example, a case will be described in which the analysis unit 22 uses the above-mentioned quality determination model. In this configuration, assume that an equipment image estimated by the estimation unit 12 to be taken in a tunnel environment is input to the quality determination model. In this case, if the confidence level output from the quality determination model is higher than the third threshold, the analysis unit 22 outputs, as an analysis result, information indicating whether the condition of the equipment is poor or not, output from the quality determination model. On the other hand, if the confidence level output from the quality determination model is lower than the third threshold, the analysis unit 22 outputs an analysis result indicating that quality determination is impossible.

[0070] Next, assume that an equipment image that the estimation unit 12 has estimated was not taken in a tunnel is input to the quality determination model. In this case, if the confidence level output from the quality determination model is higher than the fourth threshold, the analysis unit 22 outputs, as an analysis result, information indicating whether the condition of the equipment is poor or not, output from the quality determination model. On the other hand, if the confidence level output from the quality determination model is lower than the fourth threshold, the analysis unit 22 outputs an analysis result indicating that quality determination is impossible.

[0071] In this way, for equipment included as a subject in vehicle surroundings images taken outside the tunnel, the analysis unit 22 sets a high standard for determining that the equipment is defective, thereby reducing overdetection, in which equipment that is not defective is determined to be defective. On the other hand, for equipment included as a subject in vehicle surroundings images taken inside the tunnel, the analysis unit 22 sets a low standard for determining that the equipment is defective, thereby allowing overdetection while reducing overlooking of equipment defects. Furthermore, by limiting the range in which overdetection is allowed to the inside of the tunnel, overdetection can be suppressed overall while reducing overlooking of defects.

[0072] Preferably, the third threshold and the fourth threshold differ depending on the facility and the state of the facility. With this configuration, the analysis unit 22 can appropriately analyze the state of each facility and each state of the facility.

[0073] (Flow of Information Processing Method S2) The flow of the information processing method S2 according to this exemplary embodiment will be described with reference to Fig. 7. Fig. 7 is a flow chart showing the flow of the information processing method S2 according to this exemplary embodiment.

[0074] (Step S21) In step S21, the acquisition unit 11 acquires the vehicle surroundings images captured by the cameras CA1 to CA6, and stores the acquired vehicle surroundings images in the storage unit 29.

[0075] (Step S22) In step S22, the image connection unit 21 acquires a plurality of vehicle surroundings images stored in the storage unit 29. The image connection unit 21 also connects the acquired plurality of vehicle surroundings images. The image connection unit 21 stores the connected vehicle surroundings images in the storage unit 29.

[0076] (Step S23) The estimation unit 12 acquires the post-coupling vehicle surroundings image stored in the storage unit 29. The estimation unit 12 estimates whether the environment at the time of image capture was inside a tunnel from the post-coupling vehicle surroundings image. The processing in step S23 will be described with reference to Fig. 8. Fig. 8 is a diagram showing an example of the processing by the estimation unit 12 according to this exemplary embodiment.

[0077] As shown in Fig. 8 , the estimation unit 12 acquires a concatenated vehicle surroundings image c_vp. Next, as shown in the upper part of Fig. 8 , the estimation unit 12 divides the concatenated vehicle surroundings image c_vp into a plurality of divided images (divided image dp1, divided image dp2) along a direction corresponding to the traveling direction of the railway vehicle. In the upper part of Fig. 8 , the estimation unit 12 similarly divides the right side of divided image dp2 into a plurality of divided images. Furthermore, as shown in the upper part of Fig. 8 , the estimation unit 12 may divide the concatenated vehicle surroundings image c_vp into a plurality of divided images such that at least a portion of the regions overlap.

[0078] Next, the estimation unit 12 cuts out partial images from the divided images. As an example, the estimation unit 12 is configured to cut out partial images that include at least one piece of equipment as a subject, based on the angles at which the cameras CA1 to CA6 capture images, the position of the equipment, etc. For example, as shown in FIG. 8 , a partial image tp1 that includes a trolley wire as a subject is cut out from the divided image dp1.

[0079] The estimation unit 12 then estimates, for each partial image, whether the environment at the time of shooting was inside a tunnel. The method by which the estimation unit estimates whether the environment at the time of shooting was inside a tunnel is as described above. The estimation unit 12 supplies the partial image and the estimation result to the detection unit 13.

[0080] Here, the estimation unit 12 may correct the environment at the time of shooting for each partial image based on the order of the environments at the time of shooting estimated for each partial image. For example, the estimation unit 12 assumes that a sequence of N or more partial images estimated to be a first environment, a sequence of n or less partial images estimated to be a second environment, and a sequence of N or more partial images estimated to be the first environment are arranged in this order. n is an integer equal to or greater than 1, and N is an integer greater than n. The first environment is, for example, "inside a tunnel," and the second environment is, for example, "outside the tunnel."

[0081] In this case, the estimation unit 12 may correct the estimation result of the environment at the time of shooting for each of the n or less partial images that were estimated to be the second environment to the first environment. For example, in Figure 8, assume that the environment at the time of shooting of one partial image tp10 is estimated to be outside the tunnel, and the environments at the time of shooting of the two adjacent partial images (partial images tp8 to tp9, tp11 to tp12) are estimated to be inside the tunnel. In this case, since the estimation result that only partial image tp10 was outside the tunnel may be incorrect, the estimation unit 12 may correct the environment at the time of shooting of partial image tp10 and estimate it to be inside the tunnel.

[0082] In this way, if the environment at the time of photographing a certain partial image (partial image tp11 in FIG. 8 ) is estimated to be different from the environment at the time of photographing the partial images on either side (partial image tp10 and partial image tp12), it is unlikely that the environment at the time of photographing only the certain partial image was different, and therefore it is highly likely that the estimation result for the certain partial image is incorrect. In such a case, the estimation unit 12 corrects the environment at the time of photographing the certain partial image so that it is the same as the environment at the time of photographing the partial images on either side. Therefore, the estimation unit 12 can improve the accuracy of estimating the environment at the time of photographing the partial image.

[0083] Also, assume that the captured image of the vehicle's surroundings spans multiple environments. For example, partial image tp8 shown in FIG. 8 is an image of the outside of a tunnel and the inside of the tunnel. In this case, instead of the above-described estimation model, the estimation unit 12 may use an estimation model that, when a partial image is input, outputs information indicating the area inside the tunnel in the partial image. The estimation unit 12 references the information output from the estimation model, and if the area inside the tunnel is equal to or greater than a predetermined value, estimates that the partial image was captured inside the tunnel. With this configuration, the estimation unit 12 can preferably detect facilities inside the tunnel in the facility detection process described below.

[0084] As another example, the estimation unit 12 may be configured to refer to the certainty factor output from the estimation model described above. For example, when a partial image is input, if the certainty factor for estimating that the partial image is taken inside a tunnel based on the estimation model is higher than the certainty factor for estimating that the partial image is not taken inside a tunnel, the estimation unit 12 estimates that the partial image was taken inside a tunnel. On the other hand, if the certainty factor for estimating that the partial image is taken inside a tunnel based on the estimation model is lower than the certainty factor for estimating that the partial image is not taken inside a tunnel, the estimation unit 12 estimates that the partial image was taken outside the tunnel. Even with this configuration, the estimation unit 12 can preferably detect facilities inside a tunnel in the facility detection process described below.

[0085] As yet another example, the estimation unit 12 may estimate whether a partial image is an image taken inside a tunnel based on pixel values. For example, the estimation unit 12 calculates the percentage of pixels in the partial image whose pixel values ​​are lower than a predetermined value. If the calculated percentage is equal to or greater than a predetermined percentage (e.g., 30% or greater), the estimation unit 12 may estimate that the environment in which the partial image was taken is inside a tunnel. Even in this configuration, the estimation unit 12 can preferably detect facilities inside a tunnel in the facility detection process described below.

[0086] (Step S24) In step S24, the detection unit 13 refers to the estimation result supplied from the estimation unit 12 in step S23, and determines whether the environment in which the partial image supplied from the estimation unit 12 was captured is estimated to be inside a tunnel.

[0087] (Step S25) If it is estimated in step S24 that the location is inside a tunnel (step S24: YES), in step S25 the detection unit 13 detects equipment included as a subject from the partial image using a low criterion. The detection unit 13 supplies the equipment image including the detected equipment and the estimation result supplied from the estimation unit 12 to the analysis unit 22.

[0088] (Step S26) If it is estimated in step S24 that the subject is not inside a tunnel (step S25: NO), in step S26 the detection unit 13 detects equipment included as a subject from the partial image using a high standard. The detection unit 13 supplies the equipment image including the detected equipment and the estimation result supplied from the estimation unit 12 to the analysis unit 22.

[0089] (Step S27) In step S27, the analysis unit 22 analyzes the state of the equipment by using the parameters according to the environment estimated by the estimation unit 12 and by referring to the equipment image.

[0090] (Effects of information processing device 2) In this way, the information processing device 2 according to this exemplary embodiment estimates whether the environment at the time the vehicle surroundings image was captured is inside a tunnel or not, and detects equipment using lower criteria as the detection difficulty increases depending on whether the environment is inside a tunnel or not.

[0091] In a vehicle surroundings image taken inside a tunnel, it may be difficult to see the equipment because it is dark, or the pattern of the tunnel's inner wall in the background of the equipment may resemble the equipment, making it difficult to detect the equipment from the vehicle surroundings image.The information processing device 2 detects equipment included as a subject in a vehicle surroundings image taken inside a tunnel using a low standard, and therefore can effectively prevent equipment included as a subject in a vehicle surroundings image taken inside a tunnel from being overlooked.

[0092] On the other hand, for vehicle surroundings images taken outside the tunnel, the information processing device 2 detects equipment using high standards, and therefore the information processing device 2 can preferably prevent overdetection of subjects other than equipment as equipment.

[0093] [Example of Software Implementation] Some or all of the functions of the information processing devices 1 and 2 may be implemented by hardware such as an integrated circuit (IC chip), or may be implemented by software.

[0094] In the latter case, the information processing devices 1 and 2 are realized, for example, by a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in FIG. 9. The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for operating the computer C as the information processing devices 1 and 2. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing each function of the information processing devices 1 and 2.

[0095] The processor C1 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a tensor processing unit (TPU), a quantum processor, a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0096] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may also include a communication interface for transmitting and receiving data to and from other devices. The computer C may also include an input / output interface for connecting input / output devices such as a keyboard, a mouse, a display, and a printer.

[0097] The program P can also be recorded on a non-transitory, tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.

[0098] [Additional Note 1] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means disclosed in the above-described embodiments are also included in the technical scope of the present invention.

[0099] [Additional Note 2] Part or all of the above-described embodiment can also be described as follows: However, the present invention is not limited to the following described aspects.

[0100] (Supplementary Note 1) An information processing device including: an acquisition means for acquiring a vehicle surroundings image; an estimation means for estimating the environment at the time of photographing by referring to the vehicle surroundings image; and a detection means for detecting equipment included as a subject from the vehicle surroundings image using a lower criterion as the difficulty of detection according to the estimated environment increases.

[0101] (Supplementary Note 2) The information processing device described in Supplementary Note 1 further includes an analysis means for analyzing the state of equipment by referring to an equipment image obtained from the vehicle surroundings image, the equipment image including equipment included as a subject in the vehicle surroundings image, using parameters according to the estimated environment.

[0102] (Supplementary Note 3) The information processing device according to Supplementary Note 1 or 2, wherein the estimation means estimates whether the environment at the time of shooting is inside a tunnel or not, and the detection means uses a lower standard for the detection when the environment is inside a tunnel than when the environment is not inside a tunnel.

[0103] (Supplementary Note 4) The information processing device according to any one of Supplementary Notes 1 to 3, wherein the estimation means divides the vehicle surroundings image into a plurality of partial images along a direction corresponding to the vehicle's traveling direction, and corrects the environment at the time of shooting for each partial image based on the order of the environment at the time of shooting estimated for each partial image.

[0104] (Supplementary Note 5) An information processing method including: an information processing device acquiring a vehicle surroundings image; estimating the environment at the time of photographing by referring to the vehicle surroundings image; and detecting equipment included as a subject from the vehicle surroundings image using a lower criterion as the difficulty of detection according to the estimated environment increases.

[0105] (Supplementary Note 6) The information processing method according to Supplementary Note 5, further including the information processing device analyzing a state of an equipment by referring to an equipment image acquired from the vehicle surroundings image, the equipment image including the equipment included as a subject in the vehicle surroundings image, using parameters according to the estimated environment.

[0106] (Supplementary Note 7) A program that causes a computer to function as an information processing device, the program causing the computer to function as an acquisition means that acquires an image of the vehicle's surroundings, an estimation means that estimates the environment at the time of shooting by referring to the image of the vehicle's surroundings, and a detection means that detects equipment included as a subject from the image of the vehicle's surroundings using lower criteria as the difficulty of detection according to the estimated environment increases.

[0107] (Appendix 8) The program described in Appendix 7 further causes the computer to function as an analysis means for analyzing the state of equipment by referring to an equipment image obtained from the vehicle surroundings image, the equipment image including equipment that is included as a subject in the vehicle surroundings image, using parameters according to the estimated environment.

[0108] [Additional Note 3] Part or all of the above-described embodiment can also be expressed as follows.

[0109] An information processing device comprising at least one processor, which executes an acquisition process for acquiring a vehicle surroundings image, an estimation process for estimating the environment at the time of photographing by referring to the vehicle surroundings image, and a detection process for detecting equipment included as a subject from the vehicle surroundings image using lower criteria as the difficulty of detection according to the estimated environment increases.

[0110] The information processing device may further include a memory that stores a program for causing the processor to execute the acquisition process, the estimation process, and the detection process. The program may be recorded on a computer-readable, non-transitory, tangible recording medium.

[0111] 1, 2 Information processing device 11 Acquisition unit 12 Estimation unit 13 Detection unit 22 Analysis unit

Claims

1. An information processing device comprising: an acquisition means for acquiring a vehicle surroundings image; an estimation means for estimating the environment at the time of photographing by referring to the vehicle surroundings image; and a detection means for detecting equipment included as a subject in the vehicle surroundings image using a lower standard as the difficulty of detection according to the estimated environment increases.

2. The information processing device according to claim 1, further comprising an analysis means for analyzing the state of equipment by referring to an equipment image obtained from the vehicle surroundings image, the equipment image including equipment included as a subject in the vehicle surroundings image, using parameters according to the estimated environment.

3. An information processing device as described in claim 1 or 2, wherein the estimation means estimates whether the environment at the time of shooting is inside a tunnel or not, and the detection means uses a lower standard for the detection when the environment is inside a tunnel than when the environment is not inside a tunnel.

4. An information processing device as described in claim 1 or 2, wherein the estimation means divides the vehicle surroundings image into a plurality of partial images along a direction corresponding to the vehicle's traveling direction, and corrects the environment at the time of shooting for each partial image based on the order of the environment at the time of shooting estimated for each partial image.

5. An information processing method including an information processing device acquiring a vehicle surroundings image, estimating the environment at the time of photographing by referring to the vehicle surroundings image, and detecting equipment included as a subject from the vehicle surroundings image using lower criteria as the difficulty of detection increases according to the estimated environment.

6. The information processing method according to claim 5, further comprising the information processing device analyzing the state of the equipment by referring to an equipment image obtained from the vehicle surroundings image, the equipment image including the equipment included as a subject in the vehicle surroundings image, using parameters according to the estimated environment.

7. A program that causes a computer to function as an information processing device, the program causing the computer to function as: an acquisition means that acquires an image of the vehicle's surroundings; an estimation means that estimates the environment at the time of photographing by referring to the image of the vehicle's surroundings; and a detection means that detects equipment included as a subject in the image of the vehicle's surroundings using lower criteria as the difficulty of detection increases according to the estimated environment.

8. The program described in claim 7, which further causes the computer to function as an analysis means for analyzing the state of equipment by referring to equipment images obtained from the vehicle surroundings image, which include equipment that is included as a subject in the vehicle surroundings image, using parameters corresponding to the estimated environment.

Images