Information processing device, information processing method, and program

Abstract

In order to improve the accuracy of detecting a defect in a facility around a vehicle, this information processing device comprises: an acquisition unit that acquires a plurality of facility images each including one or more facilities among a plurality of facilities included as subjects in vehicle peripheral images; a determination unit that determines the presence or absence of a blur in each of the facility images; and an extraction unit that extracts, as an analysis target to be analyzed, the state of a facility included as a subject in the facility image determined to have no blur.

Description

Information processing device, information processing method, and program

[0001] The present invention relates to an information processing device, an information processing method, and a program that use images of facilities around a vehicle.

[0002] Patent Document 1 discloses a surface condition diagnosis device that diagnoses the deterioration level of overhead power transmission insulators based on surface image data obtained by photographing the surface of the overhead power transmission insulator. The surface condition diagnosis device determines whether the surface image data is out of focus, and performs a correction process to change the brightness of surface image data that is not out of focus, and a correction process to change the contrast of surface image data that is out of focus.

[0003] There is a need for a technology that can detect defects in equipment around a vehicle using images of the equipment. One idea is to apply the surface condition diagnosis device described in Patent Document 1 to detect defects in the equipment, but the surface condition detection device described in Patent Document 1 has the following problems.

[0004] Japanese Patent Application Publication No. 2017-223512

[0005] The surface condition diagnostic device described in Patent Document 1 determines whether or not an image data is out of focus on a per-image data basis. Therefore, even if the overhead power transmission insulator to be diagnosed is out of focus in the surface image data, if portions other than the overhead power transmission insulator are not out of focus, the surface image data may be processed as not being out of focus.

[0006] As described above, with the surface condition diagnosis device described in Patent Document 1, even if an overhead power transmission insulator included as a subject in surface image data is out of focus, the surface image data may be determined to be not out of focus, and the deterioration level of the overhead power transmission insulator may be diagnosed based on the out-of-focus image of the overhead power transmission insulator. As a result, the surface condition diagnosis device has a problem of low accuracy in detecting defects in overhead power transmission insulators. Therefore, when the surface condition detection device described in Patent Document 1 is used to detect defects in equipment around a vehicle, there is a problem of low accuracy in detecting defects in the equipment.

[0007] One aspect of the present invention has been made in view of the above-mentioned problems, and one object of the present invention is to provide a technique for improving the accuracy of detecting defects in equipment around a vehicle.

[0008] An information processing device according to one aspect of the present invention includes an acquisition means for acquiring a plurality of equipment images each including one or more pieces of equipment included as subjects in a vehicle surroundings image, a determination means for determining whether or not each equipment image is blurred, and an extraction means for extracting, from the plurality of equipment images, equipment images determined to be free of blur as analysis targets for analyzing the state of the equipment included as subjects in the equipment images.

[0009] An information processing method according to one aspect of the present invention includes an information processing device acquiring equipment images each including one or more pieces of equipment from among a plurality of pieces of equipment included as subjects in a vehicle surroundings image, determining whether or not each equipment image is blurred, and extracting, from among the plurality of equipment images, equipment images determined to be free of blur as analysis targets for analyzing the state of the equipment included as subjects in the equipment images.

[0010] A program according to one aspect of the present invention is a program for causing a computer to function as an information processing device, and causes the computer to function as an acquisition means for acquiring a plurality of equipment images, each of which includes one or more pieces of equipment included as subjects in a vehicle surroundings image, a determination means for determining whether or not each equipment image is blurred, and an extraction means for extracting, from the plurality of equipment images, equipment images determined to be free of blur as analysis targets for analyzing the state of the equipment included as subjects in the equipment images.

[0011] According to one aspect of the present invention, it is possible to improve the accuracy of detecting defects in equipment around a vehicle.

[0012] FIG. 1 is a block diagram showing the configuration of an information processing device according to exemplary embodiment 1 of the present invention. FIG. 2 is a flow diagram showing the flow of an information processing method according to exemplary embodiment 1 of the present invention. FIG. 3 is a table showing an example of equipment and states in exemplary embodiment 2 of the present invention. FIG. 4 is a schematic diagram showing an example of a railway vehicle and equipment in exemplary embodiment 2 of the present invention. FIG. 5 is a block diagram showing the configuration of an information processing device according to exemplary embodiment 2 of the present invention. FIG. 6 is a diagram showing a process of training a decision model in exemplary embodiment 2 of the present invention. FIG. 7 is a flow diagram showing the flow of an information processing method according to exemplary embodiment 2 of the present invention. FIG. 8 is a block diagram showing an example of the hardware configuration of an information processing device according to each exemplary embodiment of the present invention.

[0013] [First Exemplary Embodiment] A first exemplary embodiment of the present invention will be described in detail with reference to the drawings. This exemplary embodiment is a basic form of the exemplary embodiments described below.

[0014] (Configuration of Information Processing Apparatus 1) The configuration of the information processing apparatus 1 according to this exemplary embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the information processing apparatus 1 according to this exemplary embodiment.

[0015] 1, the information processing device 1 according to this exemplary embodiment includes an acquisition unit 11, a determination unit 12, and an extraction unit 13. The acquisition unit 11, the determination unit 12, and the extraction unit 13 are components that respectively realize an acquisition means, a determination means, and an extraction means in this exemplary embodiment.

[0016] The acquisition unit 11 acquires a plurality of equipment images each including one or more pieces of equipment among a plurality of pieces of equipment included as subjects in the vehicle surroundings image. The acquisition unit 11 supplies the acquired equipment images to the determination unit 12.

[0017] The determination unit 12 determines whether or not blur is present in each equipment image supplied from the acquisition unit 11. The determination unit 12 supplies the equipment image and the determination result to the extraction unit 13.

[0018] The extraction unit 13 extracts, from among the multiple equipment images, equipment images that are determined to be free of blur as analysis targets for analyzing the state of equipment included as subjects in the equipment images.

[0019] As described above, the information processing device 1 according to this exemplary embodiment employs a configuration including an acquisition unit 11 that acquires a plurality of equipment images each including one or more pieces of equipment included as subjects in a vehicle surroundings image, a determination unit 12 that determines whether or not each equipment image supplied from the acquisition unit 11 is blurred, and an extraction unit 13 that extracts, from the plurality of equipment images, equipment images that are determined to be free of blur as analysis targets for analyzing the state of the equipment included as subjects in the equipment images.

[0020] Therefore, the information processing device 1 according to this exemplary embodiment has the effect of improving the accuracy of detecting defects in the equipment around the vehicle.

[0021] (Flow of Information Processing Method S1) The flow of the information processing method S1 according to this exemplary embodiment will be described with reference to Fig. 2. Fig. 2 is a flow chart showing the flow of the information processing method S1 according to this exemplary embodiment.

[0022] In step S11, the acquisition unit 11 acquires a plurality of equipment images each including one or more pieces of equipment among the plurality of pieces of equipment included as subjects in the vehicle surroundings image. The acquisition unit 11 supplies the acquired equipment images to the determination unit 12.

[0023] (Step S12) In step S12, the determination unit 12 determines whether or not there is blur in each equipment image supplied from the acquisition unit 11. The determination unit 12 supplies the equipment image and the determination result to the extraction unit 13.

[0024] (Step S13) In step S13, the extraction unit 13 extracts an equipment image determined to be free of blur from among the multiple equipment images as an analysis target for analyzing the state of equipment included as a subject in the equipment image.

[0025] As described above, the information processing method S1 according to this exemplary embodiment includes the steps of: step S11 in which the acquisition unit 11 acquires a plurality of equipment images, each of which includes one or more pieces of equipment included as subjects in the vehicle surroundings image; step S12 in which the determination unit 12 determines whether or not each of the equipment images provided by the acquisition unit 11 is blurred; and step S13 in which the extraction unit 13 extracts, from the plurality of equipment images, equipment images determined to be blur-free as analysis targets for analyzing the status of the equipment included as subjects in the equipment images. Therefore, the information processing method S1 according to this exemplary embodiment can achieve the same effects as the information processing device 1 described above.

[0026]

[0033] A second exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first exemplary embodiment are denoted by the same reference numerals, and their description will be omitted as appropriate.

[0027] (Overview of Information Processing Device 2) The information processing device 2 according to this exemplary embodiment is a device that analyzes the state of facilities around a vehicle. Although the vehicle is not particularly limited, this exemplary embodiment will describe a case where the information processing device 2 analyzes the state of facilities around a railway vehicle.

[0028] The term "surrounding equipment for railway vehicles" refers to equipment installed around railway vehicles to allow the vehicles to run. For example, the equipment is equipment for supplying power to railway vehicles. Examples of such equipment include wires, contact wires, hangers, ears, connectors, bolts, and insulators.

[0029] The state of the equipment is a state that can be determined from the appearance of the equipment. For example, the state of the equipment may be either a good state or a bad state. Furthermore, the good state and the bad state may differ depending on the equipment, and in this case, the equipment and the state may be defined in association with each other.

[0030] An example of a case where equipment and status are defined in association with each other will be described with reference to Fig. 3. Fig. 3 is a table showing an example of equipment and status in this exemplary embodiment.

[0031] 3, equipment and states are associated with each other. For example, equipment "W Year" is associated with the state "CC Misalignment" and the state "Lost Bolt," and equipment "Protector" is associated with the state "Lost Bolt."

[0032] The information processing device 2 analyzes whether the state of the equipment is in a state associated with the equipment based on the table of Fig. 3. As an example, the information processing device 2 analyzes whether the equipment "W Year" is in the state "CC Deviation" and whether the equipment "W Year" is in the state "Bolt Missing". As another example, the information processing device 2 analyzes whether the equipment "Protector" is in the state "Bolt Missing".

[0033] 3 indicates a state in which a malfunction has occurred in the equipment, so it can be said that the information processing device 2 analyzes whether or not the equipment is in a defective state.

[0034] Furthermore, in order to analyze the condition of the equipment, the information processing device 2 acquires a vehicle surroundings image, which is an image captured by a camera installed on an externally exposed surface of the railway vehicle and includes multiple pieces of equipment as subjects. The vehicle surroundings image may be an image captured while the vehicle is in motion. The externally exposed surface may include, for example, the top surface, bottom surface, left and right sides, and some or all of the front and rear sides of the railway vehicle. In this exemplary embodiment, the surface on which the camera is installed will be described mainly as the top surface of the railway vehicle, but is not limited to this. Hereinafter, "on the top surface of the railway vehicle" will also be simply referred to as "on the railway vehicle." The camera installed on the railway vehicle and the equipment as the subject will be described with reference to FIG. 4. FIG. 4 is a schematic diagram showing an example of a railway vehicle TR and equipment in this exemplary embodiment.

[0035] As shown in Fig. 4, a plurality of cameras (cameras CA1 to CA6) are installed on the railway vehicle TR, each capturing an image of equipment included in its angle of view and outputting the captured image. It is desirable that at least a portion of the ranges included in the angles of view of the plurality of cameras are different from each other. In other words, the range included in the angle of view of each camera (hereinafter also referred to as "capture range") may partially overlap with the range included in the angle of view of at least one other camera, but it is desirable that at least a portion of the ranges are different.

[0036] Although the installation manner of the multiple cameras is not particularly limited, one example is a configuration in which three cameras CA1 to CA3 are installed at different heights on the right side of the vehicle TR when the front of the vehicle TR is viewed from the direction of travel of the vehicle TR, as shown in Figure 4. Similarly, three cameras CA4 to CA6 are installed at different heights on the left side of the vehicle TR when the front of the vehicle TR is viewed from the direction of travel of the vehicle TR.

[0037] Furthermore, the timing at which cameras CA1 to CA6 photograph the equipment is not particularly limited. One example is a configuration in which cameras CA1 to CA6 each photograph the equipment at a predetermined interval. Another example is a configuration in which multiple cameras photograph the equipment synchronously at a predetermined interval. One example of such a configuration is a configuration in which cameras CA1 and CA4 photograph the equipment synchronously, cameras CA2 and CA5 photograph the equipment synchronously, and cameras CA3 and CA6 photograph the equipment synchronously. However, due to processing delays and the like, images photographed by multiple cameras synchronously are not necessarily photographed at exactly the same time. Furthermore, as one example, as shown in FIG. 4, cameras CA1 to CA6 photograph the equipment, including a hanger HA, an ear EA, and a trolley wire TW, as their subjects.

[0038] The information processing device 2 acquires vehicle surroundings images captured by the cameras CA1 to CA6. The configuration in which the information processing device 2 acquires the vehicle surroundings images is not particularly limited. One example is a configuration in which the vehicle surroundings images captured by the cameras CA1 to CA6 are stored in a recording medium, and the information processing device 2 acquires the vehicle surroundings images from the recording medium. Another configuration is a configuration in which the information processing device 2 and the cameras CA1 to CA6 are communicably connected via a network, and the information processing device 2 acquires the vehicle surroundings images via the network.

[0039] (Configuration of Information Processing Device 2) The configuration of the information processing device 2 will be described with reference to Fig. 5. Fig. 5 is a block diagram showing the configuration of the information processing device 2 according to this exemplary embodiment.

[0040] As shown in FIG. 5, the information processing device 2 includes a control unit 20, an input / output unit 27, a communication unit 28, and a storage unit 29.

[0041] The input / output unit 27 is an interface that receives user input and outputs data. For example, the input / output unit 27 supplies information indicating the received user input to the control unit 20 and outputs information supplied from the control unit 20. Examples of the input / output unit 27 include, but are not limited to, a keyboard, a mouse, a touchpad, a microphone, and a liquid crystal display.

[0042] The communication unit 28 is an interface that transmits and receives data via a network. For example, the communication unit 28 transmits data supplied from the control unit 20 to other devices, and supplies data received from other devices to the control unit 20. Examples of the communication unit 28 include, but are not limited to, communication chips for various communication standards such as Ethernet (registered trademark), Wi-Fi (registered trademark), and wireless communication standards for mobile data communication networks, and USB-compliant connectors.

[0043] The storage unit 29 stores data referenced by the control unit 20. Examples of data stored in the storage unit 29 include, but are not limited to, vehicle surroundings images, equipment images, and analysis results. Examples of the storage unit 29 include, but are not limited to, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0044] (Functions of control unit 20) The control unit 20 controls each component included in the information processing device 2. As shown in Fig. 5 , the control unit 20 also includes an acquisition unit 11, a determination unit 12, an extraction unit 13, an image connection unit 21, an equipment detection unit 22, a first analysis unit 23, and a second analysis unit 24. In this exemplary embodiment, the acquisition unit 11, the determination unit 12, the extraction unit 13, and the first analysis unit 23 are configured to realize an acquisition means, a determination means, an extraction means, and a first analysis means, respectively.

[0045] (Acquisition Unit 11) The acquisition unit 11 acquires data supplied from the input / output unit 27 or the communication unit 28. The acquisition unit 11 also acquires data stored in the storage unit 29.

[0046] As an example, the acquisition unit 11 acquires vehicle surroundings images from a plurality of cameras, i.e., cameras CA1 to CA6. The acquisition unit 11 also acquires a plurality of equipment images from the vehicle surroundings images whose capturing ranges at least partially overlap. Each equipment image includes one or more pieces of equipment as subjects.

[0047] (Determination Unit 12) The determination unit 12 determines whether or not an image is blurred. The determination unit 12 supplies the image and the determination result to the extraction unit 13.

[0048] As one example, the determination unit 12 determines whether or not there is blur in each equipment image. As another example, the determination unit 12 determines whether or not there is blur in equipment included as a subject in each equipment image. The method by which the determination unit 12 determines whether or not there is blur is not particularly limited. As one example, there is a method in which the determination unit 12 detects an edge portion of the equipment included as a subject in the equipment image and makes a determination based on whether the detected edge portion is clear.

[0049] In this exemplary embodiment, the determination unit 12 determines whether or not each equipment image is blurred using a determination model that receives an equipment image as input and outputs whether or not there is blur. The determination model is trained using a training data set that includes, for each of multiple types of equipment that may be included in the multiple equipment, equipment images that include that type of equipment as a subject and that are blurred, and equipment images that include that type of equipment as a subject and that are not blurred.

[0050] A configuration for training a determination model will be described with reference to Fig. 6 . Fig. 6 is a diagram showing a process for training a determination model MD1 in this exemplary embodiment. The determination model MD1 is a model that receives an equipment image and equipment information as input and outputs information indicating the presence or absence of blur. The equipment information is supplemental information related to the equipment, and includes, for example, information indicating the type of equipment, but is not limited to this. The process shown in Fig. 6 may be executed by another information processing device or by the control unit 20 of the information processing device 2.

[0051] As shown in FIG. 6 , first, an equipment image ep including equipment as a subject is associated with information fi indicating the presence or absence of blur in the equipment image ep, and equipment information ei indicating the type of equipment included in the equipment image ep. The information fi indicating the presence or absence of blur is, in other words, a correct label. Next, the image size of the equipment image ep is normalized, and contrast correction is performed to match the contrast between the multiple equipment images ep (for example, by performing contrast limited adaptive histogram equalization (CLAHE)), to generate a post-image-processing equipment image p_ep. A determination model MD1 is then trained using a training dataset that includes the post-image-processing equipment image p_ep, the information fi indicating the presence or absence of blur, and the equipment information ei. Specifically, when the equipment image p_ep and the equipment information ei are input to the determination model MD1, training is performed so that the correct label, information fi indicating the presence or absence of blur, is output.

[0052] Note that the information input to the determination model MD1 may include an equipment image but not equipment information. In this case, the training dataset includes an equipment image ep and information fi indicating the presence or absence of blur, but does not include equipment information ei. Furthermore, when an equipment image ep including equipment as a subject is input, the determination model MD1 is trained to output information fi indicating the presence or absence of blur in the equipment image ep. Similarly, in this case, the determination model MD1 may be trained using a post-image-processed equipment image p_ep obtained by performing image processing on the equipment image ep, instead of the equipment image ep.

[0053] In this way, the determination unit 12 determines whether or not there is blur in the equipment image using the determination model MD1. With this configuration, the determination unit 12 can suitably determine whether or not there is blur in the equipment image.

[0054] Furthermore, the determination unit 12 determines whether or not to determine whether or not the equipment image is blurred by referring to the type of equipment included as a subject in each equipment image. Note that known techniques can be used to determine the type of equipment included as a subject in an equipment image.

[0055] As an example, if the image of equipment "wire" is blurred, it is difficult to analyze the condition and it is difficult to determine that the equipment is "defective." Therefore, if the type of equipment included as a subject in an equipment image is "wire," the determination unit 12 does not determine whether or not the equipment image is blurred. On the other hand, if the type of equipment included as a subject in an equipment image is other than "wire," the determination unit 12 determines whether or not the equipment image is blurred. Hereinafter, the type of equipment for which the determination unit 12 determines whether or not it is blurred will be referred to as the first type, and the type of equipment for which the determination unit 12 does not determine whether or not it is blurred will be referred to as the second type.

[0056] Furthermore, the determination unit 12 may refer to the type of equipment included as a subject in each equipment image and the type of state of the equipment being analyzed to determine whether or not the equipment image is blurred. For example, when analyzing a certain defect in a certain equipment, the determination unit 12 performs a blur determination. On the other hand, when analyzing another defect different from the certain defect in the certain equipment, the determination unit 12 does not perform a blur determination.

[0057] (Extraction unit 13) The extraction unit 13 extracts images to be analyzed. As an example, the extraction unit 13 extracts, from among the multiple equipment images supplied from the determination unit 12, equipment images that have been determined by the determination unit 12 to be free of blur, as analysis targets for analyzing the state of equipment included as subjects in the equipment images. The extraction unit 13 supplies the extracted equipment images to the first analysis unit 23.

[0058] (Image connection unit 21) The image connection unit 21 connects images. As an example, the image connection unit 21 connects a plurality of vehicle surroundings images stored in the storage unit 29. The image connection unit 21 stores the connected vehicle surroundings images in the storage unit 29.

[0059] For example, the image connection unit 21 acquires a plurality of vehicle surroundings images captured by the camera CA1 from the storage unit 29. Next, the image connection unit 21 sorts the acquired vehicle surroundings images according to the date and time of capture. The image connection unit 21 then connects the rearranged vehicle surroundings images so that equipment included as subjects in each of the vehicle surroundings images are connected, thereby generating a connected vehicle surroundings image. The image connection unit 21 performs similar processing on the plurality of vehicle surroundings images captured by the other cameras CA2 to CA6, respectively, to generate a connected vehicle surroundings image.

[0060] (Equipment Detection Unit 22) The equipment detection unit 22 detects equipment included as a subject in an equipment image. The equipment detection unit 22 stores the equipment image including the detected equipment as a subject in the storage unit 29.

[0061] As an example, the equipment detection unit 22 detects a plurality of pieces of equipment included as subjects in the vehicle surroundings image stored in the storage unit 29. Furthermore, the equipment detection unit 22 stores in the storage unit 29 equipment images each including one or more pieces of equipment out of the plurality of detected pieces of equipment.

[0062] Furthermore, the equipment detection unit 22 detects a plurality of pieces of equipment included as subjects in each of a plurality of coupled vehicle surroundings images. As an example, the equipment detection unit 22 detects a plurality of pieces of equipment included as subjects in each of a plurality of coupled vehicle surroundings images generated from the vehicle surroundings images captured by each of the cameras CA1 to CA6. In this configuration, the equipment detection unit 22 generates a plurality of equipment images including the same subject from the plurality of coupled vehicle surroundings images. For example, when the cameras CA1 and CA4 each capture an image of the same hanger, the equipment detection unit 22 stores two equipment images including the hanger as a subject in the storage unit 29 based on the vehicle surroundings images output by the cameras CA1 and CA4.

[0063] The method by which the equipment detection unit 22 detects equipment is not limited. As one example, the equipment detection unit 22 detects equipment using a machine learning model that has been trained to take an equipment image as input and output an area of ​​the equipment detected by area extraction (e.g., PWC). As another example, the equipment detection unit 22 detects equipment using a machine learning model that has been trained to take an equipment image as input and output an area of ​​the equipment detected by object detection (e.g., SSD (Single Shot Multibox Detector)). In these cases, the equipment detection unit 22 may perform the above-described contrast correction on the combined vehicle surroundings image that is input to the machine learning model so that the contrast is the same as that of the equipment image during machine learning.

[0064] (First analysis unit 23) When the type of equipment included in an equipment image is a first type that determines whether or not blur is present, the first analysis unit 23 analyzes the state of the equipment included as a subject in the equipment image that is determined to be free of blur. As an example, the first analysis unit 23 analyzes the state of the equipment included as a subject in the equipment image that is supplied from the extraction unit 13 and extracted as an analysis target. The first analysis unit 23 stores the analysis result in the memory unit 29. The first analysis unit 23 may also output the analysis result via the input / output unit 27 or the communication unit 28.

[0065] There are no limitations on the method by which the first analysis unit 23 analyzes the state of the equipment. In this exemplary embodiment, the first analysis unit 23 analyzes the state of the equipment using a first analysis model. In this case, the first analysis model is trained using a training dataset that includes, as a training dataset, equipment of the same type as equipment included as a subject in the equipment image and that includes equipment images that are not blurred.

[0066] In this way, the first analysis unit 23 analyzes the state of the equipment using a first analysis model trained using a training dataset that includes equipment images without blurring and does not include equipment images with blurring, thereby improving the accuracy of analyzing the state of the equipment.

[0067] In addition, the first analysis unit 23 analyzes the condition of equipment that is included as a subject in a first equipment image out of multiple equipment images extracted as analysis targets, and also analyzes the condition of equipment that is the same as the equipment that is included as a subject in a second equipment image.

[0068] For example, assume that the image capturing ranges of the cameras CA1 and CA4 shown in Fig. 4 partially overlap, and the cameras CA1 and CA4 each capture an image of the same hanger. In this case, the hanger included as a subject in the vehicle surroundings image captured by the camera CA1 is the hanger as seen from the right side when the front of the vehicle TR is viewed from the direction of travel of the vehicle TR. On the other hand, the hanger included as a subject in the vehicle surroundings image captured by the camera CA4 is the hanger as seen from the left side when the front of the vehicle TR is viewed from the direction of travel of the vehicle TR.

[0069] In this case, it is assumed that the extraction unit 13 extracts as analysis targets both a first equipment image including a hanger as a subject, which is acquired from the vehicle surroundings image captured by camera CA1, and a second equipment image including as a subject the same hanger as the hanger included as a subject in the first equipment image, which is acquired from the vehicle surroundings image captured by camera CA4. In this case, the first analysis unit 23 analyzes the hanger included as a subject in the first equipment image and also analyzes the condition of the hanger included as a subject in the second equipment image. That is, when the front of the vehicle TR is viewed from the direction of travel of the vehicle TR, the first analysis unit 23 analyzes the condition of both the hanger as viewed from the right side and the left side. For example, even if a hanger defect can only be detected when viewed from the right side when the front of the vehicle TR is viewed from the direction of travel of the vehicle TR, the first analysis unit 23 can detect the defect in the hanger.

[0070] In this way, the first analysis unit 23 analyzes the equipment using multiple equipment images that include the equipment as a subject photographed from various angles, thereby improving the accuracy of detecting defects in equipment around the railway vehicle.

[0071] (Second analysis unit 24) If the type of equipment included in the equipment image is a second type that does not determine whether blur is present, the second analysis unit 24 analyzes the state of the equipment. The second analysis unit 24 stores the analysis results in the memory unit 29. The second analysis unit 24 may also output the analysis results via the input / output unit 27 or the communication unit 28.

[0072] There are no limitations on the method by which the second analysis unit 24 analyzes the state of the equipment. In this exemplary embodiment, the second analysis unit 24 analyzes the state of the equipment using a second analysis model. In this case, the second analysis model is trained using a training data set that includes, as a training data set, blurred equipment images that include the equipment as an object and unblurred equipment images that include the equipment as an object.

[0073] (Flow of Information Processing Method S2) The flow of the information processing method S2 according to this exemplary embodiment will be described with reference to Fig. 7. Fig. 7 is a flow chart showing the flow of the information processing method S2 according to this exemplary embodiment.

[0074] (Step S21) In step S21, the acquisition unit 11 acquires the vehicle surroundings images captured by the cameras CA1 to CA6, and stores the acquired vehicle surroundings images in the storage unit 29.

[0075] (Step S22) In step S22, the image connection unit 21 acquires a plurality of vehicle surroundings images stored in the storage unit 29. The image connection unit 21 also connects the acquired plurality of vehicle surroundings images. The image connection unit 21 stores the connected vehicle surroundings images in the storage unit 29.

[0076] (Step S23) In step S23, the equipment detection unit 22 acquires the plurality of coupled vehicle surroundings images stored in the storage unit 29. The equipment detection unit 22 also detects the plurality of pieces of equipment included as subjects in each of the plurality of coupled vehicle surroundings images. The equipment detection unit 22 stores the plurality of equipment images in the storage unit 29, each of which includes one or more pieces of equipment from the plurality of detected pieces of equipment.

[0077] (Step S24) The acquisition unit 11 acquires a plurality of equipment images stored in the storage unit 29. The acquisition unit 11 supplies the acquired equipment images to the determination unit 12.

[0078] (Step S25) The determination unit 12 determines whether to determine whether or not the equipment image is blurred, by referring to the type of equipment included as a subject in the equipment image supplied from the acquisition unit 11. That is, if the type of equipment is the first type, the determination unit 12 determines to determine whether or not the equipment image is blurred. On the other hand, if the type of equipment is the second type, the determination unit 12 determines not to determine whether or not the equipment image is blurred.

[0079] (Step S26) If it is determined in step S25 that the presence or absence of blur is to be determined (step S25: YES), in step S26, the determination unit 12 determines the presence or absence of blur using a determination model that inputs the equipment image supplied from the acquisition unit 11 and outputs the presence or absence of blur. The determination unit 12 supplies the equipment image and the determination result to the extraction unit 13.

[0080] (Step S27) In step S27, the extraction unit 13 refers to the determination result supplied from the determination unit 12, and extracts the equipment images determined to be free of blur as analysis targets for analyzing the state of the equipment included as subjects in the equipment images. The extraction unit 13 supplies the extracted equipment images to the first analysis unit 23.

[0081] (Step S28) In step S28, the first analysis unit 23 uses the first analytical model to analyze the state of the equipment included as a subject in the equipment image extracted as the analysis target supplied from the extraction unit 13. The first analysis unit 23 stores the analysis result in the storage unit 29.

[0082] (Step S29) If it is determined in step S25 not to determine the presence or absence of blur (step S25: NO), the determination unit 12 supplies the equipment image to the second analysis unit 24. In step S29, the second analysis unit 24 analyzes the state of the equipment included as a subject in the equipment image supplied from the determination unit 12, using the second analysis model.

[0083] (Effect 1 of information processing device 2) In this way, the information processing device 2 according to this exemplary embodiment determines whether or not there is blur for each of a plurality of equipment images, each of which includes one or more of the plurality of pieces of equipment included as subjects in the vehicle surroundings image, and extracts the equipment images determined to be free of blur as analysis targets for analyzing the state of the equipment included as subjects in the equipment images.

[0084] That is, even if there is blur in areas other than the equipment in the vehicle surroundings image, if there is no blur in the equipment, the information processing device 2 extracts an equipment image including the equipment as a subject for analysis. On the other hand, even if there is no blur in areas other than the equipment in the vehicle surroundings image, if there is blur in the equipment, the information processing device 2 does not extract an equipment image including the equipment as a subject for analysis.

[0085] Therefore, the information processing device 2 can improve the accuracy of detecting defects in the equipment around the railway vehicle.

[0086] (Effect 2 of information processing device 2) Here, one of the effects of this exemplary embodiment will be explained by comparing it with a comparative example that uses a learning model (hereinafter simply referred to as the "learning model") that is trained to take an equipment image as input and output the condition of the equipment included as a subject in the equipment image (whether the equipment is defective or not).

[0087] In the comparative example, when a blurred equipment image is input to the learning model, there is a problem that the accuracy of analyzing whether or not the equipment included as a subject in the equipment image is defective decreases. For example, when a blurred equipment image is input to the learning model, the learning model may analyze the equipment as not being defective when it is (overlooking the defect) or analyze the equipment as being defective when it is not (overdetecting the defect).

[0088] On the other hand, the information processing device 2 according to this exemplary embodiment uses a determination model to determine whether an equipment image is blurred, and analyzes the state of equipment included as a subject in the equipment image using a first analytical model for equipment images determined to be non-blurred. The first analytical model is trained using a training dataset that includes non-blurred equipment images that include the same type of equipment as the equipment in question as a subject.

[0089] That is, in the information processing device 2, equipment images determined to be non-blurred are input to a first analysis model trained using a training dataset including non-blurred images. Therefore, compared to the training model of the comparative example, it is possible to reduce oversights of defects where equipment is analyzed as not defective when it is actually defective, and to reduce overdetections where equipment is analyzed as defective when it is not. That is, the information processing device 2 can improve the accuracy of detecting defects in equipment around railway vehicles.

[0090] Furthermore, the information processing device 2 according to this exemplary embodiment determines whether or not to determine whether or not blur is present in each equipment image by referring to the type of equipment included as a subject in the equipment image. As described above, for example, the information processing device 2 does not determine whether or not blur is present in an equipment image that includes an equipment "wire" as a subject.

[0091] In the learning model, if an equipment image containing the equipment "wire" as a subject is blurred, it is difficult to determine that the equipment "wire" is defective (for example, the wire is about to break), so there is a possibility that the equipment "wire" will be analyzed as not being defective even though it is.

[0092] On the other hand, in the information processing device 2, for an equipment image that includes the equipment "wire" as a subject, the presence or absence of blur is not determined, and the equipment image is analyzed by the second analysis unit 24. Therefore, the processing load in the information processing device 2 can be reduced.

[0093] The present exemplary embodiment provides the same effect even when a vehicle other than a railway vehicle is used as the vehicle and the equipment surrounding the vehicle other than a railway vehicle is used as the equipment. For example, the present exemplary embodiment provides the same effect even when the vehicle is an automobile and the equipment surrounding the automobile (roads, traffic lights, electric wires, etc.) is used as the vehicle.

[0094] [Example of Software Implementation] Some or all of the functions of the information processing devices 1 and 2 may be implemented by hardware such as an integrated circuit (IC chip), or may be implemented by software.

[0095] In the latter case, the information processing devices 1 and 2 are realized, for example, by a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in Figure 8. The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for operating the computer C as the information processing devices 1 and 2. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing each function of the information processing devices 1 and 2.

[0096] The processor C1 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a tensor processing unit (TPU), a quantum processor, a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0097] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may also include a communication interface for transmitting and receiving data to and from other devices. The computer C may also include an input / output interface for connecting input / output devices such as a keyboard, a mouse, a display, and a printer.

[0098] The program P can also be recorded on a non-transitory, tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.

[0099] [Additional Note 1] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means disclosed in the above-described embodiments are also included in the technical scope of the present invention.

[0100] [Additional Note 2] Part or all of the above-described embodiment can also be described as follows: However, the present invention is not limited to the following described aspects.

[0101] (Supplementary Note 1) An information processing device comprising: an acquisition means for acquiring a plurality of equipment images each including one or more pieces of equipment among a plurality of pieces of equipment included as subjects in a vehicle surroundings image; a determination means for determining whether or not each equipment image is blurred; and an extraction means for extracting, from the plurality of equipment images, equipment images determined to be free of blur as analysis targets for analyzing the state of the equipment included as subjects in the equipment images.

[0102] (Supplementary Note 2) The information processing device according to Supplementary Note 1, further comprising a first analysis means for analyzing the state of equipment included as a subject in the equipment image extracted as the analysis target using a first analysis model, wherein the first analysis model is trained using a training dataset including equipment images that include equipment of the same type as the equipment as a subject and that are not blurred.

[0103] (Supplementary Note 3) The information processing device according to Supplementary Note 1 or 2, wherein the determination means determines the presence or absence of blur using a determination model that takes the equipment image as input and outputs the presence or absence of blur, and the determination model is trained using a training data set that includes, for each of multiple types of equipment that may be included in the multiple types of equipment, an equipment image that includes the type of equipment as a subject and is blurred, and an equipment image that includes the type of equipment as a subject and is not blurred.

[0104] (Supplementary Note 4) The information processing device according to any one of Supplementary Notes 1 to 3, wherein the determination means determines whether or not to determine whether or not blur is present in each equipment image by referring to the type of equipment included as a subject in the equipment image.

[0105] (Supplementary Note 5) The information processing device described in Supplementary Note 2, wherein the acquisition means acquires the plurality of equipment images from a plurality of vehicle surrounding images whose shooting ranges at least partially overlap, and the first analysis means analyzes the condition of equipment that is included as a subject in a first equipment image among the plurality of equipment images extracted as the analysis target, and analyzes the condition of equipment that is the same as the equipment that is included as a subject in a second equipment image.

[0106] (Supplementary Note 6) An information processing method including: an information processing device acquiring equipment images each including one or more pieces of equipment from among a plurality of pieces of equipment included as subjects in a vehicle surroundings image; determining whether or not each equipment image is blurred; and extracting, from among the plurality of equipment images, equipment images determined to be free of blur as analysis targets for analyzing the state of the equipment included as subjects in the equipment images.

[0107] (Supplementary Note 7) The information processing method according to Supplementary Note 6, further comprising: analyzing, using a first analytical model, the state of equipment included as a subject in the equipment image extracted as the analysis target; the first analytical model being trained using a training dataset including equipment images that include, as a subject, equipment of the same type as the equipment in question and that are not blurred.

[0108] (Appendix 8) A program for causing a computer to function as an information processing device, the program causing the computer to function as an acquisition means for acquiring a plurality of equipment images each including one or more pieces of equipment included as subjects in a vehicle surroundings image, a determination means for determining whether or not each equipment image is blurred, and an extraction means for extracting, from the plurality of equipment images, equipment images determined to be free of blur as analysis targets for analyzing the state of the equipment included as subjects in the equipment images.

[0109] (Supplementary Note 9) The program according to Supplementary Note 8, further causing the computer to function as a first analysis means that analyzes, using a first analysis model, the state of equipment included as a subject in an equipment image extracted as the analysis target, and the first analysis model is trained using a training dataset that includes, as a training dataset, equipment images that include equipment of the same type as the equipment as the subject and that are not blurred.

[0110] [Additional Note 3] Part or all of the above-described embodiment can also be expressed as follows.

[0111] An information processing device having at least one processor that executes an acquisition process for acquiring a plurality of equipment images each including one or more pieces of equipment included as subjects in a vehicle surroundings image, a determination process for determining whether or not each equipment image is blurred, and an extraction process for extracting, from the plurality of equipment images, equipment images determined to be free of blur as analysis targets for analyzing the state of the equipment included as subjects in the equipment images.

[0112] The information processing device may further include a memory that stores a program for causing the processor to execute the acquisition process, the determination process, and the extraction process. The program may be recorded on a computer-readable, non-transitory, tangible recording medium.

[0113] REFERENCE SIGNS LIST 1, 2 Information processing device 11 Acquisition unit 12 Determination unit 13 Extraction unit 23 First analysis unit

Claims

1. An information processing device comprising: an acquisition means for acquiring a plurality of equipment images each including one or more pieces of equipment from among a plurality of pieces of equipment included as subjects in a vehicle surroundings image; a determination means for determining whether or not each equipment image is blurred; and an extraction means for extracting, from among the plurality of equipment images, equipment images determined to be free of blur as analysis targets for analyzing the state of the equipment included as subjects in the equipment images.

2. The information processing device according to claim 1, further comprising a first analysis means for analyzing the state of equipment included as a subject in the equipment image extracted as the analysis target using a first analysis model, wherein the first analysis model is trained using a training dataset that includes equipment images that include as a subject the same type of equipment as the equipment in question and that are not blurred.

3. The information processing device described in claim 1 or 2, wherein the judgment means judges the presence or absence of blur using a judgment model that inputs the equipment image and outputs the presence or absence of blur, and the judgment model is trained using a training data set that includes, for each of multiple types of equipment that may be included in the multiple types of equipment, equipment images that include that type of equipment as a subject and are blurred, and equipment images that include that type of equipment as a subject and are not blurred.

4. The information processing device according to claim 1 or 2, wherein the determination means determines whether or not to determine whether or not blur is present in each equipment image by referring to the type of equipment included as a subject in the equipment image.

5. The information processing device described in claim 2, wherein the acquisition means acquires the plurality of equipment images from a plurality of vehicle surroundings images whose shooting ranges at least partially overlap, and the first analysis means analyzes the condition of equipment that is included as a subject in a first equipment image among the plurality of equipment images extracted as the analysis target, and analyzes the condition of equipment that is the same as the equipment that is included as a subject in a second equipment image.

6. An information processing method including: an information processing device acquiring equipment images each including one or more pieces of equipment from among a plurality of pieces of equipment included as subjects in a vehicle surroundings image; determining whether or not each equipment image is blurred; and extracting, from among the plurality of equipment images, equipment images determined to be free of blur as analysis targets for analyzing the status of the equipment included as subjects in the equipment images.

7. The information processing method of claim 6, further comprising: analyzing, using a first analytical model, the state of equipment included as a subject in the equipment image extracted as the analysis target; and wherein the first analytical model is trained using a training dataset that includes, as a training dataset, equipment images that include equipment of the same type as the equipment in question as a subject and that are not blurred.

8. A program for causing a computer to function as an information processing device, the program causing the computer to function as: an acquisition means for acquiring a plurality of equipment images each containing one or more pieces of equipment from among a plurality of pieces of equipment included as subjects in a vehicle surroundings image; a determination means for determining whether or not each equipment image is blurred; and an extraction means for extracting, from among the plurality of equipment images, equipment images determined to be free of blur as analysis targets for analyzing the status of the equipment included as subjects in the equipment images.

9. The program described in claim 8, further causing the computer to function as a first analysis means that analyzes the state of equipment included as a subject in the equipment image extracted as the analysis target using a first analysis model, and the first analysis model is trained using a training data set that includes equipment images that include equipment of the same type as the equipment as a subject and that are not blurred.

Images