Information processing device

Abstract

To determine whether an orientation of a road reflector is abnormal without using correct answer data.SOLUTION: A server device 20 executes processing based on a peripheral image being an image generated by photographing the periphery of a vehicle 10 with an imaging section 11 mounted on the vehicle 10. The server device 20 performs the steps of: acquiring the peripheral image in which a road reflector 50 arranged on a road on which the vehicle 10 travels appears; and determining that an orientation of the road reflector 50 is not abnormal when it is determined that at least one of another vehicle and a pedestrian approaching an arrangement place of the road reflector 50 from a direction different from the approaching direction of the vehicle 10 appears on a mirror surface of the road reflector 50 appearing on the acquired peripheral image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device. [Background technology]

[0002] A vehicle is provided with an imaging unit that captures images of the surroundings of the vehicle. When the vehicle is traveling on a road, a road reflector installed on the road may be captured in a surrounding image, which is an image generated by the imaging unit. Patent Document 1 discloses a technology for identifying a road reflector with poor visibility based on a surrounding image in which the road reflector is captured and correct answer data indicating that the road reflector is installed at a normal position, height, and angle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-113070 A Summary of the Invention [Problem to be solved by the invention]

[0004] The technology disclosed in Patent Document 1 was unable to determine whether the orientation of a road reflector was abnormal if there was no correct data for when the road reflector was installed at the correct position, height, and angle. [Means for solving the problem]

[0005] An information processing device that achieves the above-mentioned objective is an information processing device that performs processing based on a peripheral image, which is an image generated by an imaging unit mounted on a vehicle capturing an image of the surroundings of the vehicle, and executes the following steps: acquires the peripheral image in which road reflectors installed on the road along which the vehicle is traveling are captured; and, if it is determined that the mirror surface of the road reflector captured in the acquired peripheral image captures at least one of another vehicle and a pedestrian approaching the installation location of the road reflector from a direction different from the direction in which the vehicle approaches the installation location of the road reflector, determines that the orientation of the road reflector is not abnormal. Effect of the Invention

[0006] According to the present invention, it is possible to determine whether the orientation of a road reflector is abnormal or not based on a peripheral image without using correct answer data. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram used to explain an information processing system. [Diagram 2] FIG. 2 is a diagram illustrating an example of the configuration of the server device and the control device. [Diagram 3] FIG. 3 is an enlarged view of a portion of the surrounding image showing a road reflector in a normal orientation. [Figure 4] Figure 4 shows a close-up of a portion of the surrounding image that shows an abnormally oriented road reflector. [Diagram 5] FIG. 5 is a flowchart illustrating an example of a process executed by the control device. [Figure 6] FIG. 6 is a flowchart illustrating an example of a process executed by the server device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] <Embodiment> Hereinafter, an embodiment of an information processing device will be described with reference to FIGS. <Overall configuration of information processing system 1> 1, the information processing system 1 includes, for example, one or more vehicles 10 and one or more server devices 20. The server device 20 is an example of an information processing device. In detail, the server device 20 executes a process for determining whether or not the orientation of a road reflector 50 installed on a road is abnormal.

[0009] <Road reflector 50> The road reflector 50 is installed at a position on the road where visibility is poor. For example, the road reflector 50 is installed at an intersection or a curve. The location on the road where the road reflector 50 is installed is sometimes called the "installation location 60". The road reflector 50 assists the driver of the vehicle 10 approaching the installation location 60 from a first direction to check whether there are other vehicles or pedestrians approaching the installation location 60 from a second direction different from the first direction. In addition, the road reflector 50 assists the driver of the vehicle 10 approaching the installation location 60 from the second direction to check whether there are other vehicles or pedestrians approaching the installation location 60 from the first direction.

[0010] The first direction is, for example, the left direction in the width direction of the mirror surface when facing directly on the mirror surface of the road reflector 50. The second direction is, for example, the right direction in the width direction of the mirror surface when facing directly on the mirror surface of the road reflector 50. A vehicle 10 approaching the installation location 60 from the first direction is an example of a first vehicle. A vehicle 10 approaching the installation location 60 from the second direction is an example of a second vehicle. Note that the first direction and the second direction are merely examples and are not limited thereto, and the left-right relationship may be reversed.

[0011] In the example shown in Fig. 1, the installation location 60 of the road reflector 50 is an intersection where a first road 61 and a second road 62 intersect. In this case, the road reflector 50 is used by the driver of the vehicle 10 approaching the installation location 60 from the first road 61 to check whether there are other vehicles or pedestrians approaching the installation location 60 from the second road 62. The road reflector 50 is also used by the driver of the vehicle 10 approaching the installation location 60 from the second road 62 to check whether there are other vehicles or pedestrians approaching the installation location 60 from the first road 61.

[0012] 1, the direction approaching the installation location 60 of the road reflector 50 from a first road 61 corresponds to the first direction, and the direction approaching the installation location 60 from a second road 62 corresponds to the second direction.

[0013] <Vehicle 10 Configuration> The vehicle 10 includes, for example, an imaging unit 11, a position information acquisition unit 12, and a control device 30.

[0014] The imaging unit 11 is, for example, a digital camera using a solid-state imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). One example of the imaging unit 11 is a digital camera constituting a drive recorder. The imaging unit 11 captures images of the surroundings of the vehicle 10. Depending on the installation position of the imaging unit 11 on the vehicle 10, the imaging unit 11 can capture images of the front, rear, and sides of the vehicle 10. The imaging unit 11 generates a surrounding image by capturing images of the surroundings of the vehicle 10. When the vehicle 10 approaches an installation location 60 of a road reflector 50, the road reflector 50 may be captured in the surrounding image generated by the imaging unit 11.

[0015] The position information acquisition unit 12 is, for example, a Global Navigation Satellite System (GNSS) receiver. The position information acquisition unit 12 identifies the position of the vehicle 10 based on a signal received from a GNSS satellite, and acquires position information indicating the position of the vehicle 10.

[0016] The control device 30 is connected to the server device 20 via a communication network 40 so as to be able to transmit and receive information. The communication network 40 includes, for example, the Internet, a wide area network (WAN), a local area network (LAN), a provider terminal, a wireless communication network, a wireless base station, a dedicated line, etc. When a road reflector 50 is captured in the surrounding image generated by the imaging unit 11, the control device 30 transmits the surrounding image to the server device 20.

[0017] <Configuration of the control device 30> As shown in FIG. 2, the control device 30 includes, for example, a processing circuit 31, a communication unit 32, and a storage unit 33. The processing circuit 31 is configured with, for example, a hardware processor such as a CPU (Central Processing Unit). The hardware processor executes a program, which allows the processing circuit 31 to execute various processes. The program may be stored in the storage unit 33 in advance. The storage unit 33 is realized by, for example, a non-transitory storage medium such as an HDD (Hard Disk Drive) or a flash memory. The communication unit 32 includes, for example, a communication circuit or a communication module, and transmits and receives various data to and from the server device 20 and other devices based on the control of the processing circuit 31.

[0018] The storage unit 33 stores, for example, a trained model 331. The trained model 331 is a model that performs machine learning to determine whether or not the road reflector 50 is reflected in the peripheral image generated by the imaging unit 11. An example of the trained model 331 is a neural network. When the peripheral image is input to the trained model 331 as an input variable, the trained model 331 outputs an output variable indicating the probability that the road reflector 50 is reflected in the peripheral image. For example, the trained model 331 outputs a larger value as the output variable as the probability that the road reflector 50 is reflected in the peripheral image is higher. In this case, when the output variable is equal to or greater than a threshold value, the processing circuit 31 of the control device 30 determines that the road reflector 50 is reflected in the peripheral image. On the other hand, when the output variable is less than the threshold value, the processing circuit 31 determines that the road reflector 50 is not reflected in the peripheral image. The processing circuit 31 causes the communication unit 32 to transmit to the server device 20 the surrounding image in which it has been determined that the road reflector 50 is included, and the position information of the position where the surrounding image was captured.

[0019] <Configuration of the server device 20> As shown in FIG. 2, the server device 20 includes, for example, a processing circuit 21, a communication unit 22, and a storage unit 23. The processing circuit 21 is configured with, for example, a hardware processor such as a CPU. The hardware processor executes a program, which allows the processing circuit 21 to execute various processes. The program may be stored in the storage unit 23 in advance. The storage unit 23 is realized by, for example, a non-transitory storage medium such as an HDD or a flash memory. The communication unit 22 includes, for example, a communication circuit or a communication module, and transmits and receives various data to and from the control device 30 and other devices based on the control of the processing circuit 21.

[0020] The storage unit 23 stores, for example, a trained model 231 and road reflector information 232. The trained model 231 is a machine learning model that performs instance segmentation of objects reflected on the mirror surface of the road reflector 50. An example of the trained model 231 is a convolutional neural network (CNN). When a surrounding image is input to the trained model 231 as an input variable, the trained model 231 outputs a segmentation result indicating each of the objects reflected on the mirror surface in the range of pixels constituting the mirror surface of the road reflector 50 in the surrounding image. In the segmentation, for example, other vehicles, pedestrians, bicycles, road trees, roads, sidewalks, traffic lights, dividing lines, road signs, buildings, etc. that may exist around the road reflector 50 are identified.

[0021] Hereinafter, in the instance segmentation in the trained model 231, it is assumed that it is possible to distinguish whether a vehicle reflected on the mirror surface of the road reflector 50 is the subject vehicle or another vehicle by distinguishing between objects reflected on the mirror surface of the road reflector 50 and objects reflected on areas other than the mirror surface of the road reflector 50 in the surrounding image. The subject vehicle is the vehicle 10 equipped with the imaging unit 11 that generated the surrounding image, which is the original data of the segmentation result.

[0022] Specifically, when the mirror surface of the road reflector 50 in the peripheral image faces forward, the trained model 231 identifies the vehicle reflected in the mirror surface of the road reflector 50 as the vehicle itself. When the road reflector 50 is tilted from the normal orientation, the mirror surface of the road reflector 50 faces the vehicle 10 equipped with the imaging unit 11. Therefore, the mirror surface of the road reflector 50 facing forward is reflected in the peripheral image. On the other hand, when the mirror surface of the road reflector 50 in the peripheral image does not face forward, the trained model 231 identifies the vehicle reflected in the mirror surface of the road reflector 50 as another vehicle. When the road reflector 50 is in the normal orientation, the mirror surface of the road reflector 50 does not face the vehicle 10 equipped with the imaging unit 11. Therefore, the mirror surface of the road reflector 50 not facing forward is reflected in the peripheral image. Specifically, the road reflector 50 with its mirror surface facing the center between the first direction and the second direction is reflected in the peripheral image. The mirror surface of the road reflector 50 is basically circular. The trained model 231 distinguishes between the vehicle itself and other vehicles, for example, depending on whether the mirror surface of the road reflector 50 is a perfect circle or an ellipse. The road reflector information 232 will be described in detail later.

[0023] When the processing circuit 21 receives a surrounding image and position information from the control device 30, the processing circuit 21 inputs the surrounding image to the trained model 231. Then, the processing circuit 21 obtains a segmentation result as an output of the trained model 231. Based on the segmentation result, the processing circuit 21 determines whether or not at least one of another vehicle and a pedestrian is reflected on the mirror surface of the road reflector 50.

[0024] When the segmentation result includes at least one of another vehicle and a pedestrian, the processing circuit 21 determines that at least one of another vehicle and a pedestrian is reflected on the mirror surface of the road reflector 50. When the processing circuit 21 determines that at least one of another vehicle and a pedestrian is reflected on the mirror surface of the road reflector 50, it determines that the orientation of the road reflector 50 is not abnormal.

[0025] 3 and 4 show an enlarged view of the mirror surface range 80 of the road reflector 50 in the peripheral image captured and generated by the imaging unit 11 of the vehicle 10 traveling on the first road 61. As shown in FIG. 3, when the orientation of the road reflector 50 is normal, the mirror surface range 80 includes the second road 62, another vehicle 91 traveling on the second road 62, and a pedestrian 92 walking on the second road 62. Therefore, the segmentation result includes other vehicles, pedestrians, and roads. Therefore, the processing circuit 21 determines that at least one of the other vehicle 91 and the pedestrian 92 is reflected on the mirror surface of the road reflector 50 based on the segmentation result, while determining that the orientation of the road reflector 50 is not abnormal.

[0026] As shown in FIG. 4, when the orientation of the road reflector 50 is abnormal, the second road 62, other vehicles 91 traveling on the second road 62, and pedestrians 92 walking on the second road 62 are not reflected in the mirror surface range 80. On the other hand, depending on the orientation of the road reflector 50, only the sky and roadside trees are reflected. Note that the mirror surface range 80 when the orientation of the road reflector 50 is abnormal is an example and is not limited to this, and the mirror surface range 80 may include the vehicle 10 as the vehicle traveling on the first road 61, the first road 61, or only the ground, and there are various other modes. In any case, the segmentation result when the orientation of the road reflector 50 is abnormal does not include other vehicles and pedestrians. Therefore, the processing circuit 21 determines that the other vehicles 91 and pedestrians 92 are not reflected on the mirror surface of the road reflector 50 based on the segmentation result.

[0027] On the other hand, even if the orientation of the road reflector 50 is normal, if there are no other vehicles 91 or pedestrians 92 on the second road 62, the other vehicles 91 or pedestrians 92 are not reflected in the mirror surface range 80. Accordingly, when the processing circuit 21 determines that neither the other vehicles 91 nor the pedestrians 92 are reflected on the mirror surface of the road reflector 50, it derives the ratio of the part reflecting the road to the mirror surface range 80 of the road reflector 50 reflected in the peripheral image based on the segmentation result. The processing circuit 21 determines whether the derived ratio is equal to or greater than a predetermined ratio.

[0028] If the orientation of the road reflector 50 is abnormal, the road may not be reflected at all on the mirror surface of the road reflector 50, or the above-mentioned ratio may be very small even if the road is reflected. In such a case, even if a vehicle or pedestrian passes through the road, the vehicle or pedestrian may not be reflected on the mirror surface of the road reflector 50. Therefore, for example, a predetermined ratio is set as a criterion for determining whether or not a vehicle or pedestrian passing through the road is reflected on the mirror surface.

[0029] The predetermined ratio is determined in advance based on the surrounding images that satisfy the conditions (A1) and (A2), for example. (A1) The imaging unit 11 captured and generated an image in a scene in which the orientation of the road reflector 50 was normal.

[0030] (A2) There are no other vehicles 91 or pedestrians 92 in the second direction. Specifically, the predetermined ratio is the ratio of road pixels to the pixels in the mirror surface range 80 of the road reflector 50 that are captured in the peripheral image that satisfies the conditions (A1) and (A2). For example, the predetermined ratio is a statistical value of the ratio calculated based on multiple peripheral images that are peripheral images at all positions and that satisfy the conditions (A1) and (A2).

[0031] When the processing circuit 21 determines that the derived ratio is equal to or greater than the predetermined ratio, it determines that the orientation of the road reflector 50 is not abnormal. On the other hand, when the processing circuit 21 determines that the derived ratio is less than the predetermined ratio, it determines that the orientation of the road reflector 50 is abnormal.

[0032] The processing circuit 21 updates the road reflector information 232 based on the judgment result. The road reflector information 232 includes one or more records in which identification information capable of identifying the road reflector 50, information indicating the approach direction to the installation location 60 of the road reflector 50, and information indicating the judgment result of the orientation of the road reflector 50 are associated with each other. The identification information capable of identifying the road reflector 50 is, for example, information indicating the installation location 60 of the road reflector 50.

[0033] The processing circuit 21 searches the road reflector information 232 using the position information acquired from the control device 30 as a search key to identify the record related to the road reflector 50 to be judged. Specifically, the processing circuit 21 identifies the record of the road reflector 50 having the identification information indicating that the road reflector 50 is within a predetermined distance from the position information as the record of the road reflector 50 to be judged. In addition, the processing circuit 21 identifies the approach direction to the installation location 60 of the road reflector 50 for the vehicle 10 that transmitted the position information from the positional relationship between the position information and the identification information. As described above, the approach direction to the installation location 60 of the road reflector 50 is the first direction or the second direction. The processing circuit 21 updates the road reflector information 232 by generating or updating the record of the road reflector 50 to be judged based on the identified approach direction and the judgment result of the orientation of the road reflector 50.

[0034] Based on the updated road reflector information 232, the processing circuit 21 determines that the orientation of the road reflector 50 for which both conditions (B1) and (B2) are satisfied is normal.

[0035] (B1) Based on a surrounding image generated by the imaging unit 11 of a first vehicle approaching the installation location 60 of the road reflector 50 from a first direction, it is determined that the orientation of the road reflector 50 is not abnormal.

[0036] (B2) Based on a surrounding image generated by the imaging unit 11 of a second vehicle approaching the installation location 60 of the road reflector 50 from a second direction, it was determined that the orientation of the road reflector 50 was not abnormal.

[0037] The processing circuit 21 judges whether or not both of the conditions (B1) and (B2) are satisfied for each record of the updated road reflector information 232. Then, the processing circuit 21 judges that the orientation of the road reflector 50 for which both of the conditions are satisfied is normal.

[0038] <Processing of the control device 30> 5, a series of processes executed by the control device 30 to transmit a surrounding image including the road reflector 50 to the server device 20 will be described. The series of processes are executed by the processing circuit 31 at each predetermined control cycle.

[0039] The processing circuit 31 acquires a surrounding image generated by the imaging unit 11 (step S100). The processing circuit 31 judges whether or not the road reflector 50 is captured in the acquired surrounding image (step S102). When the processing circuit 31 judges that the road reflector 50 is not captured in the surrounding image (step S102; NO), there is no information to be transmitted to the server device 20, and the series of processes is terminated. When the processing circuit 31 judges that the road reflector 50 is captured in the surrounding image (step S102; YES), the processing circuit 31 acquires position information from the position information acquisition unit 12 (step S104). The position information acquired by the processing circuit 31 in step S104 is the position information of the vehicle 10 when the surrounding image acquired in step S100 was generated by the imaging unit 11. The processing circuit 31 causes the communication unit 32 to transmit the acquired surrounding image and the position information to the server device 20 (step S106). After that, the processing circuit 31 terminates the series of processes.

[0040] <Processing of the server device 20> 6, a series of processes executed by the server device 20 to determine whether the orientation of the road reflector 50 is normal or not based on the peripheral image received from the vehicle 10 will be described. When the communication unit 22 receives the peripheral image from the vehicle 10, the processing circuit 21 of the server device 20 executes the series of processes.

[0041] The processing circuit 21 acquires the peripheral image and the position information transmitted from the control device 30 of the vehicle 10 (step S200). The processing circuit 21 inputs the acquired peripheral image to the trained model 231 to acquire the segmentation result as the output of the trained model 231 (step S202). The processing circuit 21 determines whether or not at least one of the other vehicle 91 and the pedestrian 92 is reflected on the mirror surface of the road reflector 50 reflected in the peripheral image based on the segmentation result (step S204). If at least one of the other vehicle and the pedestrian is included in the segmentation result, the processing circuit 21 determines that at least one of the other vehicle 91 and the pedestrian 92 is reflected on the mirror surface of the road reflector 50 reflected in the peripheral image. In addition, if neither the other vehicle nor the pedestrian is included in the segmentation result, the processing circuit 21 determines that the other vehicle 91 and the pedestrian 92 are not reflected on the mirror surface of the road reflector 50 reflected in the peripheral image.

[0042] When the processing circuit 21 judges that at least one of the other vehicle 91 and the pedestrian 92 is reflected on the mirror surface of the road reflector 50 reflected in the peripheral image (step S204; YES), the processing circuit 21 judges that the orientation of the road reflector 50 is not abnormal (step S206). When the processing circuit 21 judges that the other vehicle 91 and the pedestrian 92 are not reflected on the mirror surface of the road reflector 50 reflected in the peripheral image (step S204; NO), the processing circuit 21 derives the ratio of the part in which the road is reflected to the range 80 of the mirror surface of the road reflector 50 reflected in the peripheral image (step S208). The processing circuit 21 judges whether the derived ratio is equal to or greater than a predetermined ratio (step S210). When the derived ratio is equal to or greater than a predetermined ratio (step S210; YES), the processing circuit 21 judges that the orientation of the road reflector 50 is not abnormal (step S206). If the derived ratio is less than a predetermined ratio (step S210; NO), the processing circuit 21 determines that the orientation of the road reflector 50 is abnormal (step S212).

[0043] The processing circuit 21 updates the road reflector information 232 based on the result of the judgment in step S206 or step S212 (step S214). Next, the processing circuit 21 judges that the orientation of the road reflector 50 that satisfies both the conditions (B1) and (B2) is normal based on the updated road reflector information 232 (step S218). Then, the processing circuit 21 ends the series of processes.

[0044] <Actions and Effects of the Embodiments> According to the above embodiment, the following actions and effects can be obtained. (1) When the vehicle 10 is traveling, the imaging unit 11 generates a surrounding image by capturing an image of the surroundings of the vehicle 10. When the vehicle 10 approaches an installation location 60 of a road reflector 50, the road reflector 50 may appear in the surrounding image generated by the imaging unit 11 of the vehicle 10. The control device 30 acquires the surrounding image acquired from the imaging unit 11. At this time, when the control device 30 determines that the road reflector 50 appears in the acquired surrounding image, it transmits the surrounding image to the server device 20.

[0045] The processing circuit 21 of the server device 20 acquires the peripheral image transmitted from the control device 30. Then, the processing circuit 21 judges whether or not at least one of another vehicle 91 and a pedestrian 92 is reflected on the mirror surface of the road reflector 50 reflected in the acquired peripheral image. When the processing circuit 21 judges that at least one of another vehicle 91 and a pedestrian 92 is reflected on the mirror surface of the road reflector 50 reflected in the acquired peripheral image, it judges that the orientation of the road reflector 50 is not abnormal. According to this configuration, the server device 20 can judge whether or not the orientation of the road reflector 50 is abnormal by using the peripheral image generated by the imaging unit 11, without using the correct answer data when the road reflector 50 is attached at the normal position, height, and angle.

[0046] (2) When a vehicle 10 is approaching the installation location 60 of the road reflector 50 from the first direction, there may be no other vehicles or pedestrians approaching the installation location 60 from the second direction. In this case, even if the orientation of the road reflector 50 is normal, other vehicles and pedestrians will not be reflected on the mirror surface of the road reflector 50. If the orientation of the road reflector 50 is normal, the road connected to the installation location 60 should be reflected on the mirror surface of the road reflector 50. In other words, if the road is not reflected on the mirror surface of the road reflector 50, or if the road is reflected on the mirror surface but most of the mirror surface reflects something other than the road, the orientation of the road reflector 50 can be considered to be abnormal.

[0047] Therefore, the processing circuit 21 derives the ratio of the portion of the road reflected in the range 80 of the mirror surface of the road reflector 50 reflected in the peripheral image. If the derived ratio is equal to or greater than a predetermined ratio, the processing circuit 21 determines that the orientation of the road reflector 50 is not abnormal. With this configuration, the server device 20 can determine whether the orientation of the road reflector 50 is abnormal or not even if there are no other vehicles 91 or pedestrians 92 around the installation location 60 of the road reflector 50.

[0048] (3) The processing circuit 21 refers to the road reflector information 232, and when both conditions (B1) and (B2) are satisfied, it determines that the orientation of the road reflector 50 is normal. Here, even if the road reflector 50 is normal when viewed from the first direction, it may not be normal when viewed from the second direction. In other words, the orientation of the road reflector 50 is normal when neither the first direction nor the second direction is abnormal. With this configuration, the server device 20 can determine whether the orientation of the road reflector 50 is normal or not based on the road reflector information 232.

[0049] <Example of change> The above embodiment may be modified as follows: The above embodiment and each of the following modifications may be combined with each other within a range that does not cause technical contradiction.

[0050] The imaging unit 11 can generate a video consisting of multiple surrounding images that are consecutive in time series. Therefore, the processing circuit 31 may transmit a video based on the surrounding images that are determined to include the road reflector 50 to the server device 20. In this case, the processing circuit 31 generates a video consisting of the surrounding images that are consecutively captured and generated in time series from a predetermined time before to a predetermined time after the timing when the surrounding images that are determined to include the road reflector 50 are captured and generated. Then, in step S106 in FIG. 5, the processing circuit 31 transmits the generated video to the server device 20 together with the position information.

[0051] The processing circuit 21 of the server device 20 judges, based on the acquired video, whether or not at least one of another vehicle 91 and a pedestrian 92 is reflected on the mirror surface of the road reflector 50 captured in the video. Specifically, when the video satisfies the condition (C), the processing circuit 21 judges that at least one of another vehicle 91 and a pedestrian 92 is reflected on the mirror surface of the road reflector 50 captured in the peripheral image.

[0052] (C) The state in which at least one of another vehicle 91 and a pedestrian 92 is reflected on the mirror surface of the road reflector 50 continues for a predetermined period of time or longer. 6, the processing circuitry 21 inputs each of the peripheral images constituting the moving image as an input variable to the trained model 231. The processing circuitry 21 acquires each segmentation result as an output of the trained model 231. In the following description, a plurality of segmentation results corresponding to peripheral images captured and generated consecutively in time series are also referred to as consecutive segmentation results.

[0053] The processing circuit 21 judges whether or not at least one of another vehicle and a pedestrian is included on the mirror surface of the road reflector 50 for each of the consecutive segmentation results. The processing circuit 21 judges that the condition (C) is satisfied when at least one of another vehicle and a pedestrian is included in all of the consecutive segmentation results. The processing circuit 21 also judges that the condition (C) is not satisfied when no other vehicle or pedestrian is included in any of the consecutive segmentation results. The processing circuit 21 may also judge that the condition (C) is satisfied when at least one of another vehicle and a pedestrian is continuously included in a predetermined number of the consecutive segmentation results based on the relationship between the frame rate of the video and a predetermined time. In this case, the predetermined number is the product of the frame rate and the predetermined time.

[0054] Here, if the orientation of the road reflector 50 is normal, other vehicles 91 and pedestrians 92 approaching the installation location 60 of the road reflector 50 from a distance should be continuously reflected on the mirror surface of the road reflector 50. If the video satisfies the condition (C), the processing circuit 21 judges that at least one of the other vehicles 91 and the pedestrians 92 is reflected on the mirror surface of the road reflector 50 reflected in the peripheral image (step S204; YES). Then, if the video satisfies the condition (C), the processing circuit 21 judges that the orientation of the road reflector 50 is not abnormal (step S206). Therefore, the server device 20 can judge whether the orientation of the road reflector 50 is abnormal or not based on the video.

[0055] The processing circuit 21 may not execute the processes of steps S214 to S128. The processing circuit 21 may execute the processes of steps S214 to S128 as a series of processes at predetermined time intervals, separately from the series of processes shown in Fig. 6. In this case, the predetermined time is a time that is long enough to collect a sufficient number of judgment results of the road reflector 50.

[0056] The processing circuit 21 may output a part or all of the judgment results obtained in steps S206, S212, and S218 to an information terminal of a road operator. This allows the operator to use the judgment results regarding the orientation of the road reflector 50 for maintenance management of the road reflector 50.

[0057] The processing circuit 31 may determine whether or not a road reflector 50 is reflected in the surrounding image based on the results obtained by general image analysis, instead of or in addition to the process of determining whether or not a road reflector 50 is reflected in the surrounding image using the trained model 331.

[0058] The processing circuit 21 may determine that at least one of another vehicle 91 and a pedestrian 92 is reflected on the mirror surface of the road reflector 50 based on the results obtained by general image analysis, instead of or in addition to the process of obtaining segmentation results using the trained model 231.

[0059] The processing circuit 31 may transmit a peripheral image only when the server device 20 requests the provision of the peripheral image. In this case, the processing circuit 31 transmits position information to the server device 20 at a predetermined time interval. When the vehicle 10 is located near the road reflector 50 based on the received position information, the server device 20 requests the vehicle 10 to provide a peripheral image. The processing circuit 31 transmits the peripheral image generated at the time of the request to the server device 20.

[0060] Some of the various determinations performed by the processing circuit 31 may be performed by the processing circuit 21 of the server device 20. In this case, every time the imaging unit 11 generates a peripheral image, the processing circuit 31 transmits to the server device 20 the peripheral image in association with the position information acquired by the position information acquisition unit 12 when the peripheral image was generated by the imaging unit 11. The processing circuit 21 determines whether or not the road reflector 50 is captured in the peripheral image based on the peripheral image received from the control device 30. When the processing circuit 21 determines that the road reflector 50 is captured in the peripheral image, it executes the process from step S202 onward among the series of processes shown in FIG. 6.

[0061] The control device 30 of the vehicle 10 may execute a process corresponding to the series of processes shown in Fig. 6. In this case, the control device 30 determines whether the orientation of the road reflector 50 is abnormal. In other words, the control device 30 functions as an information processing device.

[0062] Some or all of the components of the processing circuit 21 and the processing circuit 31 may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by a combination of software and hardware. [Explanation of symbols]

[0063] 1...information processing system, 10...vehicle, 11...imaging unit, 12...location information acquisition unit, 20...server device, 21,31...processing circuit, 22,32...communication unit, 23,33...memory unit, 30...control device, 40...communication network, 50...road reflector, 60...installation location, 61...first road, 62...second road, 80...range, 91...other vehicles, 92...pedestrians, 231,331...trained model, 232...road reflector information.

Claims

1. An information processing device that performs processing based on a surrounding image that is an image generated by an imaging unit mounted on a vehicle by imaging the surroundings of the vehicle, acquiring the surrounding image in which road reflectors installed on a road along which the vehicle is traveling are captured; When it is determined that at least one of another vehicle and a pedestrian approaching the installation location of the road reflector from a direction different from the direction in which the vehicle approaches the installation location of the road reflector is captured on the mirror surface of the road reflector captured in the acquired peripheral image, it is determined that the orientation of the road reflector is not abnormal. Information processing device.

2. acquiring a video comprising a sequence of the peripheral images; When it is determined that the duration of a state in which at least one of the other vehicle and the pedestrian is reflected on the mirror surface of the road reflector captured in the video is equal to or longer than a predetermined time, it is determined that at least one of the other vehicle and the pedestrian is reflected on the mirror surface of the road reflector captured in the peripheral image. The information processing device according to claim 1 .

3. Deriving a ratio of a portion of the road to a range of the mirror surface of the road reflector captured in the peripheral image; If the derived ratio is equal to or greater than a predetermined ratio, it is determined that the orientation of the road reflector is not abnormal. The information processing device according to claim 1 .

4. It is determined that the orientation of the road reflector is not abnormal based on the surrounding image generated by the imaging unit of a first vehicle approaching the installation location of the road reflector from a first direction and imaging the surroundings of the first vehicle; It is determined that the orientation of the road reflector is not abnormal based on the surrounding image generated by the imaging unit of the second vehicle approaching the installation location from a second direction that is different from the first direction and imaging the surroundings of the second vehicle; If any of the above is true, the direction of the road reflector is determined to be normal. The information processing device according to claim 1 .

Images