STATE DETERMINATION DEVICE, STATE DETERMINATION SYSTEM, STATE DETERMINATION METHOD, AND PROGRAM

Abstract

In order to improve the determination accuracy of the state of a road, this state determination device is provided with: multiple determination models trained using training data in which at least one of the state of a mobile body mounted with an imaging device for capturing road images, the state of the road, and the external environment, differs; and an output combining means which, for an inputted road image, combines output from the multiple determination models.

Description

[Technical field]

[0001] The present invention relates to the processing of road related information, and in particular to determining road conditions. [Background technology]

[0002] In local governments and the like, the management of roads under their jurisdiction requires a lot of effort and expense. Therefore, a device for collecting road information has been proposed (for example, see Patent Document 1).

[0003] The road measurement device described in Patent Document 1 uses image data of the road surface to determine the deterioration level of the road, and outputs a map in which the deterioration level is mapped to the presented reward value. Note that the road measurement device described in Patent Document 1 uses a judgment model that uses image data as input to judge the deterioration. This judgment model is constructed by a machine learning technique. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-079166 A Summary of the Invention [Problem to be solved by the invention]

[0005] Road images are captured outdoors under various weather conditions and at various times of the day, so the captured images are captured under a variety of conditions.

[0006] When images captured under various shooting conditions are judged using a single judgment model, it is difficult to ensure the accuracy of the judgment.

[0007] The reason for this will be explained using weather as an example of a shooting condition.

[0008] The decision model performs learning before making a decision.

[0009] For example, a determination model trained using images taken on a sunny day can make appropriate judgments about images taken on a sunny day, but a determination model trained using images taken on a sunny day may not be able to make appropriate judgments about images taken on a cloudy or rainy day.

[0010] Alternatively, a determination model trained using images of multiple weather conditions, such as sunny, cloudy, and rainy, can determine images in each weather condition with a certain degree of accuracy. However, it is difficult to improve the accuracy of determination for images in all weather conditions even with such a determination model.

[0011] In this way, when one determination model is used to determine images for all weather conditions, it is difficult to improve the accuracy of the determination.

[0012] The technology described in Patent Document 1 uses one determination model. Therefore, the technology described in Patent Document 1 has a problem in that it is difficult to improve the accuracy of determining the road condition.

[0013] An object of the present invention is to provide a condition determining device or the like that solves the above problems and improves the accuracy of determining road conditions. [Means for solving the problem]

[0014] A state determination device according to one embodiment of the present invention includes: A plurality of judgment models each trained using training data differing in at least one of the state of a moving body equipped with an image capturing device for capturing an image of a road, the state of a road, and the external environment; an output summing means for summing outputs from a plurality of decision models for an input road image; Includes.

[0015] A state determination system according to one embodiment of the present invention includes: The above state determination device, an image capturing device that captures an image of the object to be determined and outputs the image to the state determination device, and / or a storage device that stores the image of the object to be determined; A display device that displays the road condition output by the condition determination device; Includes.

[0016] A state determination method according to one aspect of the present invention includes: A state determination device including a plurality of determination models each trained using teacher data differing in at least one of the state of a moving body equipped with an image capture device for capturing an image of a road, the state of a road, and the external environment, The outputs from multiple decision models for an input road image are summed.

[0017] A state determination method according to one aspect of the present invention includes: A state determination device executes the state determination method described above, The photographing device captures an image of the object to be determined and outputs it to the state determination device, and / or the storage device stores the image of the object to be determined; A display device displays the road condition output by the condition determination device.

[0018] In one aspect of the present invention program teeth, A computer includes a plurality of judgment models each trained using teacher data that differs in at least one of the state of a moving body equipped with an image capturing device for capturing an image of a road, the state of a road, and the external environment, A process that combines the outputs of multiple decision models for an input road image of Run . Effect of the Invention

[0019] According to the present invention, it is possible to obtain an effect of improving the accuracy of determining road conditions. [Brief description of the drawings]

[0020] [Figure 1]FIG. 1 is a block diagram illustrating an example of a configuration of a state determining device according to the first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of weights. [Diagram 3] FIG. 3 is a flow diagram showing an example of an operation in the learning phase of the determination model. [Figure 4] FIG. 4 is a flow diagram illustrating an example of an operation in the determination phase of the state determining device according to the first embodiment. [Diagram 5] FIG. 5 is a block diagram illustrating an example of a hardware configuration of the state determining device. [Figure 6] FIG. 6 is a block diagram showing an example of the configuration of a state determining system including the state determining device according to the first embodiment. As shown in FIG. [Figure 7] FIG. 7 is a block diagram illustrating an example of a configuration of a state determining device according to the second embodiment. As illustrated in FIG. [Figure 8] FIG. 8 is a flow diagram illustrating an example of an operation in the determination phase of the state determining device according to the second embodiment. [Figure 9] FIG. 9 is a block diagram illustrating an example of a configuration of a state determining device according to the third embodiment. As shown in FIG. [Figure 10] FIG. 10 is a flow diagram illustrating an example of an operation in the learning phase of the weight determining unit according to the third embodiment. [Figure 11] FIG. 11 is a block diagram illustrating an example of a configuration of a state determining device according to the fourth embodiment. As shown in FIG. [Figure 12] FIG. 12 is a flow diagram illustrating an example of an operation in the learning phase of the weight determining unit according to the fourth embodiment. [Figure 13] FIG. 13 is a block diagram showing an example of the configuration of a state determining system including a state determining device according to the fourth embodiment. As shown in FIG. [Figure 14] FIG. 14 is a block diagram illustrating an example of a configuration of a state determining device according to the fifth embodiment. As shown in FIG. [Figure 15] FIG. 15 is a diagram showing an example in which a plurality of drive recorders are used as imaging devices. [Figure 16]FIG. 16 is a diagram showing an example of the display. [Figure 17] FIG. 17 is a diagram showing an example of displaying a plurality of states. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] Next, an embodiment of the present invention will be described with reference to the drawings.

[0022] The drawings are for explaining the embodiments. However, the embodiments are not limited to the description in the drawings. In addition, the same numbers are used for similar configurations in the drawings, and repeated explanations may be omitted.

[0023] In addition, in the drawings used in the following description, in the description of each embodiment, the description of the configuration of parts that are not related to solving the problem of the present invention may be omitted or not shown.

[0024] Before describing each embodiment, the terminology used in the following description will be clarified.

[0025] A "mobile body" is an object that moves and is equipped with an image capturing device that captures images of the road.

[0026] In each embodiment, the moving object is arbitrary. For example, the moving object may be a vehicle (four-wheeled vehicle or two-wheeled vehicle) equipped with an image capturing device, or a drone. Alternatively, the moving object may be a person moving while holding an image capturing device.

[0027] The "state of the moving object" is the state or characteristics of the moving object that are relevant to the captured image.

[0028] For example, different vehicle types (models) may have different mounting positions, angles, and orientations of the image capturing device relative to the traveling direction. Therefore, the model of the vehicle in which the image capturing device is mounted is an example of a characteristic of the moving body.

[0029] Alternatively, a two-wheeled vehicle has a larger change in the inclination of the body than a four-wheeled vehicle. Therefore, the number of wheels is an example of a characteristic of a moving body.

[0030] Alternatively, when a moving body is moving at high speed, objects in the captured image may become unclear (for example, motion blur may occur). Judgment using such an image reduces the accuracy of the judgment. In other words, the speed of the moving body affects the state of the image used for judgment. In this way, the moving speed of the moving body is an example of the state of the moving body.

[0031] Alternatively, the acceleration and vibration of the moving body during shooting affect the captured image, and therefore the acceleration and vibration of the moving body are examples of the state of the moving body.

[0032] The term "road" is not limited to roads on which vehicles and people travel, but also includes structures related to roads. For example, a "road" may include signs, white lines, guardrails, reflectors, traffic lights, and / or lighting fixtures. Furthermore, in the following description, a "road" is not limited to roads on which vehicles and people travel, but may also be a passage on which other objects travel. For example, in each embodiment, a "road" may be a runway, taxiway, or apron on which airplanes travel.

[0033] "Road conditions" refers to the conditions of the road that are the subject of the judgment and the conditions of the road that affect the judgment.

[0034] For example, each embodiment may use types of road deterioration, such as deterioration of the road surface (cracks (vertical, horizontal, or tortoiseshell), ruts, and / or potholes, etc.), deterioration of the road surface seal, and fraying around the edges of the seal, as the road condition to be determined.

[0035] The road condition to be determined is not limited to the deterioration of the road surface. For example, in each embodiment, the road condition to be determined may be the deterioration of structures related to the road (for example, the fading of white lines and road markings on the road surface, and / or the damage of marks).

[0036] Furthermore, the road condition to be determined is not limited to the deterioration of the road and road-related structures. For example, white lines and road markings on the road surface are constructed so as to be reflective at night. Thus, in each embodiment, the road condition to be determined may be the reflective state of the white lines and road markings.

[0037] Alternatively, in each embodiment, the lighting state of lights installed on a road, the brightness of the lights, or the illuminance of the road surface may be determined.

[0038] Alternatively, the classification of the road (such as a community road, a city road, a prefectural road, a national road, or an expressway) and the number of lanes affect the number of vehicles traveling on the road surface, etc. Therefore, in each embodiment, the classification of the road and the number of lanes may be used as the road condition that affects the determination.

[0039] Alternatively, the type of road pavement, the shape of the pavement material, and the condition of the road surface affect the captured image, so that in each embodiment, the type of road pavement, the shape of the pavement material, and the condition of the road surface may be used as the road condition that affects the determination.

[0040] The type of road pavement may be asphalt, concrete, stone, brick, gravel, etc. The type of pavement may include the construction method of the pavement, such as drainage pavement. The pavement material that affects the image may be the grain size and / or color.

[0041] Or, a manhole appears differently in an image on a sunny day than on a rainy day. Or, the degree of wetness of the road surface due to rain changes the state of the captured image. Furthermore, the amount of rain affects the size of puddles that hide the state of the road surface. In this way, the dryness or wetness of the road surface affects the image. Also, surface treatment of the road surface (for example, straight grooves for drainage or circular grooves for preventing slipping on slopes) affects the image. In other words, the dryness or wetness of the road surface and the surface treatment affect the image. Therefore, each embodiment may use the dryness or wetness of the road surface and / or the surface treatment as the road condition that affects the image.

[0042] In the following description, "road deterioration" is used as an example of the "road condition" to be determined.

[0043] The "external environment" refers to information that affects the determination using the image of the road, excluding the above-mentioned states of the moving body and the state of the road.

[0044] For example, the shooting conditions (e.g., the time of day and weather (sunny, cloudy, rainy, after rain, snow, etc.) when an image is shot) affect the determination using the image. Therefore, the external environment may include the shooting conditions (e.g., the time of day and weather when an image is shot).

[0045] Alternatively, ambient noise during shooting increases when the road is congested. The congested roads affect the progression of deterioration.

[0046] Alternatively, when the road surface is deteriorated, a moving body such as a vehicle will experience more vibrations during movement. Therefore, when the road surface is deteriorated, the sound generated by the vibrations during movement of a moving body such as a vehicle (hereinafter referred to as "vibration sound") will become louder.

[0047] Alternatively, the amount of rain affects the captured image, and the sound of rain in rainy weather is proportional to the amount of rain to some extent.

[0048] In this way, sound is one of the pieces of information that is effective in determining the deterioration of a road, and therefore the external environment may include sound (for example, ambient sound, vibration sound, or rain sound).

[0049] Alternatively, the image quality and size are one of the factors that affect the determination. Alternatively, in the case of a moving image, the frame rate affects the image. Therefore, the external environment may include image specifications such as the image quality, size, and frame rate.

[0050] Alternatively, structures installed around the road may cast shadows or the like onto the image of the road surface. Such shadows affect the determination using the image. Therefore, the external environment may include structures installed around the road (for example, advertising billboards and signs installed on the side of the road, trees around the road, and / or mid- to high-rise buildings (buildings and bridges)).

[0051] In this way, the state of the moving object, the state of the road, and the external environment include many things. Therefore, a user or the like can select information to be used as the state of the moving object, the state of the road, and the external environment as appropriate, taking into consideration the object of judgment.

[0052] In the following description, the weather-related terms are as follows. However, these terms may differ from meteorological terms in the strict sense. When a user classifies images to be applied to a model, the user may decide on the weather terms appropriately, taking into consideration the effectiveness and convenience of the operation.

[0053] "Clear weather" refers to weather that includes both "clear skies" where the cloud cover in the sky is 1 or less, and "sunny" where the cloud cover is 2 to 8.

[0054] "Cloudy" means weather with a cloud cover of 9 or more and no precipitation.

[0055] "Rainy weather" means weather with precipitation.

[0056] Cloud cover is the percentage of the sky that is covered by clouds, and is an integer ranging from 0 when there are no clouds at all to 10 when the entire sky is covered by clouds.

[0057] <First embodiment> Next, a first embodiment will be described with reference to the drawings.

[0058] [Configuration Description] First, the configuration of a state determining device 10 according to the first embodiment will be described with reference to the drawings.

[0059] FIG. 1 is a block diagram showing an example of a configuration of a state determining device 10 according to the first embodiment.

[0060] The state determination device 10 includes a plurality of determination models 110, an output summing section 120, and a state determination section 130. Note that the number of determination models 110 included in the state determination device 10 is not particularly limited as long as there is a plurality of determination models 110.

[0061] The determination model 110 includes a model that determines the state of a road using an input road image. For example, the determination model 110 uses an image of a road to output the location of deterioration of the road and its score.

[0062] The determination model 110 is a trained model that has been trained in advance using artificial intelligence (AI) or machine learning. For example, the determination model 110 is a trained model that has been trained on road conditions using training data including images of roads and correct labels of road conditions, using techniques such as AI or machine learning.

[0063] In the following description, the stage in which a model (eg, the determination model 110) is learned using training data is referred to as a "learning phase."

[0064] In addition, the stage in which an image of the road to be judged is input, the road condition (e.g., the location and score of road deterioration) is judged using a trained model (e.g., judgment model 110), and the judgment result is output is called the “judgment phase.”

[0065] In other words, the judgment model 110 is a trained model that has been trained to make judgments in a learning phase before operating in the actual judgment phase.

[0066] At least some of the judgment models 110 may have a different structure from the other judgment models 110. For example, at least some of the judgment models 110 may have a neural network with a different structure from the other judgment models 110. However, all of the judgment models 110 may have the same structure.

[0067] The learning phase of the multiple determination models 110 shown in FIG. 1 will now be described.

[0068] In a learning phase, each of the determination models 110 learns to determine the road condition using training data including images in which at least one of the moving object condition, the road condition, and the external environment is different from one another.

[0069] For example, the teacher data may have different moving body conditions. Alternatively, the teacher data may have different road conditions. Alternatively, the teacher data may have different external environments. Alternatively, the teacher data may have different two of the moving body conditions, the road conditions, and the external environment. Alternatively, the teacher data may have different all of the moving body conditions, the road conditions, and the external environment.

[0070] In addition, the three or more teacher data may have different images. For example, one teacher data may have a different state of a moving object from another teacher data, and a different state of a road from another teacher data.

[0071] The determination model 110 is a trained model trained using different teacher data in this manner.

[0072] Then, in the learning phase, each of the judgment models 110 learns to judge the road condition using training data including images in which at least one of the moving body condition, the road condition, and the external environment differs from one another, as described above.

[0073] For example, in a learning phase, one determination model 110 learns to determine road conditions using training data including images of moving objects in different states, while another determination model 110 learns to determine road conditions using training data including images of different external environments.

[0074] For example, before executing learning, a user of the state determination device 10 prepares a plurality of teacher data including images that are different from each other in at least one of the state of the moving body, the state of the road, and the external environment as teacher data. For example, the user may classify the images prepared as teacher data into groups that are different from each other in at least one of the state of the moving body, the state of the road, and the external environment, label them, and each group may be used as teacher data.

[0075] In addition, as long as at least some of the images included in the training data are different from other images included in the training data, some of the images included in the training data may be the same as other images.

[0076] In addition, the amount of images used in the training data for each group may be the same, or the amount of images in the training data for at least some groups may be different from the amount of images in the training data for other groups.

[0077] For example, the state determination device 10 executes a learning phase for the determination models 110 so that each determination model 110 learns using different teacher data.

[0078] In addition, in the state determination device 10, the timing of learning each determination model 110 is arbitrary. For example, the state determination device 10 may execute learning of the determination models 110 continuously or in parallel. Alternatively, the state determination device 10 may execute learning of at least a part of the determination models 110 discretely.

[0079] Note that a device other than the state determination device 10 may execute at least a part of the learning of the determination model 110. For example, the state determination device 10 may acquire a trained determination model 110 as at least a part of the determination model 110 from another device (not shown).

[0080] In addition, when all trained determination models 110 are acquired from another device, the state determination device 10 does not need to include a learning phase function.

[0081] Furthermore, the state determination device 10 may perform additional learning on at least a portion of the determination model 110.

[0082] In this manner, the state determination device 10 includes a plurality of determination models 110 each trained using training data including images that differ from one another in at least one of the state of the moving body, the state of the road, and the external environment.

[0083] In the following description, except for the description of the learning phase of the determination model 110, it is assumed that the determination model 110 has already been trained.

[0084] In the judgment phase, the judgment model 110 judges the condition of the road using images of the input road, and outputs the judged road condition, specifically the deterioration of the road, and its score to the output summation unit 120.

[0085] The output summing unit 120 acquires outputs from the multiple determination models 110. Then, the output summing unit 120 sums up the outputs of the multiple determination models 110 using a predetermined method.

[0086] For example, the output summation section 120 may store in advance weights for the outputs of each of the multiple determination models 110, and sum up the outputs of the determination models 110 using the stored weights.

[0087] An example of the summation operation using weights will be described.

[0088] The output summation unit 120 stores in advance the weights for each judgment model 110. Then, the judgment model 110 outputs the position of road deterioration and its score using, for example, an AI for object detection. In this case, the output summation unit 120 multiplies the position of deterioration and its score output by the judgment model 110 by the weights for each judgment model 110, and then outputs the sum result as the summation result.

[0089] Note that the output summation unit 120 may store multiple weights, rather than one weight, as the weight for each determination model 110. For example, the output summation unit 120 may store multiple weights (e.g., weights for sunny weather, cloudy weather, and rainy weather) corresponding to the state (e.g., weather) at the time when the image to be determined was captured.

[0090] An example in which the output summation unit 120 uses a plurality of weights will be described.

[0091] The state determination device 10 includes the following three models as the determination model 110. (1) A decision model 110 trained using training data for sunny days (hereinafter, the "sunny day model"); (2) A determination model 110 trained using training data on cloudy weather (hereinafter, the “cloudy weather model”), (3) A determination model 110 trained using training data for rainy weather (hereinafter, the "rainy weather model").

[0092] It is also assumed that the output summation unit 120 determines the weights using the cloud cover and the amount of precipitation.

[0093] FIG. 2 is a diagram illustrating an example of weights.

[0094] The output summation unit 120 stores in advance the weights shown in Fig. 2. Then, the output summation unit 120 sums up the outputs of the determination models 110 as follows.

[0095] The output summing unit 120 acquires the cloud cover and the amount of precipitation of the image to be determined. For example, the output summing unit 120 acquires the cloud cover and the amount of precipitation from the transmission source of the image to be determined.

[0096] The output summing unit 120 may obtain the rainfall and precipitation amount by any method. For example, the output summing unit 120 may obtain the photographing position and photographing date and time of the image, and obtain the cloud cover and precipitation amount at the obtained position and date and time from a company that provides meteorological data. Alternatively, the output summing unit 120 may estimate the cloud cover and precipitation amount using measurement data of sensors (e.g., an illuminance sensor and a humidity sensor) mounted on the moving object. Alternatively, the output summing unit 120 may estimate the cloud cover and precipitation amount by applying a predetermined image analysis method to the image to be determined.

[0097] Then, the output summation unit 120 selects a weight to be used for summation from the weights shown in FIG. 2 as a weight for the determination model 110, based on the cloud cover and precipitation amount at the time the image to be determined was captured.

[0098] 2, a column marked with "-" indicates that the weight is not taken into consideration. For example, when the cloud cover is between 1 and 8, the output summation unit 120 determines the weight without considering the amount of precipitation. The output summation unit 120 also compares the amount of precipitation with a predetermined threshold value to determine the amount of precipitation.

[0099] For example, when the cloud cover is 6, the output summation unit 120 selects a weight of "0.7" for the fine weather model, a weight of "0.3" for the cloudy weather model, and a weight of "0.0" for the rainy weather model.

[0100] Then, the output summation unit 120 sums up the outputs of the fine weather model, the cloudy weather model, and the rainy weather model using the weights.

[0101] In this manner, the output summing unit 120 sums up the results of judgments made by a plurality of judgment models 110 using a predetermined method.

[0102] 2, the weight "0.0" for the rain model when the cloud cover is 6 means that the output of the rain model is not used for the summation in the output summation unit 120. In this way, the output summation unit 120 does not need to sum the outputs of some of the determination models 110.

[0103] Furthermore, if the sum of the weights is not "1", the output summation section 120 may calculate the sum by dividing the added value by the number of determination models 110 (that is, the weighted average value) as the summation result.

[0104] Alternatively, when the accuracy of the judgments made by the judgment models 110 differs, the output summation unit 120 may use weights that take the accuracy of the judgment models 110 into consideration.

[0105] In this manner, the output summing unit 120 sums up the outputs (determination results) of the multiple determination models 110 using a predetermined method.

[0106] However, the method of adding up the outputs is not limited to the above and may be any method. A user of the state determination device 10 may select a method of adding up the outputs based on knowledge or the like.

[0107] For example, the output summation unit 120 may output, as the summation result, a position that is determined as a position of deterioration by more determination models 110 than a threshold value, as the "position of deterioration."

[0108] Alternatively, the output summation unit 120 may use a summation method learned using AI.

[0109] Returning to the explanation with reference to FIG.

[0110] The output summation unit 120 outputs the summation result to the state determination unit 130.

[0111] The state determination unit 130 determines the state of the road in the image to be determined based on the result of the addition in the output addition unit 120.

[0112] Each judgment model 110 judges the state of the road. Therefore, the result of the summation in the output summation unit 120 is the judgment of the state of the road. Therefore, the user of the state judgment device 10 may use the result of the summation in the output summation unit 120.

[0113] However, the road condition required by the user may be determined using the result of the summation in addition to the result of the summation.

[0114] For example, when the determination model 110 outputs the location and type of deterioration, the result of addition in the output addition unit 120 becomes the location and type of deterioration, so that the user can grasp the location and type of deterioration.

[0115] However, for example, when determining whether or not repairs are necessary, the user needs the degree of deterioration in addition to the location and type of deterioration. Therefore, the condition determination unit 130 may determine the degree of deterioration (hereinafter referred to as the "degree of deterioration") based on the output location and type of deterioration as the road condition. In this way, the condition determination unit 130 determines the road condition using the result of the summation in the output summation unit 120.

[0116] Any method may be used to determine the road condition in the condition determining unit 130. For example, the condition determining unit 130 may determine the road condition using a general image processing method.

[0117] The condition determining unit 130 may include information acquired from the output combining unit 120 (for example, the location and type of deterioration) in the road condition that it outputs.

[0118] Then, the state determination unit 130 outputs the state of the road, which is the result of the determination, to a predetermined device. For example, the state determination unit 130 outputs the result of the determination to a device that displays the state of the road.

[0119] The condition determining unit 130 may store the road condition, which is the result of the determination, in a predetermined storage device.

[0120] The condition determining unit 130 may output an image used for the determination in addition to the road condition.

[0121] Furthermore, the condition determining device 10 may determine the road condition using a plurality of images (a plurality of still images or moving images) instead of a single image.

[0122] [Explanation of operation] Next, the operation of the state determining device 10 will be described with reference to the drawings.

[0123] (1) Learning Phase First, the operation of the learning phase of the determination model 110 will be described.

[0124] FIG. 3 is a flow diagram showing an example of the operation of the determination model 110 in the learning phase.

[0125] It is assumed that the state determination device 10 has previously acquired a plurality of teacher data sets that differ in at least one of the state of a moving body equipped with an imaging device that captures images of a road, the state of the road, and the external environment.

[0126] The state determination device 10 executes the following operations for all determination models 110.

[0127] First, the state determination device 10 selects one determination model 110 from the unlearned determination models 110 (step S401).

[0128] The state determining device 10 selects one piece of teacher data from the teacher data that has not been used for learning (step S402).

[0129] Then, the state determination device 10 uses the selected teacher data to execute learning of the selected determination model 110 (step S403). Note that the determination model 110 may use general machine learning for learning.

[0130] The state determination device 10 repeats the above operations until there are no unlearned determination models 110 remaining.

[0131] (2) Judgment phase Next, the operation of the state determining device 10 in the determination phase will be described.

[0132] FIG. 4 is a flow diagram showing an example of the operation of the state determining device 10 according to the first embodiment in the determination phase.

[0133] It should be noted that the judgment model 110 has already been trained in the judgment phase.

[0134] The state determination device 10 acquires an image of the road to be determined from a predetermined device (step S411). For example, the state determination device 10 acquires an image of the road captured by a photographing device (not shown). Alternatively, the state determination device 10 acquires an image of the road stored in a storage device (not shown).

[0135] Then, each of the judgment models 110 judges the condition of the road using the image of the road (step S412). For example, each of the judgment models 110 judges the deterioration of the road (for example, the location and type of the deterioration). Then, the judgment models 110 output the result of the judgment to the output summation unit 120.

[0136] The output summing unit 120 sums up the outputs (results of the judgment) of the judgment models 110 (step S413). Then, the output summing unit 120 outputs the summed result to the state judgment unit .

[0137] The condition determining unit 130 determines the condition (eg, the deterioration level) of the road using the result of the summation in the output summation unit 120 (step S414).

[0138] Then, the condition determining unit 130 outputs the road condition (e.g., the deterioration level) to a predetermined device (step S415). The condition determining unit 130 may include the result of the summation by the output summation unit 120 (e.g., the position and type of deterioration) in the road condition.

[0139] [Effect description] Next, effects of the state determining device 10 according to the first embodiment will be described.

[0140] The condition determining device 10 according to the first embodiment can obtain the effect of improving the accuracy of determining the condition of the road.

[0141] The reasons are as follows:

[0142] The state determination device 10 includes a plurality of determination models 110 and an output summing unit 120. The determination models 110 are each a determination model trained using teacher data that differs in at least one of the state of a moving object equipped with an image capturing device that captures an image of a road, the state of the road, and the external environment. The output summing unit 120 sums up outputs from the plurality of determination models for the input road image.

[0143] In this way, the state determination device 10 adds up the results of determinations of a plurality of determination models 110 that have been trained using teacher data with different states of moving bodies, etc. Therefore, the state determination device 10 can improve the accuracy of determinations for input road images.

[0144] As an example, consider the weather-related case.

[0145] "Clear skies" includes cloud cover of up to 8. On the other hand, "cloudy skies" is cloud cover of 9 or more. However, the actual cloud cover may be between 8 and 9. In such cases, it is expected that a judgment that corresponds more closely to the actual cloud cover will be made by using the results of both the clear skies model and the cloudy skies model, rather than using the results of either one of them.

[0146] In this way, depending on the weather, taking into consideration the outputs of multiple determination models 110 can improve the accuracy of road condition determination.

[0147] Therefore, the output summation unit 120 uses a predetermined method (for example, weighting) to sum up the judgment results of the multiple judgment models 110. Therefore, the summation result of the output summation unit 120 is a more accurate judgment than the judgment of a single model.

[0148] Furthermore, the state determination device 10 includes a state determination unit 130. The state determination unit 130 determines and outputs the state of the road based on the summation result in the output summation unit 120. As a result, the state determination unit 130 determines the state of the road using the result of summing up the determinations of the multiple determination models 110, that is, a more accurate result.

[0149] Therefore, the condition determining device 10 can improve the accuracy of determining the condition of the road compared to a determination using a single determination model.

[0150] In addition, there are cases where the accuracy of each of the determination models 110 is different. In such a case, the accuracy of the determination cannot be ensured by simply using any one of the determination models 110 for the determination.

[0151] For example, a device that uses one of the models (hereinafter referred to as an "associated device") is equipped with a fine weather model, a cloudy weather model, and a rainy weather model, and the associated device selects and makes a determination based on the model according to the weather.

[0152] However, for example, it is assumed that the accuracy of the rainy weather model is lower than the accuracy of the fine weather model and the cloudy weather model.

[0153] In this case, the related device uses a rainy weather model for the image of rainy weather, and therefore the related device cannot ensure the accuracy of the determination in rainy weather.

[0154] Meanwhile, the state determination device 10 includes an output summing unit 120. The output summing unit 120 may sum up the outputs in consideration of the accuracy of the multiple determination models 110. Therefore, even if the accuracy of some of the determination models 110 is low, the state determination device 10 can improve the accuracy of the determination by using a result of summing the results of the determination of the other determination models 110.

[0155] [Hardware configuration] Next, the hardware configuration of the state determining device 10 will be described.

[0156] Each component of the state determining device 10 may be configured as a hardware circuit.

[0157] Alternatively, each component of the state determination device 10 may be configured using a plurality of devices connected via a network. For example, the state determination device 10 may be configured using cloud computing.

[0158] Alternatively, in state determination device 10, a plurality of components may be configured as a single piece of hardware.

[0159] Alternatively, the status determination device 10 may be realized as a computer device including a Central Processing Unit (CPU), a Read Only Memory (ROM), and a Random Access Memory (RAM). The status determination device 10 may be realized as a computer device including a network interface circuit (NIC) in addition to the above configuration.

[0160] FIG. 5 is a block diagram showing an example of a hardware configuration of the state determining device 10. As shown in FIG.

[0161] The state determining device 10 includes a CPU 610, a ROM 620, a RAM 630, a storage device 640, and a NIC 650, and constitutes a computer device.

[0162] The CPU 610 reads a program from the ROM 620 and / or the storage device 640. Then, the CPU 610 controls the RAM 630, the storage device 640, and the NIC 650 based on the read program. Then, the computer including the CPU 610 controls these configurations and realizes the functions of the determination model 110, the output summation unit 120, and the state determination unit 130 shown in FIG.

[0163] When implementing each function, the CPU 610 may use the RAM 630 or the storage device 640 as a temporary storage medium for a program.

[0164] Furthermore, the CPU 610 may read a program contained in a recording medium 690 that stores a program in a computer-readable manner, using a recording medium reading device (not shown). Alternatively, the CPU 610 may receive a program from an external device (not shown) via the NIC 650, store the program in the RAM 630 or the storage device 640, and operate based on the stored program.

[0165] The ROM 620 stores fixed data and programs executed by the CPU 610. The ROM 620 is, for example, a programmable ROM (P-ROM) or a flash ROM.

[0166] The RAM 630 temporarily stores programs and data executed by the CPU 610. The RAM 630 is, for example, a dynamic RAM (D-RAM).

[0167] The storage device 640 stores data and programs that are to be stored long-term by the state determination device 10. The storage device 640 may also operate as a temporary storage device for the CPU 610. The storage device 640 is, for example, a hard disk device, a magneto-optical disk device, a solid state drive (SSD), or a disk array device.

[0168] The ROM 620 and the storage device 640 are non-volatile (non-transitory) recording media. On the other hand, the RAM 630 is a volatile (transitory) recording medium. The CPU 610 can operate based on a program stored in the ROM 620, the storage device 640, or the RAM 630. In other words, the CPU 610 can operate using a non-volatile recording medium or a volatile recording medium.

[0169] The NIC 650 relays data exchange with external devices (not shown) (for example, a source of an image for determination and an output device of road conditions) via a network. The NIC 650 is, for example, a Local Area Network (LAN) card. Furthermore, the NIC 650 is not limited to being wired, and may be wireless.

[0170] The state determination device 10 of FIG. 5 configured in this manner can obtain the same effects as the state determination device 10 of FIG.

[0171] The reason is that the CPU 610 of the state determining device 10 in FIG. 5 can realize each function of the state determining device 10 in FIG. 1 based on a program.

[0172] [System configuration] FIG. 6 is a block diagram showing an example of the configuration of a state determining system 50 including the state determining device 10 according to the first embodiment.

[0173] The condition determination system 50 includes a condition determination device 10 , an image capture device 20 and / or an image storage device 25 , and a display device 30 .

[0174] The imaging device 20 is mounted on a moving object, acquires images of the teacher data and / or the object to be determined, and outputs the images to the status determination device 10. Alternatively, the imaging device 20 may acquire images of the teacher data and / or the object to be determined, and store the images in the image storage device 25. The imaging device 20 is, for example, a camera or a drive recorder.

[0175] The number of image capturing devices 20 may be one or more.

[0176] Fig. 15 is a diagram showing an example of a case where a plurality of drive recorders 820 are used as imaging devices. Fig. 15 shows a vehicle 810 as an example of a moving body.

[0177] The network 850 is a communication path to which the state determining device 10 is connected.

[0178] The wireless communication path 830 connects the drive recorder 820 and the wireless base station 840 .

[0179] The wireless base station 840 relays between the network 850 to which the state determining device 10 is connected and the wireless communication path 830 .

[0180] A vehicle 810 is equipped with a drive recorder 820 and travels on a road.

[0181] Drive recorder 820 is mounted on vehicle 810 and captures images of the road on which vehicle 810 is traveling. Note that, in order to make drive recorder 820 easier to understand, Fig. 15 shows drive recorder 820 adjacent to the exterior of vehicle 810.

[0182] Then, the drive recorder 820 transmits the captured image to the state determination device 10 via a wireless communication path 830, a wireless base station 840, and a network 850. Note that the wireless communication path 830, the wireless base station 840, and the network 850 are examples of communication paths between the drive recorder 820 and the state determination device 10. The drive recorder 820 may be connected to the state determination device 10 by a device or path different from that shown in FIG. 15.

[0183] There are no particular limitations on the vehicle 810, the drive recorder 820, the wireless communication path 830, the wireless base station 840, and the network 850. Generally available products may be used as the vehicle 810, the drive recorder 820, the wireless communication path 830, the wireless base station 840, and the network 850. Therefore, detailed descriptions of these will be omitted.

[0184] The state determination device 10 may be mounted on a moving object. In this case, the state determination device 10 may obtain an image for determination from the image capture device 20 directly or via a communication path within the moving object.

[0185] Returning to the explanation with reference to FIG.

[0186] The image storage device 25 stores the teacher data and / or the image of the judgment target. Then, the image storage device 25 outputs the teacher data and / or the image of the judgment target to the state judgment device 10. The image storage device 25 is, for example, a hard disk device, a magneto-optical disk device, an SSD, or a disk array device.

[0187] The image storage device 25 receives the teacher data and / or the image to be judged from the photographing device 20. However, the image storage device 25 may receive the teacher data and / or the image to be judged from a device (not shown) different from the photographing device 20.

[0188] A user may appropriately determine whether the condition determination device 10 obtains the teacher data and the image of the determination target from the photographing device 20 or the image storage device 25. For example, the condition determination device 10 may obtain the teacher data from the image storage device 25 and obtain the image of the determination target from the photographing device 20.

[0189] The state determination device 10 operates as already described, and executes a learning phase using the teacher data.

[0190] Furthermore, the condition determination device 10 operates as already described, and in the determination phase, determines the road condition using the image of the determination target, and outputs the determined road condition.

[0191] The display device 30 displays the result of the determination (road condition) output by the condition determination device 10. The display device 30 is, for example, a liquid crystal display, an organic electroluminescence display, or electronic paper.

[0192] The display on the display device 30 is optional. The user may display the road conditions on the display device 30 as appropriate and necessary.

[0193] FIG. 16 is a diagram showing an example of the display.

[0194] Fig. 16 shows the location of deterioration using a rectangular frame in the image of the road to be judged. Furthermore, Fig. 16 highlights (hatched) the location that the state judgment device 10 judges to be "highly deteriorated."

[0195] The display device 30 may display not just one road condition but a plurality of road conditions collectively as the display of the road condition determined by the condition determination device 10.

[0196] FIG. 17 is a diagram showing an example of displaying a plurality of states.

[0197] Fig. 17 uses arrows to indicate parts of a road in a given area that are determined to be deteriorated. Furthermore, Fig. 17 highlights (in black) the areas that are "highly deteriorated." Note that the direction of the arrows in Fig. 17 is the traveling direction of vehicles on the road.

[0198] <Second embodiment> Next, a second embodiment will be described with reference to the drawings.

[0199] [Configuration Description] FIG. 7 is a block diagram showing an example of the configuration of a state determining device 11 according to the second embodiment.

[0200] The state determination device 11 includes a plurality of determination models 110, an output summation unit 121, a state determination unit 130, and a weight determination unit 140. The determination models 110 and the state determination unit 130 are similar to the determination models 110 and the state determination unit 130 of the first embodiment. Therefore, detailed description thereof will be omitted.

[0201] The state determination device 11 may be configured using a computer as shown in Fig. 5. Alternatively, the state determination system 50 may include the state determination device 11 instead of the state determination device 10.

[0202] The output summation unit 121 sums up the judgment results of the judgment model 110 using the weights determined by the weight determination unit 140. For example, when the state judged by the judgment model 110 is a position of deterioration and a score, the output summation unit 121 uses the weights determined by the weight determination unit 140 to sum up the position of deterioration output by the judgment model 110 and its score.

[0203] The output summation unit 121 operates in the same manner as the output summation unit 120 of the first embodiment, except that the output summation unit 121 uses the weights determined by the weight determination unit 140.

[0204] The weight determination unit 140 determines the weights to be used when the output summation unit 121 sums the determination models 110, using the image to be determined.

[0205] Therefore, the weight determination unit 140 stores in advance a mechanism for determining weights using an image (for example, an equation or a determinant for calculating weights). In the following description, the mechanism for determining weights is called a "weight determination formula." However, the weight determination formula is not limited to a simple scalar mathematical formula. For example, the weight determination formula may be an equation, a vector formula, a determinant, a function, or a combination of these. The weight determination formula may include a function including a conditional statement.

[0206] The operation of the weight determining section 140 will be explained using a simple example.

[0207] For example, the following equation (1) is assumed to be a weight determination equation representing the summation in the output summation unit 121. Y=a1(p)X1(p)+a2(p)X2(p)+…+a n (p)X n (p) … (1) In the above equation (1), p is the image to be judged (specifically, for example, the pixel value of the image). n is the number of judgment models 110. i is a variable indicated by the judgment model 110 (i=1, 2, ..., n). Y is the result of the summation. i (p) (i=1, 2, ..., n) is a function that determines the weight for each determination model 110 for image p. i (p) (i=1, 2, . . . , n) is the output of the decision model 110.

[0208] The data format of the variables in equation (1) may be appropriately selected from scalars, vectors, matrices, functions, or combinations thereof. i The coefficient of (p) is an example of a parameter of the weight determination formula.

[0209] In this case, the weight determination unit 140 calculates the image p using the function a i (p) to determine the weight. The decision model 110 also uses the image p to determine the decision result "X i Then, the output summation unit 120 calculates the weights (a i (p)) and the judgment result output by the judgment model 110 (X i (p)) is used to calculate the sum, “Y”.

[0210] The weight determination formula is stored in advance in the state determination device 11.

[0211] The configuration for storing the weight determination formula is arbitrary. The state determination device 11 may store the weight determination formula in a storage unit (not shown). In this case, the weight determination unit 140 acquires the weight determination formula stored in the storage unit as necessary, applies the image to the weight determination formula to determine the weight, and provides the determined weight to the output summation unit 121. Alternatively, the weight determination unit 140 may store the weight determination formula. Alternatively, the weight determination unit 140 may acquire the weight determination formula from a device (not shown) that stores the weight determination formula when determining the weight.

[0212] The weight determination unit 140 may use a weight determination formula that has been learned in advance using a predetermined method such as AI. For example, the weight determination unit 140 may determine the parameters (a i A weighting formula that learns the coefficient of (p) may be used.

[0213] [Explanation of operation] Next, the operation of the state determining device 11 according to the second embodiment in the determination phase will be described with reference to the drawings.

[0214] FIG. 8 is a flow diagram showing an example of the operation of the state determining device 11 according to the second embodiment in the determination phase.

[0215] It should be noted that the determination model 110 has already been trained.

[0216] The state determination device 11 acquires an image of the road to be determined from a predetermined device (step S411). For example, the state determination device 11 may acquire an image of the road photographed by the photographing device 20. Alternatively, the state determination device 11 may acquire an image of the road stored in the image storage device 25.

[0217] Then, each of the judgment models 110 judges the condition of the road using the image of the road (step S412). For example, each of the judgment models 110 judges the deterioration of the road (for example, the location and type of the deterioration). Then, the judgment models 110 output the result of the judgment to the output summation unit 120.

[0218] The weight determining unit 140 determines the weight using the image (step S416).

[0219] The output summation unit 121 sums up the outputs (results of the judgment) of the judgment models 110 using the determined weights (step S417). Then, the output summation unit 121 outputs the summation result to the state judgment unit .

[0220] The state determining unit 130 determines the state of the road using the result of the summation in the output summation unit 121 (step S414).

[0221] Then, the condition determination unit 130 outputs the road condition (e.g., the deterioration level) to a predetermined device (step S415). The condition determination unit 130 may include the result of the summation by the output summation unit 121 (e.g., the position and type of deterioration) in the road condition.

[0222] [Effect description] Next, the effects of the state determining device 11 according to the second embodiment will be described.

[0223] The state determining device 11 according to the second embodiment can obtain the same effects as those of the state determining device 10 according to the first embodiment.

[0224] The reasons are as follows:

[0225] This is because the output summation unit 121 operates in the same manner as the output summation unit 120, using the weights determined by the weight determination unit 140.

[0226] Furthermore, the state determining device 11 according to the second embodiment can obtain the effect of further improving the accuracy of determination.

[0227] The reasons are as follows:

[0228] The weight determining unit 140 determines the weights to be used by the output summing unit 121 based on the image to be determined. That is, the weight determining unit 140 determines the weights corresponding to the image to be determined.

[0229] Then, the output summation unit 121 uses weights corresponding to the image to be judged to sum up the outputs of the judgment model 110. That is, the output summation unit 121 can perform more appropriate summation.

[0230] As a result, the condition determining device 11 can further improve the accuracy of determining the road condition.

[0231] <Third embodiment> Next, a third embodiment will be described with reference to the drawings.

[0232] [Configuration Description] FIG. 9 is a block diagram showing an example of the configuration of a state determining device 12 according to the third embodiment.

[0233] The state determination device 12 includes a plurality of determination models 110 , an output summation unit 121 , a state determination unit 130 , a weight determination unit 141 , a loss calculation unit 150 , and a parameter correction unit 160 .

[0234] The state determination device 12 may be configured using a computer as shown in Fig. 5. Alternatively, the state determination system 50 may include the state determination device 12 instead of the state determination devices 10 and 11.

[0235] The determination model 110 and the state determination unit 130 are similar to the determination model 110 and the state determination unit 130 of the first and second embodiments. Moreover, the output summing unit 121 is similar to the output summing unit 121 of the second embodiment.

[0236] Therefore, detailed descriptions of the configurations and operations similar to those of the first and second embodiments will be omitted, and only the configurations and operations unique to the third embodiment will be described.

[0237] The loss calculation unit 150 and the parameter modification unit 160 operate in a weight learning phase in the weight determination unit 141. The weight determination unit 141 operates in the same manner as the weight determination unit 140, except that the weight determination unit 141 cooperates with the loss calculation unit 150 and the parameter modification unit 160 to learn the weights.

[0238] Therefore, the configuration related to the learning phase in weight determining section 141 will be described.

[0239] The loss calculation unit 150 calculates the difference (loss) between data indicating the correct answer for the teacher data for weight learning (for example, the position of degradation in each image and its score, hereinafter referred to as the "correct label") and the result of the summation by the output summation unit 121. In the following description, the data indicating the correct answer is referred to as the "correct label". In the following description, the loss calculated by the loss calculation unit 150 is referred to as the "summed loss" or the "first loss".

[0240] The configuration for storing the correct label is arbitrary. The loss calculation unit 150 may store the correct label corresponding to the teacher data for weight learning in advance. Alternatively, the state determination device 12 may store the correct label in a storage unit (not shown). Alternatively, the loss calculation unit 150 may acquire the correct label from a device or configuration that stores the teacher data for weight learning in calculating the loss.

[0241] The parameter correction unit 160 corrects the parameters of the weight determination formula used by the weight determination unit 140 (for example, the function a in equation (1)) based on the loss calculated by the loss calculation unit 150. i (coefficient of p) is corrected.

[0242] [Explanation of operation] Next, the operation of the state determining device 12 according to the third embodiment will be described with reference to the drawings.

[0243] FIG. 10 is a flow diagram showing an example of the operation of the weight determining unit 141 in the learning phase according to the third embodiment.

[0244] Note that, before the operation of the learning phase, the weight determination unit 141 sets a predetermined weight determination formula (for example, function a i The weighting equation in which the coefficient of (p) is a predetermined value is stored.

[0245] The state determination device 12 acquires an image as teacher data for weight learning (step S421).

[0246] The state determination device 12 selects one image from the teacher data for weight learning. Then, each of the determination models 110 determines the road state using the selected image (step S422). Then, the determination model 110 outputs the result of the determination to the output summation unit 121.

[0247] The weight determination unit 141 applies the selected image to the weight determination formula to determine the weight (step S423).

[0248] The output summation unit 121 sums up the outputs of the determination models 110 using the weights determined by the weight determination unit 141 (step S424).

[0249] The loss calculation unit 150 calculates the loss (combined loss) between the correct label and the result of the combination by the output combination unit 121 (step S425).

[0250] The parameter correction unit 160 corrects the parameters of the weight determination formula (for example, the function a i (coefficient of p) is corrected (step S426).

[0251] The state determination device 12 determines whether the learning is completed (step S427). Specifically, the state determination device 12 determines whether a predetermined end condition (for example, the loss value is smaller than a threshold, a predetermined number of executions, or the end of teacher data for weight learning) is satisfied.

[0252] If the end condition is not satisfied (No in step S427), state determining device 12 returns to step S422.

[0253] If the end condition is satisfied (Yes in step S427), state determining device 12 ends the learning phase.

[0254] In this manner, the state determining device 12 uses teacher data for weight learning to learn the weights (for example, parameters of a weight determining formula) determined by the weight determining unit 141.

[0255] In the determination phase, the state determination device 12 operates in the same manner as the state determination device 11 of the second embodiment. That is, in the determination phase, the state determination device 12 determines the state of the road using the determination model 110, the output summation unit 121, the state determination unit 130, and the weight determination unit 141.

[0256] The operation of state determining device 12 in the determination phase will be described with reference to FIG.

[0257] The state determination device 12 acquires an image of the road to be determined from a predetermined device (step S411).

[0258] The judgment model 110 outputs the state of the road using the image of the road (step S412).

[0259] The weight determination unit 141 determines weights by applying the image of the road to the learned weight determination formula (step S416).

[0260] The output summation unit 121 sums up the outputs of the determination models 110 using the determined weights (step S417).

[0261] The condition determining unit 130 uses the result of the addition to determine the condition of the road (step S414).

[0262] The state determination unit 130 outputs the result of the determination (road state) to a predetermined device (step S415).

[0263] [Effect description] Next, effects of the state determining device 12 according to the third embodiment will be described.

[0264] The state determining device 12 according to the third embodiment can obtain the effect of further improving the accuracy of determination in addition to the effects of the first and second embodiments.

[0265] The reasons are as follows:

[0266] In the learning phase, the weight determination unit 141 learns the weights using teacher data for weight learning, and in the judgment phase, the weight determination unit 141 determines the weights using the learned weight determination formula.

[0267] The output summation unit 121 sums up the outputs of the determination model 110 using the determined weights. That is, the output summation unit 121 sums up the outputs using weights learned using teacher data for weight learning. Therefore, the output summation unit 121 can perform more appropriate summation.

[0268] As a result, the state determining device 12 can improve the accuracy of determining the road state.

[0269] <Fourth embodiment> Next, a fourth embodiment will be described with reference to the drawings.

[0270] [Configuration Description] FIG. 11 is a block diagram showing an example of the configuration of the state determining device 13 according to the fourth embodiment.

[0271] The state determination device 13 includes a plurality of determination models 110 , an output summation unit 121 , a state determination unit 130 , a weight determination unit 142 , a loss calculation unit 150 , a parameter correction unit 161 , and an external environment loss calculation unit 170 .

[0272] The state determination device 13 may be configured using a computer as shown in FIG.

[0273] The determination model 110 and the state determination unit 130 are similar to the determination model 110 and the state determination unit 130 of the first to third embodiments. The output summing unit 121 is similar to the output summing unit 121 of the second and third embodiments. The loss calculation unit 150 is similar to the loss calculation unit 150 of the third embodiment.

[0274] Therefore, detailed descriptions of the configurations and operations similar to those of the first to third embodiments will be omitted, and only the configurations and operations unique to the fourth embodiment will be described.

[0275] The loss calculation unit 150, the parameter correction unit 161, and the external environment loss calculation unit 170 operate in a weight learning phase in the weight determination unit 142. The weight determination unit 142 operates in the same manner as the weight determination unit 141, except for the operation of learning the weights in the learning phase.

[0276] Therefore, the configuration related to the learning phase in weight determining section 142 will be described.

[0277] The weight determination unit 142 determines weights using images of teacher data for weight learning, similarly to the weight determination unit 141. Furthermore, the weight determination unit 142 estimates the external environment using the images, and outputs the estimated external environment to the external environment loss calculation unit 170.

[0278] It is preferable that the weight determining unit 142 share at least a part of the processing between the calculation of the weights and the estimation of the external environment.

[0279] For example, when the weight determination unit 142 uses a deep neural network, the weight determination unit 142 may standardize the lower layers of the deep neural network and specialize the upper layers for calculating weights and estimating the external environment. In this case, the weight determination unit 142 standardizes some of the parameters for determining the weights and the parameters for estimating the external environment. In this case, the learning in the weight determination unit 142 is so-called multitask learning.

[0280] Since the weight determination unit 142 uses a partially shared deep neural network for the calculation of weights and the estimation of the external environment, the weights for the determination model 110 trained using a similar external environment tend to be similar. As a result, the state determination device 13 can obtain the effect of stabilizing the learning in the weight determination unit 142.

[0281] However, the weight determining unit 142 may use different processes for calculating the weights and estimating the external environment.

[0282] The external environment loss calculation unit 170 acquires an external environment related to the image of the training data for weight learning. The source of the external environment is arbitrary. The external environment loss calculation unit 170 may acquire the external environment related to the image of the training data for weight learning from the provider of the training data for weight learning.

[0283] Alternatively, the external environment loss calculation unit 170 may acquire the external environment from a device not shown, with reference to the teacher data for weight learning. For example, the external environment loss calculation unit 170 may acquire meteorological data such as weather from a company that provides meteorological data, with reference to the shooting date and time and the location of the image of the teacher data for weight learning.

[0284] Then, the external environment loss calculation unit 170 calculates the difference (loss) between the acquired external environment and the external environment estimated by the weight determination unit 142. In the following description, the loss calculated by the external environment loss calculation unit 170 is called the "external environment loss" or the "second loss."

[0285] Then, the external environment loss calculation unit 170 outputs the calculated loss (external environment loss) to the parameter correction unit 161.

[0286] The parameter correction unit 161 corrects the parameters of the weight determination formula used by the weight determination unit 142 to calculate the weight, based on the loss (total loss) calculated by the loss calculation unit 150 and the loss (external environment loss) calculated by the external environment loss calculation unit 170.

[0287] [Explanation of operation] Next, the operation of the state determining device 13 according to the fourth embodiment will be described with reference to the drawings.

[0288] FIG. 12 is a flow diagram showing an example of the operation of the weight determining unit 142 in the learning phase according to the fourth embodiment.

[0289] Note that the weight determination unit 142 stores a predetermined weight determination formula as the initial value of the weight before the operation of the learning phase.

[0290] Steps S421 to S425 and S427 are the same operations as those in FIG.

[0291] The state determination device 13 acquires an image as teacher data for weight learning (step S421).

[0292] The state determination device 13 selects one image from the teacher data for weight learning. Then, each of the determination models 110 determines the road state using the selected image (step S422). Then, the determination model 110 outputs the result of the determination to the output summation unit 121.

[0293] The weight determination unit 142 applies the selected image to the weight determination formula to determine the weight (step S423).

[0294] The output summation unit 121 sums up the outputs of the determination models 110 using the weights determined by the weight determination unit 141 (step S424).

[0295] The loss calculation unit 150 calculates the loss (combined loss) between the correct label and the result of the combination by the output combination unit 121 (step S425).

[0296] Furthermore, the weight determining unit 142 estimates the environment using the image (step S431).

[0297] The external environment loss calculation unit 170 calculates the external environment loss (step S432).

[0298] The parameter correction unit 161 corrects the parameters of the weight determination formula used by the weight determination unit 142 to calculate the weight, based on the loss (combined loss) calculated by the loss calculation unit 150 and the loss (external environment loss) calculated by the external environment loss calculation unit 170 (step S433).

[0299] The state determination device 13 determines whether the learning is completed (step S427). Specifically, the state determination device 13 determines whether a predetermined end condition (for example, the value of either or both losses is smaller than a threshold, a predetermined number of executions, or the end of teacher data for weight learning) is satisfied.

[0300] If the end condition is not satisfied (No in step S427), state determination device 13 returns to step S422.

[0301] If the end condition is satisfied (Yes in step S427), the state determining device 13 ends the learning phase.

[0302] The state determination device 13 may switch the order of execution of the operations from steps S422 to S425 and the operations from S431 to S432. Alternatively, the state determination device 13 may execute at least a part of the operations from steps S422 to S425 and the operations from S431 to S432 in parallel.

[0303] In this way, the state determination device 13 learns the weights to be determined by the weight determination unit 142 using teacher data for weight learning and the external environment.

[0304] In the determination phase, the state determination device 13 operates in the same manner as the state determination device 11 of the second embodiment and the state determination device 12 of the third embodiment. That is, in the determination phase, the state determination device 13 determines the state of the road using the determination model 110, the output summation unit 121, the state determination unit 130, and the weight determination unit 142.

[0305] The operation of the state determining device 13 in the determination phase will be described with reference to FIG.

[0306] The state determination device 13 acquires an image of the road to be determined from a predetermined device (step S411).

[0307] The judgment model 110 outputs the state of the road using the image of the road (step S412).

[0308] The weight determination unit 142 determines weights by applying the image of the road to the learned weight determination formula (step S412).

[0309] The output summation unit 121 sums up the outputs of the determination models 110 using the determined weights (step S417).

[0310] The condition determining unit 130 uses the result of the addition to determine the condition of the road (discard step S414).

[0311] The state determination unit 130 outputs the result of the determination (road state) to a predetermined device (step S415).

[0312] [Effect description] Next, the effects of the state determining device 13 according to the fourth embodiment will be described.

[0313] The state determining device 13 according to the fourth embodiment can obtain the effect of further improving the accuracy of determination in addition to the effects of the first to third embodiments.

[0314] The reasons are as follows:

[0315] In the learning phase, the weight determination unit 142 learns parameters of a weight determination formula using the external environment in addition to teacher data for weight learning. Then, in the judgment phase, the weight determination unit 142 determines weights using the learned weight determination formula.

[0316] The output summation unit 121 sums up the outputs of the determination model 110 using the determined weights. That is, the output summation unit 121 sums up the outputs using weights learned using the external environment in addition to the teacher data for weight learning. Therefore, the output summation unit 121 can perform more appropriate summation.

[0317] As a result, the state determining device 13 can further improve the accuracy of determining the state of the road.

[0318] [system] FIG. 13 is a block diagram showing an example of the configuration of a state determination system 51 including a state determination device 13 according to the fourth embodiment.

[0319] The condition determination system 51 includes a condition determination device 13, an image capture device 20 and / or an image storage device 25, a display device 30, and an information providing device 40.

[0320] The photographing device 20, the image storage device 25, and the display device 30 are the same as those in the first embodiment, and therefore detailed descriptions thereof will be omitted.

[0321] The information providing device 40 outputs the external environment to the state determining device 13. Note that the image capturing device 20 may output at least a part of the external environment to the state determining device 13.

[0322] Whether the state determination device 13 acquires information about the external environment from the information providing device 40 or from the image capturing device 20 may be appropriately determined by the user or the like.

[0323] Then, the condition determining device 13 operates as already explained, and outputs the road condition as a result of the determination to the display device 30.

[0324] <Fifth embodiment> The fifth embodiment will be described with reference to the drawings.

[0325] [Configuration Description] FIG. 14 is a diagram showing an example of the configuration of the state determining device 14 according to the fifth embodiment.

[0326] The state determination device 14 includes a plurality of determination models 110 and an output summing unit 120. The determination models 110 are similar to the determination models 110 of the first to fourth embodiments. The output summing unit 120 is similar to the output summing unit 120 of the first embodiment.

[0327] Each component of the state determination device 14 operates in the same manner as the corresponding component in the state determination device 10 of the first embodiment and the like.

[0328] The state determination device 14 may be configured using a computer as shown in Fig. 5. Alternatively, the state determination system 50 may include the state determination device 14 instead of the state determination devices 10 to 12.

[0329] [Effect description] The condition determining device 14 has the effect of improving the accuracy of determining the condition of the road.

[0330] The reasons are as follows:

[0331] The state determination device 14 includes a plurality of determination models 110 and an output summing unit 120. The determination models 110 are each a determination model trained using teacher data that differs in at least one of the state of a moving object equipped with an image capturing device that captures road images, the state of the road, and the external environment. The output summing unit 120 sums up outputs from the plurality of determination models for the input road image.

[0332] In this way, each component of the state determination device 14 operates in the same manner as the corresponding component in the state determination device 10 of the first embodiment, etc. That is, the output summation unit 120 of the state determination device 14 sums up the determination results of the multiple determination models 110 using a predetermined method. Therefore, the state determination device 14 can output a more accurate road state compared to the case where one model is used.

[0333] The user may use the summation result of the output summation unit 120 in the state determination device 14 as the road state. Alternatively, the user may provide the output of the state determination device 14 to a device (not shown) that determines the road state in detail, to determine the road state.

[0334] The state determination device 14 according to the fifth embodiment has the minimum configuration in each embodiment.

[0335] A part or all of the above-described embodiments can be described as, but is not limited to, the following supplementary notes.

[0336] (Appendix 1) A plurality of judgment models each trained using training data differing in at least one of the state of a moving body equipped with an image capturing device for capturing an image of a road, the state of a road, and the external environment; an output summing means for summing outputs from a plurality of decision models for an input road image; A state determination device including:

[0337] (Appendix 2) A state determination means for determining and outputting the state of the road based on the result of the summation by the output summation means. 2. A state determination device according to claim 1, comprising:

[0338] (Appendix 3) A plurality of judgment models are trained using a plurality of training data including images of roads that are different from each other in at least one of the state of the moving object, the state of the road, and the external environment. 3. The state determination device according to claim 1 or 2.

[0339] (Appendix 4) The method further includes a weight determination means for determining weights to be used for combining outputs of the plurality of determination models in response to an input road image, The output summation means sums up the outputs of the plurality of decision models using the determined weights. 4. A state determination device according to claim 1.

[0340] (Appendix 5) The weight determination means learns parameters for determining the weights using training data for weight learning. 5. The state determination device according to claim 4.

[0341] (Appendix 6) The weight determining means further learns parameters for determining the weights in response to an external environment. 6. The state determination device according to claim 5.

[0342] (Appendix 7) A state determination device according to any one of appendix 1 to 5, an image capturing device that captures an image of the road to be determined and outputs it to the state determination device, and / or a storage device that stores the image of the road to be determined; A display device that displays the road condition output by the condition determination device; A condition determination system including:

[0343] (Appendix 8) A state determination device according to claim 6; an image capturing device that captures an image of the road to be determined and outputs the image to the state determination device, and / or a storage device that stores the image of the road to be determined; an information providing device that outputs an external environment to the state determining device; A display device that displays the road condition output by the condition determination device; A condition determination system including:

[0344] (Appendix 9) A state determination device including a plurality of determination models each trained using teacher data differing in at least one of the state of a moving body equipped with an image capture device for capturing an image of a road, the state of a road, and the external environment, Combine the outputs of multiple decision models for an input road image Status determination method.

[0345] (Appendix 10) The road condition is judged based on the combined results. 10. The state determination method according to claim 9.

[0346] (Appendix 11) A plurality of judgment models are trained using a plurality of training data including images of roads that are different from each other in at least one of the state of the moving object, the state of the road, and the external environment. 11. The state determination method according to claim 9 or 10.

[0347] (Appendix 12) determining weights to be used for combining outputs of a plurality of decision models corresponding to the input road image; The outputs of multiple decision models are summed using the determined weights. A state determination method according to any one of appendix 9 to 11.

[0348] (Appendix 13) Using training data for weight learning, the parameters for determining the weights are learned. 13. The state determination method according to claim 12.

[0349] (Appendix 14) In addition, the weights are learned based on the external environment. 14. The state determination method according to claim 13.

[0350] (Appendix 15) A state determination device executes a state determination method according to any one of appendices 9 to 13, The photographing device captures an image of the object to be determined and outputs it to the state determination device, and / or the storage device stores the image of the object to be determined; A display device displays the road condition output by the condition determination device. Status determination method.

[0351] (Appendix 16) A state determination device executes the state determination method described in Supplementary Note 14, The photographing device captures an image of the road to be judged and outputs it to the state judgment device, and / or the storage device stores the image of the road to be judged; The information providing device outputs the external environment to the state determining device, A display device displays the road condition output by the condition determination device. Status determination method.

[0352] (Appendix 17) A computer includes a plurality of judgment models each trained using teacher data that differs in at least one of the state of a moving body equipped with an image capturing device for capturing an image of a road, the state of a road, and the external environment, A process that combines the outputs of multiple decision models for an input road image A recording medium for recording a program for executing the above.

[0353] (Appendix 18) Process to judge road conditions based on the summation result 18. A recording medium according to claim 17, which records a program for causing a computer to execute the above.

[0354] (Appendix 19) A process for training a plurality of determination models using a plurality of training data including images of roads in which at least one of the conditions of a moving object, the condition of a road, and the external environment is different from one another 19. A recording medium according to claim 17 or 18, which records a program for causing a computer to execute the above.

[0355] (Appendix 20) A process of determining weights to be used for combining outputs of a plurality of decision models corresponding to an input road image; A process of summing the outputs of multiple decision models using the determined weights. 20. A recording medium according to any one of appendices 17 to 19, which records a program for causing a computer to execute the above.

[0356] (Appendix 21) A process of learning parameters for determining weights using training data for weight learning 21. A recording medium according to claim 20, for recording a program for causing a computer to execute the above.

[0357] (Appendix 22) In addition, the process of learning parameters that determine the weights in response to the external environment 22. A recording medium according to claim 21, for recording a program for causing a computer to execute the above.

[0358] Although the present invention has been described above with reference to the embodiment, the present invention is not limited to the above embodiment. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. [Explanation of symbols]

[0359] 10. Status determination device 11 Status determination device 12. Status determination device 13. Status determination device 20 Imaging Equipment 25 Image storage device 30 Display device 40 Information provision device 50 Status Judgment System 51 Status Judgment System 110 Decision Model 120 Output summing unit 121 Output summing unit 130 Status determination unit 140 Weight determination unit 141 Weight determination unit 142 Weight determination unit 150 Loss Calculation Section 160 Parameter Modification Section 161 Parameter Modification Section 170 External Environmental Loss Calculation Section 610 CPU 620 ROM 630 RAM 640 Storage device 650NIC 690 Recording media 810 Vehicles 820 Drive Recorder 830 Wireless communication channel 840 Wireless base station 850 Network

Claims

1. A plurality of judgment models each trained using training data differing in at least one of the state of a moving body equipped with an image capturing device for capturing an image of a road, the state of a road, and the external environment; an output summing means for summing outputs from a plurality of said judgment models for an input road image; and a weight determining means for determining weights to be used for combining outputs of a plurality of the judgment models in response to an input road image; The output summing means estimates an external environment from an input road image, selects the weights corresponding to the estimated external environment, and sums up outputs of the plurality of judgment models using the selected weights. Status determination device.

2. A state determining means for determining and outputting a road state based on the result of the summation by the output summation means. The state determination device according to claim 1 .

3. A plurality of the judgment models are trained using a plurality of training data including images of roads that are different from each other in at least one of the state of the moving object, the state of the road, and the external environment. The state determination device according to claim 1 or 2.

4. The weight determination means learns parameters for determining the weights by using teacher data for learning the weights. The state determination device according to claim 1 .

5. The weight determining means further learns the parameters for determining the weights in response to an external environment. The state determination device according to claim 4.

6. The weight determining means performs at least a part of the same process for determining the weight and estimating the external environment. The state determination device according to claim 5 .

7. The state determination device according to any one of claims 1 to 4, an image capturing device that captures an image of a road to be determined and outputs the image to the state determining device, and / or a storage device that stores the image of the road to be determined; a display device that displays the road condition output by the condition determination device; A condition determination system including:

8. The state determination device according to claim 5 or 6, an image capturing device that captures an image of the road to be determined and outputs the image to the state determination device, and / or a storage device that stores the image of the road to be determined; an information providing device that outputs an external environment to the state determining device; a display device that displays the road condition output by the condition determination device; A condition determination system including:

9. A state determination device including a plurality of determination models each trained using teacher data differing in at least one of the state of a moving body equipped with an image capture device for capturing an image of a road, the state of a road, and the external environment, summing outputs from a plurality of said decision models for an input road image; and determining weights to be used for summing outputs of a plurality of said judgment models in response to the input road image; The output summation involves estimating an external environment from an input road image, selecting the weights corresponding to the estimated external environment, and summing the outputs of the multiple decision models using the selected weights. Status determination method.

10. A computer includes a plurality of judgment models each trained using teacher data that differs in at least one of the state of a moving body equipped with an image capturing device for capturing an image of a road, the state of a road, and the external environment, executing a process of summing outputs from a plurality of said determination models for an input road image; and Executing a process for determining weights to be used for summing outputs of a plurality of said judgment models in response to an input road image; The process of summing the outputs includes estimating an external environment from an input road image, selecting the weights corresponding to the estimated external environment, and summing the outputs of the plurality of judgment models using the selected weights. program.

Images