Information processing device, information processing method, and program

Abstract

To provide a technique that enables reduction in processing time required for image data-based evaluation and improvement in evaluation quality.SOLUTION: An information processing device is provided, comprising an acquisition unit for acquiring first image data obtained by an image capturing unit, an identification unit for identifying an evaluation target area based on the first image data, and an evaluation unit for making a predetermined evaluation based on the evaluation target area.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In recent years, techniques for performing evaluation based on image data have become known. For example, Patent Document 1 discloses a technique for performing evaluation based on image data by detecting, based on the image data, points where breaks in lane markings on a road appearing in the image data and determining the cause of the breaks in the lane markings. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2008-241446 Summary of the Invention [Problem to be solved by the invention]

[0004] However, it is desirable to provide a technique that can reduce the processing time required for evaluation based on image data and improve the evaluation accuracy. [Means for solving the problem]

[0005] In order to solve the above problem, according to one aspect of the present invention, an information processing device is provided, comprising: an acquisition unit that acquires first image data obtained by an imaging unit; an identification unit that identifies an area to be evaluated based on the first image data; and an evaluation unit that performs a predetermined evaluation based on the area to be evaluated.

[0006] The identification unit may acquire second image data corresponding to the characteristics of the subject appearing in the first image data, which is output from a first inference model to which the first image data is input, and identify the evaluation target area based on the second image data.

[0007] The feature of the subject may be the type of the subject or the distance from the imaging unit to the subject.

[0008] The feature of the subject may be the type of the subject, and the first inference model may recognize the type of the subject for each pixel and output the type of the subject as the second image data.

[0009] The identification unit may identify pixels where the type of the subject is a predetermined type, and identify the evaluation target area based on the identified pixels.

[0010] The feature of the subject may be the distance from the imaging unit to the subject, and the first inference model may recognize the distance from the imaging unit to the subject as the depth of the subject for each pixel, and output the depth of the subject as the second image data.

[0011] The identification unit may identify pixels of the subject whose depth falls within a predetermined range, and identify the evaluation target area based on the identified pixels.

[0012] The evaluation unit may acquire third image data corresponding to the type of subject appearing in the evaluation target area, which is output from a second inference model to which the evaluation target area is input, and perform the specified evaluation based on the third image data.

[0013] The second inference model may recognize the type of subject for each pixel as a non-degraded lane line class, which is not severely deteriorated, or a degraded lane line class, which is severely deteriorated, on a road.The evaluation unit may perform the specified evaluation by grouping consecutive pixels of the non-degraded lane line class and pixels of the degraded lane line class as a judgment area and estimating the degree of deterioration of the judgment area based on the number of pixels of the non-degraded lane line class and the number of pixels of the degraded lane line class.

[0014] The information processing device may include a determination unit that determines whether the determination area is deteriorated based on whether the degree of deterioration is higher than a threshold value.

[0015] The determination unit may lower the threshold value based on the fact that the deterioration levels of other determination areas show a predetermined tendency.

[0016] The determination unit may determine whether the degree of deterioration of the other judgment area shows the predetermined trend based on whether the average value of the degree of deterioration of the other judgment area or the degree of deterioration of the other judgment area is higher than a predetermined reference value.

[0017] The first inference model may be a trained model generated by machine learning.

[0018] In addition, according to another aspect of the present invention, in order to solve the above problem, an information processing method executed by a computer is provided, which includes acquiring first image data obtained by an imaging unit, identifying an area to be evaluated based on the first image data, and performing a predetermined evaluation based on the area to be evaluated.

[0019] In addition, according to another aspect of the present invention, in order to solve the above problem, a program is provided that causes a computer to function as an acquisition unit that acquires first image data obtained by an imaging unit, an identification unit that identifies an area to be evaluated based on the first image data, and an evaluation unit that performs a predetermined evaluation based on the area to be evaluated. [Effects of the Invention]

[0020] As described above, the present invention provides a technique that can reduce the processing time required for evaluation based on image data and improve the evaluation accuracy. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a diagram illustrating an example of the configuration of a system according to a first embodiment of the present invention. [Figure 2] 1 is a diagram illustrating an example of the configuration of an information processing device 100 according to a first embodiment of the present invention. [Figure 3] 10 is a diagram showing an example of a captured image acquired by an input unit 110. FIG. [Figure 4] FIG. 2 is a diagram illustrating an example of the functional configuration of an identification unit 120. [Figure 5] FIG. 10 is a diagram showing an example of a first segmentation image. [Figure 6] FIG. 10 is a diagram illustrating an example of an evaluation target area. [Figure 7] FIG. 2 is a diagram illustrating an example of the functional configuration of an evaluation unit 130. [Figure 8] FIG. 10 is a diagram showing an example of a second segmentation image. [Figure 9] FIG. 10 is a diagram illustrating an example of a white line binary mask image. [Figure 10] FIG. 10 is a diagram illustrating an example of a degraded binary mask image. [Figure 11] 10 is a diagram showing an example of a deterioration degree estimated by a deterioration degree estimation unit 132. FIG. [Figure 12] 4 is a flowchart showing an example of operation of the information processing device 100 according to the first embodiment of the present invention. [Figure 13] FIG. 10 is a diagram illustrating an example of the configuration of an information processing device 200 according to a second embodiment of the present invention. [Figure 14] 10 is a flowchart showing an example of operation of the information processing device 200 according to the second embodiment of the present invention. [Figure 15] FIG. 10 is a diagram illustrating an example of the functional configuration of an identification unit 220 according to a modified example. [Figure 16] 1 is a diagram showing a hardware configuration of an information processing device 900 as an example of the information processing device 100 according to the first embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0022] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant explanations will be omitted.

[0023] In addition, in this specification and drawings, multiple components having substantially the same functional configuration may be distinguished by adding different numbers after the same reference numeral. However, when there is no particular need to distinguish between multiple components having substantially the same functional configuration, only the same reference numeral will be used.

[0024] (0. Overview) First, an outline of an embodiment of the present invention will be described. In recent years, techniques for performing evaluation based on image data have become known. For example, Patent Document 1 discloses a technique for performing evaluation based on image data by detecting, based on image data, points where breaks occur in road marking lines (hereinafter also referred to as "white lines") on the image data and determining the cause of the breaks in the marking lines.

[0025] However, it is desirable to provide a technique that can reduce the processing time required for evaluation based on image data and improve the evaluation accuracy.

[0026] Specifically, it is not always desirable to use the entire image data as the evaluation target area, and there may be areas in the image data that are better excluded from the evaluation target area (hereinafter also referred to as "unnecessary areas"). By performing an evaluation based on the evaluation target area from which unnecessary areas have been excluded, it is possible to reduce the processing time required for the evaluation and improve the evaluation accuracy.

[0027] For example, when an evaluation is performed to estimate the deterioration level of white lines on roads shown in image data, excluding areas other than roads from the evaluation target area can reduce the processing time required to estimate the deterioration level of the white lines. Furthermore, excluding areas other than roads from the evaluation target area can reduce the possibility of erroneous detection of white lines. By reducing the possibility of erroneous detection of white lines, the accuracy of estimating the deterioration level of white lines can be improved.

[0028] This specification mainly proposes a technology in which unnecessary areas are automatically determined by a system without manual intervention by a user, and a predetermined evaluation is performed based on an evaluation target area from which the unnecessary areas have been excluded. Note that, in the following, an evaluation based on an evaluation target area is mainly described using an example of estimating the deterioration level of white lines based on the evaluation target area, but the evaluation based on the evaluation target area does not have to be limited to estimating the deterioration level of white lines.

[0029] The outline of the embodiment of the present invention has been described above.

[0030] (1. First embodiment) Next, a first embodiment of the present invention will be described with reference to FIGS.

[0031] (1-1. System Configuration) An example of the configuration of a system according to a first embodiment of the present invention will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of a system according to the first embodiment of the present invention. As shown in Fig. 1, the system 1 includes an imaging unit 10, a communication antenna 20, and an information processing device 100. The imaging unit 10 and the communication antenna 20 are configured to be able to communicate wirelessly. Furthermore, the communication antenna 20 and the information processing device 100 are configured to be able to communicate with each other via wire or wirelessly.

[0032] (imaging unit 10) The imaging unit 10 includes an image sensor, captures an image of the road using the image sensor, and obtains image data showing the road as first image data (hereinafter also referred to as "captured image"). FIG. 1 shows a case where the imaging unit 10 is an on-board device provided in a vehicle C1 traveling on a road. However, the imaging unit 10 may be provided in a moving object other than the vehicle C1. Alternatively, the imaging unit 10 may be carried by a moving person. Alternatively, when the area to be evaluated is small, the imaging unit 10 may be fixed to a predetermined location.

[0033] The imaging unit 10 transmits the captured images obtained by capturing images of a road to the communication antenna 20. For example, the imaging unit 10 may continuously capture images of a road in time series and continuously transmit the captured images to the communication antenna 20.

[0034] In the first embodiment of the present invention, it is mainly assumed that the imaging direction of the imaging unit 10 is in front of the vehicle C1. However, the imaging direction of the imaging unit 10 does not have to be in front of the vehicle C1. For example, the imaging direction of the imaging unit 10 may be behind the vehicle C1. Furthermore, as long as the condition that the road is captured within the imaging range of the imaging unit 10 is satisfied, the angle formed between the imaging direction of the imaging unit 10 and the road may be set arbitrarily.

[0035] Furthermore, in the first embodiment of the present invention, it is mainly assumed that one imaging unit 10 is provided in the vehicle C1. However, multiple imaging units 10 may be provided in the vehicle C1. By providing multiple imaging units 10 in the vehicle C1, the evaluation accuracy based on the captured images obtained by the imaging units 10 can be improved.

[0036] (Communication Antenna 20) The communication antenna 20 receives captured images from the imaging unit 10 and transmits the received captured images to the information processing device 100. In the example shown in Fig. 1, the communication antenna 20 is installed on the side of the road. However, the location where the communication antenna 20 is installed does not have to be limited to the side of the road. Furthermore, if the imaging unit 10 can directly transmit captured images to the information processing device 100, the communication antenna 20 does not have to be installed.

[0037] (Information processing device 100) The information processing device 100 is realized by a computer. The information processing device 100 receives captured images from the imaging unit 10 via the communication antenna 20 and performs evaluation based on the received captured images. More specifically, the information processing device 100 can perform evaluation based on the captured images by estimating the degree of deterioration of the white lines shown in the captured images based on the received captured images.

[0038] In the example shown in Fig. 1, the road on which vehicle C1 is traveling has three lanes. Fig. 1 also shows white lines W21-W23 and W31-W33 that separate lanes, and white lines W11 and W41 that separate the inside and outside of the lane. However, the number of lanes on the road on which vehicle C1 is traveling does not have to be limited as long as it is one or more. Furthermore, the number of white lines on the road does not have to be limited.

[0039] 1, various objects are omitted from the illustration to improve visibility of the white lines W11, W21 to W23, W31 to W33, and W41, the vehicle C1, etc. For example, a fence may exist separating the inside and outside of the road, but the fence is omitted from FIG.

[0040] An example of the configuration of the system according to the first embodiment of the present invention has been described above.

[0041] (1-2. Configuration of Information Processing Device 100) Next, an example of the configuration of the information processing device 100 according to the first embodiment of the present invention will be described.

[0042] Fig. 2 is a diagram showing an example of the configuration of an information processing device 100 according to the first embodiment of the present invention. As shown in Fig. 2, the information processing device 100 according to the first embodiment of the present invention includes an input unit 110, an identification unit 120, an evaluation unit 130, a determination unit 140, and an output unit 150. The input unit 110, the identification unit 120, the evaluation unit 130, the determination unit 140, and the output unit 150 can be realized by a control unit (not shown).

[0043] The control unit (not shown) includes a calculation device such as a CPU (Central Processing Unit), and its functions can be realized by the calculation device expanding a program stored in a ROM (Read Only Memory) into a RAM (Random Access Memory) and executing it. In this case, a computer-readable recording medium on which the program is recorded can also be provided.

[0044] Alternatively, the control unit (not shown) may be configured by dedicated hardware or by a combination of multiple pieces of hardware. Data necessary for the calculation by the calculation device is stored as appropriate in a storage unit (not shown).

[0045] The storage unit (not shown) is a memory capable of storing programs and data for operating the control unit (not shown). The storage unit (not shown) can also temporarily store various data required in the process of operation of the control unit (not shown). For example, the storage device may be a non-volatile memory.

[0046] (input unit 110) The input unit 110 functions as an acquisition unit and acquires the captured image transmitted by the imaging unit 10 via the communication antenna 20. More specifically, the input unit 110 acquires the captured image transmitted from the imaging unit 10, relayed by the communication antenna 20, and received by a communication unit (not shown). The captured image acquired by the input unit 110 is output to the identification unit 120. Here, an example of a captured image acquired by the input unit 110 will be described with reference to FIG. 3.

[0047] Fig. 3 is a diagram showing an example of a captured image acquired by the input unit 110. Referring to Fig. 3, captured image G10 is shown as an example of a captured image acquired by the input unit 110. A road is captured in captured image G10, and white lines W11, W21, W31, W41, and W42 that exist on the road are also captured in captured image G10. The road is captured in the lower area of ​​captured image G10.

[0048] (Specific section 120) The specifying unit 120 specifies the evaluation target area based on the captured image G10 acquired by the input unit 110. Here, examples of functions of the specifying unit 120 will be described with reference to FIGS.

[0049] Fig. 4 is a diagram illustrating an example of the functional configuration of the identification unit 120. As illustrated in Fig. 4, the identification unit 120 includes a first segmentation AI (Artificial Intelligence) model 121 and an area identification unit 122. The identification unit 120 inputs the captured image G10 acquired by the input unit 110 to the first segmentation AI model 121, and obtains second image data (hereinafter also referred to as "first segmentation image") output from the first segmentation AI model 121.

[0050] The first segmentation AI model 121 is an inference model (first inference model) that, based on input of a captured image G10, outputs a first segmentation image according to the characteristics of the subject appearing in the captured image G10. Here, it is assumed that the characteristics of the subject are the type of the subject. That is, the first segmentation AI model 121 recognizes the type of the subject for each pixel and outputs the type of the subject for each pixel as a first segmentation image.

[0051] Here, we mainly assume that the types of objects recognized by the first segmentation AI model 121 are divided into two classes: a road class and a non-road class. That is, we assume that the first segmentation AI model 121 is a two-class classification model. However, the types of objects recognized by the first segmentation AI model 121 do not have to be limited to these two classes.

[0052] For example, the types of subjects recognized by the first segmentation AI model 121 may be three classes: a white line class that is not deteriorated (hereinafter also referred to as the "non-deteriorated white line class"), a white line class that is deteriorated (hereinafter also referred to as the "deteriorated white line class"), and a class other than white lines. In other words, the first segmentation AI model 121 may be a three-class classification model. Note that a white line that has been scratched or rubbed to a certain extent or more can be said to be a deteriorated white line.

[0053] Alternatively, the types of subjects recognized by the first segmentation AI model 121 may be four classes: a non-degraded white line class, a degraded white line class, a road other than a white line class, and a non-road class. In other words, the first segmentation AI model 121 may be a four-class classification model.

[0054] It is also conceivable that both the first segmentation AI model 121 and the second segmentation AI model 131 are three-class classification models or four-class classification models. In such a case, the first segmentation image output from the first segmentation AI model 121 may be used as the second segmentation image by the evaluation unit 130, and the second segmentation AI model 131 (FIG. 7), which will be described later, may be omitted.

[0055] The first segmentation AI model 121 may be a trained model generated in advance by machine learning. For example, the trained model may be generated by machine learning based on training data, which is image data showing a subject, and correct answer data, in which the type of subject shown in the training data is set as a correct answer label for each pixel. An example of a first segmentation image output from the first segmentation AI model 121 will be described with reference to FIG. 5.

[0056] 5 is a diagram showing an example of a first segmentation image, in which a captured image G10 is input to the first segmentation AI model 121, and a first segmentation image G20 is output from the first segmentation AI model 121 based on the input.

[0057] The first segmentation image G20 is classified into a non-road class area R21 and a road class area R22. The road class area R22 extends from the bottom edge of the first segmentation image G20 to a road disappearance position P20 in the first segmentation image G20, and disappears at the road disappearance position P20.

[0058] That is, in an xy coordinate system in which the right direction of the first segmentation image G20 is the positive direction of the x axis and the height direction (upward) of the first segmentation image G20 is the positive direction of the y axis, the road disappearance position P20 corresponds to the height coordinate ymax, which is the maximum value among the height coordinates y of the pixels included in the road class area R22.

[0059] The area identification unit 122 identifies the evaluation target area based on the first segmentation image G20. More specifically, the area identification unit 122 identifies pixels whose subject type is a predetermined type based on the first segmentation image G20, and identifies the evaluation target area based on the identified pixels. In this case, the area identification unit 122 may recognize the maximum height coordinate ymax among the height coordinates y of the pixels included in the road class area R22 based on the first segmentation image G20 as the road disappearance position P20, and identify the evaluation target area based on the road disappearance position P20. An example of the evaluation target area will be described with reference to FIG. 6.

[0060] Fig. 6 is a diagram showing an example of an evaluation target area. Referring to Fig. 6, an evaluation target area G30 identified by the area identification unit 122 is shown. Specifically, Fig. 6 shows an example in which the area identification unit 122 sets the road disappearance position P20 (Fig. 5) as a reference position and identifies a rectangular area below this reference position as the evaluation target area G30. However, the method of identifying the evaluation target area G30 does not have to be limited to this example.

[0061] For example, it may be assumed that the evaluation accuracy of an area far from the imaging unit 10 will not improve as much as the evaluation accuracy of an area close to the imaging unit 10. Therefore, areas far from the imaging unit 10 may be excluded from the areas to be evaluated.

[0062] That is, the area identification unit 122 may set a position below the road disappearance position P20 as a reference position, and identify a rectangular area below that reference position as the evaluation target area. For example, if the y coordinate of the bottom edge of the first segmentation image G20 is 0, the area identification unit 122 may calculate the reference position by multiplying the height coordinate ymax, which is the road disappearance position P20, by a predetermined coefficient (for example, 0.5). The area identification unit 122 outputs the identified evaluation target area G30 to the evaluation unit 130.

[0063] It is also possible that a three-class classification model is used as the first segmentation AI model 121. In such a case, the area identification unit 122 may recognize the maximum height coordinate among the height coordinates of the pixels included in the non-degraded white line class and the degraded white line class as the road disappearance position.

[0064] Alternatively, a four-class classification model may be used as the first segmentation AI model 121. In such a case, the area identification unit 122 may recognize the maximum height coordinate among the height coordinates of pixels included in the road class other than the white line as the road disappearance position.

[0065] (Evaluation unit 130) The evaluation unit 130 performs a predetermined evaluation based on the evaluation target area G30 identified by the identification unit 120. More specifically, the evaluation unit 130 performs a predetermined evaluation based on the evaluation target area G30 by estimating the degree of deterioration of white lines present on the road based on the evaluation target area G30 identified by the identification unit 120. Here, examples of functions possessed by the evaluation unit 130 will be described with reference to FIGS. 7 to 11.

[0066] Fig. 7 is a diagram illustrating an example of the functional configuration of the evaluation unit 130. As illustrated in Fig. 7, the evaluation unit 130 includes a second segmentation AI model 131 and a deterioration level estimation unit 132. The evaluation unit 130 inputs the evaluation target area G30 identified by the identification unit 120 to the second segmentation AI model 131, and obtains third image data (hereinafter also referred to as a "second segmentation image") output from the second segmentation AI model 131.

[0067] The second segmentation AI model 131 is an inference model (second inference model) that outputs a second segmentation image (third image data) according to the characteristics of the subject reflected in the evaluation target area G30 based on the input of the evaluation target area G30. Here, it is assumed that the characteristics of the subject are the type of the subject. That is, the second segmentation AI model 131 recognizes the type of the subject for each pixel and outputs the type of the subject for each pixel as a second segmentation image.

[0068] Here, we mainly assume that the types of subjects recognized by the first segmentation AI model 121 are three classes: a non-deteriorated white line class (first lane marking), a deteriorated white line class (second lane marking), and a non-white line class. In other words, we assume that the second segmentation AI model 131 is a three-class classification model. However, the types of subjects recognized by the second segmentation AI model 131 do not have to be limited to these three types.

[0069] For example, the types of subjects recognized by the second segmentation AI model 131 may be four classes: a non-degraded white line class, a degraded white line class, a road other than a white line class, and a non-road class. In other words, the second segmentation AI model 131 may be a four-class classification model.

[0070] As described above, it is also possible that both the first segmentation AI model 121 (FIG. 4) and the second segmentation AI model 131 are three-class classification models or four-class classification models. In such a case, the first segmentation image output from the first segmentation AI model 121 may be used as the second segmentation image by the evaluation unit 130, and the second segmentation AI model 131 may be omitted.

[0071] The second segmentation AI model 131 may be a trained model generated in advance by machine learning. For example, the trained model may be generated by machine learning based on training data, which is image data showing a subject, and correct answer data, in which the type of subject shown in the training data is set as a correct answer label for each pixel. An example of a second segmentation image output from the second segmentation AI model 131 will be described with reference to FIG. 8.

[0072] 8 is a diagram showing an example of a second segmentation image. Referring to FIG. 8, a second segmentation image G40 is shown that is output from the second segmentation AI model 131 based on the evaluation target area G30 being input to the second segmentation AI model 131.

[0073] The second segmentation image G40 is classified into non-degraded white line class areas A11, A21, A31, A41, and A42, deteriorated white line class areas B11, B21, B31, and B41, and a non-white line class C.

[0074] The deterioration level estimation unit 132 generates a white line binary mask image, which is an image in which different values ​​are set for the union of the non-deteriorated white line class areas A11, A21, A31, A41, and A42 and the deteriorated white line class areas B11, B21, B31, and B41, and the non-white line class C. An example of the white line binary mask image will be described with reference to FIG.

[0075] Fig. 9 is a diagram showing an example of a white line binary mask image. Fig. 9 shows a white line binary mask image G50. The white line binary mask image G50 is an image in which different values ​​are set for the union of the non-degraded white line class areas A11, A21, A31, A41, and A42 and the degraded white line class areas B11, B21, B31, and B41, and for the non-white line class C.

[0076] In the second segmentation image G40, the deterioration level estimation unit 132 groups a continuous non-deteriorated white line class area A11 and a deteriorated white line class area B11 into a determination area N11. Similarly, in the second segmentation image G40, the deterioration level estimation unit 132 groups a continuous non-deteriorated white line class area A21 and a deteriorated white line class area B21 into a determination area N21.

[0077] Furthermore, in the second segmentation image G40, the deterioration level estimation unit 132 groups a continuous non-deteriorated white line class area A31 and a deteriorated white line class area B31 into a determination area N31. Furthermore, in the second segmentation image G40, the deterioration level estimation unit 132 groups a continuous non-deteriorated white line class area A41 and a deteriorated white line class area B41 into a determination area N41.

[0078] Furthermore, the deterioration level estimation unit 132 determines that there is no deteriorated white line class area adjacent to the non-deteriorated white line class area A42, and sets the non-deteriorated white line class area A42 as the determination area N42.

[0079] Furthermore, the deterioration level estimation unit 132 generates a degraded binary mask image, which is an image in which different values ​​are set for degraded white line class areas B11, B21, B31, and B41, non-degraded white line class areas A11, A21, A31, A41, and A42, and non-white line class C. An example of the degraded binary mask image will be described with reference to FIG.

[0080] Fig. 10 is a diagram showing an example of a degraded binary mask image. Fig. 10 shows a degraded binary mask image G60. The degraded binary mask image G60 is an image in which different values ​​are set for the degraded white line class areas B11, B21, B31, and B41, the union of the non-degraded white line class areas A11, A21, A31, A41, and A42, and the non-white line class C.

[0081] The deterioration level estimation unit 132 calculates the area of ​​the judgment area N11 by counting the number of pixels included in the judgment area N11, and calculates the area of ​​the deteriorated white line class area B11 by counting the number of pixels included in the deteriorated white line class area B11. Then, the deterioration level estimation unit 132 estimates the deterioration level of the judgment area N11 based on the area of ​​the judgment area N11 and the area of ​​the deteriorated white line class area B11.

[0082] More specifically, the deterioration level estimation unit 132 can estimate the ratio of the area of ​​the deteriorated white line class area B11 to the area of ​​the determination area N11 as the deterioration level of the determination area N11.

[0083] Similarly, the deterioration level estimation unit 132 may estimate the ratio of the area of ​​the deteriorated white line class area B21 to the area of ​​the determination area N21 as the deterioration level of the determination area N21. Also, the deterioration level estimation unit 132 may estimate the ratio of the area of ​​the deteriorated white line class area B31 to the area of ​​the determination area N31 as the deterioration level of the determination area N31.

[0084] Furthermore, the deterioration level estimation unit 132 may estimate the ratio of the area of ​​the deteriorated white line class area B41 to the area of ​​the determination area N41 as the deterioration level of the determination area N41. Also, since the determination area N42 does not include any deteriorated white line class areas, the deterioration level estimation unit 132 may estimate the deterioration level of the determination area N41 to be 0 (zero). The deterioration level estimation unit 132 may perform a predetermined evaluation by estimating the deterioration level in this way.

[0085] Fig. 11 is a diagram showing an example of the degree of deterioration estimated by the deterioration degree estimation unit 132. Referring to Fig. 11, an example is shown in which the degree of deterioration of the determination area N21 (Fig. 8) is estimated to be "0.280".

[0086] 11, the degradation level of the judgment area N31 is estimated as "0.396", the degradation level of the judgment area N41 is estimated as "0.011", the degradation level of the judgment area N11 is estimated as "0.054", and the degradation level of the judgment area N42 is estimated as "0.000". As shown in FIG. 11, the degradation level estimation unit 132 may assign an ID (IDentification) to each degradation level.

[0087] (Judgment unit 140) The determination unit 140 determines whether or not the determination area N11 has deteriorated based on the degree of deterioration of the determination area N11 estimated by the evaluation unit 130. More specifically, the determination unit 140 can determine whether or not the determination area N11 has deteriorated based on whether or not the determination area N11 estimated by the evaluation unit 130 is higher than a threshold value.

[0088] Similarly, the determination unit 140 determines whether or not the determination area N21 has deteriorated based on the deterioration level of the determination area N21 estimated by the evaluation unit 130. The determination unit 140 also determines whether or not the determination area N31 has deteriorated based on the deterioration level of the determination area N31 estimated by the evaluation unit 130.

[0089] The determination unit 140 determines whether the determination area N41 has deteriorated based on the deterioration level of the determination area N41 estimated by the evaluation unit 130. The determination unit 140 also determines whether the determination area N42 has deteriorated based on the deterioration level of the determination area N42 estimated by the evaluation unit 130. In the following description, the threshold value will also be referred to as "Th." Furthermore, in the following description, when the current time is T, the deterioration level at the current time T will also be referred to as "D(T)."

[0090] (output unit 150) The output unit 150 outputs the determination result obtained by the determination unit 140. The determination result indicates whether or not each of the determination areas N11, N21, N31, N41, and N42 is deteriorated. Furthermore, the output unit 150 may output the degree of deterioration of each of the determination areas N11, N21, N31, N41, and N42 in addition to or instead of the determination result. The output unit 150 may output the determination result and the degree of deterioration in association with an ID.

[0091] Note that the destination to which the determination result and the degradation level by the output unit 150 are output does not need to be limited. For example, the output unit 150 may output the determination result and the degradation level to a storage unit (not shown) and store them in the storage unit (not shown). Alternatively, the output unit 150 may output the determination result and the degradation level to a display (not shown) and display them on the display (not shown). Alternatively, the output unit 150 may transmit the determination result and the degradation level to a device external to the information processing device 100.

[0092] The configuration example of the information processing device 100 according to the first embodiment of the present invention has been described above.

[0093] (1-3. Operational Example of Information Processing Device 100) Next, an example of the operation of the information processing device 100 according to the first embodiment of the present invention will be described with reference to FIG. 12 (and also with reference to FIGS. 1 to 11 as appropriate).

[0094] 12 is a flowchart showing an example of operation of the information processing device 100 according to the first embodiment of the present invention. As shown in Fig. 12, the input unit 110 acquires a captured image transmitted from the imaging unit 10, relayed by the communication antenna 20, and received by a communication unit (not shown) (S11). The identification unit 120 identifies an evaluation target area G30 based on the captured image G10 acquired by the input unit 110 (S12).

[0095] The evaluation unit 130 estimates the deterioration level D(T) of the determination area corresponding to the current time T, based on the evaluation target area G30 identified by the identification unit 120 (S13).

[0096] The determining unit 140 determines whether the deterioration level D(T) is higher than the threshold value Th (S14).

[0097] If the output unit 150 determines that the deterioration level D(T) is higher than the threshold value Th ("YES" in S14), it outputs a determination result indicating deterioration (S15). On the other hand, if the determination unit 140 determines that the deterioration level D(T) is not higher than the threshold value Th ("NO" in S14), it outputs a determination result indicating non-deterioration (S16). Note that the output unit 150 may output the deterioration level D(T) of the determination area in addition to or instead of the determination result.

[0098] If the input unit 110 continues the operation ("NO" in S17), it returns to S11 and continues the operation. On the other hand, if the output unit 150 ends the operation ("YES" in S17), it ends the operation.

[0099] An example of the operation of the information processing device 100 according to the first embodiment of the present invention has been described above.

[0100] (1-4. Effects of Information Processing Device 100) According to the first embodiment of the present invention, the degree of degradation for each decision area can be quantified at the pixel level.

[0101] In the first embodiment of the present invention, the angle between the imaging direction of the imaging unit 10 and the road may be set arbitrarily as long as the condition that the road is captured within the imaging range of the imaging unit 10 is satisfied. Furthermore, if a segmentation AI model is used that is generated based on learning data obtained in a special environment (for example, a shadowy environment or an environment with varying light and dark), a decrease in the accuracy of estimating the degree of deterioration can be suppressed even in the special environment.

[0102] For these reasons, the first embodiment of the present invention can be applied to various environments. Therefore, according to the first embodiment of the present invention, in formulating road safety or maintenance plans, decision-making based on more appropriate information such as the degree of deterioration for each judgment area can be supported.

[0103] The first embodiment of the present invention has been described above.

[0104] (2. Second Embodiment) Next, a second embodiment of the present invention will be described with reference to FIGS.

[0105] Here, white lines are often drawn on roads all at once, and deterioration of the white lines often progresses depending on the traffic conditions of vehicles on the white lines. For example, near lane entry points, vehicles frequently change lanes, so deterioration of white lines near the entry points progresses quickly. Therefore, the deterioration trends of white lines that are close to each other tend to be similar. In a second embodiment of the present invention, when determining whether a white line is deteriorated, the deterioration trends of white lines near the white line in question are taken into consideration.

[0106] In the following, a description of the common parts between the second embodiment of the present invention and the first embodiment of the present invention will be omitted, and only parts of the second embodiment of the present invention that are different from the first embodiment of the present invention will be described.

[0107] (2-1. System Configuration) An example of the configuration of an information processing device 200 according to the second embodiment of the present invention will be described with reference to FIG.

[0108] Fig. 13 is a diagram showing an example of the configuration of an information processing device 200 according to the second embodiment of the present invention. As shown in Fig. 13, the information processing device 200 according to the second embodiment of the present invention differs from the information processing device 100 according to the first embodiment of the present invention in that the determination unit 140 is replaced with a determination unit 240 and that the information processing device 200 includes a degradation level storage unit 260. Therefore, in the following description, the determination unit 240 and the degradation level storage unit 260 will be mainly described.

[0109] (Deterioration degree storage unit 260) The deterioration level storage unit 260 stores the deterioration level for each determination area estimated by the evaluation unit 130. Here, it is assumed that the deterioration level is estimated at a predetermined frame interval (for example, every one second) based on captured images transmitted continuously in time series from the imaging unit 10. At this time, it is assumed that the deterioration level estimated last time and the deterioration level estimated two times before last are stored in the deterioration level storage unit 260, with the current time being used as a reference.

[0110] In the following description, the degradation degree estimated last time with the current time as the reference will also be referred to as D(T-1). Also, the degradation degree estimated two times before with the current time as the reference will also be referred to as D(T-2). However, the degradation degrees stored in the degradation degree storage unit 260 do not have to be for two times, but may be for one time, or for three or more times.

[0111] Since the vehicle C1 equipped with the image capturing unit 10 is traveling on a road, the white line portion shown in the captured image also changes over time. If the frame interval at which the degradation level is estimated is not too long, the judgment area where the degradation level D(T-1) is estimated can be said to be close to the judgment area where the degradation level D(T) is estimated. Also, the judgment area where the degradation level D(T-2) is estimated and the judgment area where the degradation level D(T) is estimated can be said to be close to each other.

[0112] In the following description, the judgment area for which the degradation level D(T) has been estimated is a "judgment area" for which it will be judged whether or not there is degradation by the judgment unit 240. Moreover, the judgment areas for which the degradation levels D(T-1) and D(T-2) have been estimated are judgment areas for which it has already been judged whether or not there is degradation, and are "other judgment areas" of the judgment area for which the degradation level D(T) has been estimated.

[0113] (Judgment unit 240) Similar to the determination unit 140 according to the first embodiment of the present invention, the determination unit 240 determines whether a determination area is deteriorated based on whether the deterioration level D(T) of the determination area is higher than the threshold value Th. In the second embodiment of the present invention, the determination unit 240 lowers the threshold value Th based on the deterioration levels D(T-1) and D(T-2) of other determination areas showing a predetermined trend.

[0114] This makes it easier to determine that the judgment area is deteriorated when the deterioration levels D(T-1) and D(T-2) of the other judgment areas show a predetermined trend. On the other hand, the judgment unit 240 maintains the threshold value Th based on the fact that the deterioration levels D(T-1) and D(T-2) of the other judgment areas do not show a predetermined trend. Note that the extent to which the threshold value Th is reduced does not need to be limited. In the following description, the amount of reduction in the threshold value Th is also referred to as "α".

[0115] The determination unit 240 determines whether the deterioration levels D(T-1) and D(T-2) of the other determination areas show a predetermined trend based on whether the average value (hereinafter also referred to as "Dave") of the deterioration levels D(T-1) and D(T-2) of the other determination areas is higher than a predetermined reference value. For example, the average may be an arithmetic average or a weighted average in which the weight of D(T-1) is greater than the weight of D(T-2).

[0116] Alternatively, the determination unit 240 may determine whether the deterioration levels D(T-1) and D(T-2) of the other determination areas show a predetermined trend based on whether the deterioration levels D(T-1) and D(T-2) of the other determination areas are higher than a predetermined reference value. Note that the following description mainly assumes that the predetermined reference value is the same as the threshold value Th. However, the predetermined reference value may be a value different from the threshold value Th.

[0117] The configuration example of the information processing device 200 according to the second embodiment of the present invention has been described above.

[0118] (2-2. Operational Example of Information Processing Device 200) Next, an example of the operation of the information processing device 200 according to the second embodiment of the present invention will be described with reference to FIG. 14 (and also with reference to FIGS. 1 to 13 as appropriate).

[0119] Fig. 14 is a flowchart showing an example of operation of the information processing device 200 according to the second embodiment of the present invention. As shown in Fig. 14, steps S11 to S13 are executed. These steps S11 to S13 are executed in the same manner as steps S11 to S13 (Fig. 12) executed by the information processing device 100 according to the first embodiment of the present invention. Although not shown, the degradation level storage unit 260 stores the degradation level D(T) estimated in S13.

[0120] Next, the judgment unit 140 judges whether the deterioration level D(T-1) corresponding to time T-1 and the deterioration level D(T-2) corresponding to time T-2 are stored in the deterioration level storage unit 260 (S21).

[0121] If the degradation level D(T-1) corresponding to time T-1 and the degradation level D(T-2) corresponding to time T-2 are not stored in the degradation level storage unit 260 ("NO" in S21), the determination unit 140 proceeds to S14. On the other hand, if the degradation level D(T-1) corresponding to time T-1 and the degradation level D(T-2) corresponding to time T-2 are stored in the degradation level storage unit 260 ("YES" in S21), the determination unit 140 calculates the average value Dave of the degradation levels D(T-1) and D(T-2) (S22).

[0122] The determination unit 140 determines whether the average value Dave is higher than the threshold value Th (S23). If the determination unit 140 determines that the average value Dave is not higher than the threshold value Th (NO in S23), the process proceeds to S14. On the other hand, if the determination unit 140 determines that the average value Dave is higher than the threshold value Th (YES in S23), the determination unit 140 decreases the threshold value Th by the decrease amount α (S24) and proceeds to S14.

[0123] Subsequently, steps S14 to S17 are executed in the same manner as steps S14 to S17 (FIG. 12) executed by the information processing apparatus 100 according to the first embodiment of the present invention.

[0124] An example of the operation of the information processing device 200 according to the second embodiment of the present invention has been described above.

[0125] (2-3. Effects of Information Processing Device 200) According to the second embodiment of the present invention, the same effects as those achieved by the first embodiment of the present invention are achieved.

[0126] In the second embodiment of the present invention, when determining whether a white line has deteriorated, the deterioration tendency of white lines near the white line is taken into consideration, thereby making it possible to more accurately determine whether the white line has deteriorated.

[0127] The second embodiment of the present invention has been described above.

[0128] (3. Depth estimation AI model) Next, an example of the functions of the identification unit 220 according to the modified example will be described with reference to FIG.

[0129] Fig. 15 is a diagram illustrating an example of the functional configuration of the identification unit 220 according to a modified example. As illustrated in Fig. 15, the identification unit 220 includes a depth estimation AI model 221 and an area identification unit 222. The identification unit 220 inputs the captured image G10 acquired by the input unit 110 to the depth estimation AI model 221, and obtains second image data (hereinafter also referred to as "depth image") output from the depth estimation AI model 221.

[0130] The depth estimation AI model 221 is an inference model that outputs a depth image according to the characteristics of the subject shown in the captured image G10 based on the input of the captured image G10. Here, it is assumed that the characteristics of the subject are the distance from the imaging unit 10 to the subject. That is, the depth estimation AI model 221 recognizes the distance from the imaging unit 10 to the subject as the depth of the subject for each pixel, and outputs the depth of the subject for each pixel as a depth image.

[0131] The depth estimation AI model 221 may be a trained model generated in advance by machine learning. For example, the trained model may be generated by machine learning based on training data, which is image data showing a subject, and correct answer data in which the distance from the imaging unit to the subject shown in the training data is set as the correct answer label for each pixel.

[0132] The area identification unit 222 identifies pixels whose subject depths fall within a predetermined range based on the depth image, and identifies an evaluation target area based on the identified pixels. Here, it is considered that in the depth image, roads have small depths and subjects other than roads have large depths. Therefore, the area identification unit 222 may recognize the maximum height coordinate ymax among the height coordinates y of pixels whose depths are smaller than a threshold based on the depth image as the road disappearance position P20 (FIG. 5), and identify the evaluation target area based on the road disappearance position P20.

[0133] The method by which the area specifying unit 222 specifies the evaluation target area based on the road disappearance position P20 may be the same as the method by which the area specifying unit 122 (FIG. 4) specifies the evaluation target area based on the road disappearance position P20.

[0134] The above describes examples of the functions of the identification unit 220 according to the modified example.

[0135] (4. Hardware configuration example) Next, an example of the hardware configuration of the information processing device 100 according to the first embodiment of the present invention will be described.

[0136] Hereinafter, an example of the hardware configuration of the information processing device 900 will be described as an example of the hardware configuration of the information processing device 100 according to the first embodiment of the present invention. Note that the example of the hardware configuration of the information processing device 900 described below is merely one example of the hardware configuration of the information processing device 100. Therefore, the hardware configuration of the information processing device 100 may be such that unnecessary components are deleted from the hardware configuration of the information processing device 900 described below, or new components may be added. The hardware configuration of the information processing device 200 according to the second embodiment of the present invention may also be realized in the same manner as the hardware configuration of the information processing device 100 according to the first embodiment of the present invention.

[0137] 16 is a diagram showing the hardware configuration of an information processing device 900 as an example of the information processing device 100 according to the first embodiment of the present invention. The information processing device 900 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, a RAM (Random Access Memory) 903, a host bus 904, a bridge 905, an external bus 906, an interface 907, an input device 908, an output device 909, a storage device 910, and a communication device 911.

[0138] The CPU 901 functions as an arithmetic processing unit and control unit, and controls the overall operation of the information processing device 900 in accordance with various programs. The CPU 901 may also be a microprocessor. The ROM 902 stores programs used by the CPU 901, calculation parameters, etc. The RAM 903 temporarily stores programs used in the execution of the CPU 901, parameters that change as appropriate during the execution, etc. These are interconnected by a host bus 904 that is composed of a CPU bus, etc.

[0139] The host bus 904 is connected to an external bus 906, such as a PCI (Peripheral Component Interconnect / Interface) bus, via a bridge 905. The host bus 904, bridge 905, and external bus 906 do not necessarily need to be configured separately, and these functions may be implemented on a single bus. The external bus 906 is a data transmission path between external devices and the bridge 905. The interface 907 is a connection interface for external devices.

[0140] The input device 908 is composed of input means such as a mouse, keyboard, touch panel, buttons, microphone, switches, and levers that allow the user to input information, and an input control circuit that generates an input signal based on the user's input and outputs it to the CPU 901. By operating this input device 908, the user operating the information processing device 900 can input various data to the information processing device 900 and instruct the information processing device 900 to perform processing operations.

[0141] The output device 909 includes, for example, a display device such as a CRT (Cathode Ray Tube) display device, a liquid crystal display (LCD) device, an OLED (Organic Light Emitting Diode) device, or a lamp, and an audio output device such as a speaker.

[0142] The storage device 910 is a device for storing data. The storage device 910 may include a storage medium, a recording device for recording data on the storage medium, a reading device for reading data from the storage medium, and a deletion device for deleting data recorded on the storage medium. The storage device 910 is configured, for example, with an HDD (Hard Disk Drive). This storage device 910 drives a hard disk and stores programs executed by the CPU 901 and various data.

[0143] The communication device 911 is, for example, a communication interface configured with a communication device for connecting to a network, etc. The communication device 911 may be compatible with either wireless communication or wired communication.

[0144] An example of the hardware configuration of the information processing device 100 according to the first embodiment of the present invention has been described above.

[0145] (5. Supplementary Information) Although the preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings, the present invention is not limited to these examples. It is clear that a person skilled in the art to which the present invention pertains can conceive of various modifications and alterations within the scope of the technical ideas set forth in the claims, and it is understood that these also naturally fall within the technical scope of the present invention.

[0146] As described above, the segmentation AI model 121 and the depth estimation AI model 221 may each be generated by machine learning. The amount of calculation required to generate these models can be enormous compared to the amount of calculation required for inference using the models. Therefore, it is desirable to generate these models using a high-spec computer for learning.

[0147] In the above description, it is mainly assumed that the identification unit 120 uses an AI model to identify the evaluation target area. However, the identification unit 120 may identify the evaluation target area based on a rule without using an AI model. For example, the identification unit 120 may identify the evaluation target area based on the captured image G10. Specifically, the identification unit 120 may identify a predetermined range (e.g., the lower part of the captured image G10) from the captured image G10 as the evaluation target area. The lower part of the captured image G10 may be an area extending from the bottom of the captured image G10 to a predetermined height of the captured image G10. The predetermined height of the captured image G10 may be a position elevated from the bottom of the captured image G10 by half the height of the captured image G10, or may be a position elevated from the bottom of the captured image G10 by one-third the height of the captured image G10. [Explanation of symbols]

[0148] 1 System 10. Imaging unit 100, 200 Information processing device 110 Input section 120, 220 Specific part 121 First Segmentation AI Model 122, 222 Area Specific Section 130 Evaluation Department 131 Second Segmentation AI Model 132 Deterioration degree estimation section 140, 240 Judgment section 150 Output section 20 Communication Antenna 221 Depth Estimation AI Model 260 Deterioration degree storage section

Claims

1. an acquisition unit that acquires first image data obtained by the imaging unit; a specifying unit that specifies an evaluation target area based on the first image data; an evaluation unit that performs a predetermined evaluation based on the evaluation target area; An information processing device comprising:

2. the identification unit acquires second image data corresponding to the characteristics of the subject shown in the first image data, which is output from a first inference model to which the first image data has been input, and identifies the evaluation target area based on the second image data; The information processing device according to claim 1 .

3. the feature of the subject is the type of the subject or the distance from the imaging unit to the subject; The information processing device according to claim 2 .

4. the feature of the subject is the type of the subject, the first inference model recognizes the type of the object for each pixel and outputs the type of the object as the second image data; The information processing device according to claim 3 .

5. the specifying unit specifies pixels where the type of the subject is a predetermined type, and specifies the evaluation target area based on the specified pixels. The information processing device according to claim 4 .

6. the feature of the subject is a distance from the imaging unit to the subject, The first inference model recognizes the distance from the imaging unit to the subject as the depth of the subject for each pixel, and outputs the depth of the subject as the second image data. The information processing device according to claim 3 .

7. the specifying unit specifies pixels whose depths of the subject belong to a predetermined range, and specifies the evaluation target area based on the specified pixels; The information processing device according to claim 6 .

8. the evaluation unit acquires third image data corresponding to the type of subject appearing in the evaluation target area, which is output from a second inference model to which the evaluation target area is input, and performs the predetermined evaluation based on the third image data. The information processing device according to claim 1 .

9. The second inference model recognizes, for each pixel, the type of subject as a non-degraded lane marking class that is not deteriorated on the road or a deteriorated lane marking class that is deteriorated on the road; the evaluation unit performs the predetermined evaluation by grouping consecutive pixels of the non-deteriorated lane line class and pixels of the deteriorated lane line class as a judgment area and estimating the degree of deterioration of the judgment area based on the number of pixels of the non-deteriorated lane line class and the number of pixels of the deteriorated lane line class. The information processing device according to claim 8 .

10. the information processing device includes a determination unit that determines whether the determination area is deteriorated based on whether the degree of deterioration is higher than a threshold value. The information processing device according to claim 9 .

11. the determination unit reduces the threshold value based on the fact that the deterioration levels of other determination areas show a predetermined tendency. The information processing device according to claim 10.

12. the determination unit determines whether the deterioration level of the other judgment area shows the predetermined trend based on whether the average value of the deterioration levels of the other judgment areas or the deterioration level of the other judgment area is higher than a predetermined reference value. The information processing device according to claim 11.

13. The first inference model is a trained model generated by machine learning. The information processing device according to claim 2 .

14. acquiring first image data obtained by an imaging unit; Identifying an evaluation target area based on the first image data; performing a predetermined evaluation based on the evaluation target area; 2. A computer-implemented information processing method, comprising:

15. Computer, an acquisition unit that acquires first image data obtained by the imaging unit; a specifying unit that specifies an evaluation target area based on the first image data; an evaluation unit that performs a predetermined evaluation based on the evaluation target area; A program that functions as a

Images