Information processing device, information processing method, and program

Abstract

To provide an information processing device, information processing method, and program, which allow for starting driving assistance at an earlier stage.SOLUTION: An information processing device according to an embodiment comprises a control unit. The control unit is configured to estimate the attitude of a vehicle-mounted camera, perform calibration processing in stages, and cause content of image recognition processing for an image captured by the vehicle-mounted camera to be changed in stages according to the step of the calibration processing.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The disclosed embodiments relate to an information processing device, an information processing method, and a program. [Background technology]

[0002] There is a vehicle control system that detects a detection target by performing image recognition processing on an image captured by an on-board camera and uses the detection result for driving support of the vehicle. In such a vehicle control system, since the attitude of the on-board camera attached to the vehicle significantly affects the detection accuracy of the detection target, a calibration process is performed to adjust the attitude of the on-board camera for a predetermined period of time from the time the on-board camera is attached.

[0003] However, the vehicle control system cannot sufficiently improve the detection accuracy of the detection target by the image recognition process until the calibration process is completed. For this reason, there is a vehicle control system that suppresses driving assistance from the start to the completion of the calibration process (for example, see Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-6275 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, if driving assistance is suppressed until the calibration process is completed, a problem occurs in that the start time of driving assistance is delayed.

[0006] One aspect of the embodiment has been made in consideration of the above, and aims to provide an information processing device, an information processing method, and a program that can start driving assistance earlier. [Means for solving the problem]

[0007] According to an embodiment, there is provided an information processing device including a control unit that estimates an attitude of an on-board camera, performs a calibration process in stages, and changes content of an image recognition process of an image captured by the on-board camera in stages according to a stage of the calibration process. Effect of the Invention

[0008] An information processing device, an information processing method, and a program according to an aspect of an embodiment provide an effect of enabling driving assistance to be started earlier. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram (part 1) outlining the posture estimation method according to the embodiment. [Diagram 2] FIG. 2 is a schematic explanatory diagram (part 2) of the posture estimation method according to the embodiment. [Diagram 3] FIG. 3 is a schematic explanatory diagram (part 3) of the posture estimation method according to the embodiment. [Figure 4] FIG. 4 is a block diagram illustrating an example of the configuration of the in-vehicle device according to the embodiment. [Diagram 5] FIG. 5 is an explanatory diagram (part 1) of the road surface ROI and the superimposed ROI. [Figure 6] FIG. 6 is an explanatory diagram (part 2) of the road surface ROI and the superimposed ROI. [Figure 7] FIG. 7 is a block diagram illustrating an example of the configuration of the posture estimation unit. [Figure 8] FIG. 8 is an explanatory diagram showing an example of an instruction from the in-car device to the external device according to the embodiment. [Figure 9] FIG. 9 is an explanatory diagram showing an example of an instruction from the in-car device to the external device according to the embodiment. [Figure 10] FIG. 10 is a flowchart showing a processing procedure executed by the in-vehicle device according to the embodiment. [Figure 11]FIG. 11 is a flowchart showing a processing procedure executed by the in-vehicle device according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, with reference to the accompanying drawings, an embodiment of an information processing device, an information processing method, and a program disclosed in the present application will be described in detail. Note that the present invention is not limited to the embodiment described below. In addition, in the following, it is assumed that the information processing device according to the embodiment is an in-vehicle device 10 mounted on a vehicle. The in-vehicle device 10 is, for example, a drive recorder.

[0011] The in-vehicle device 10 according to the embodiment is a device that records images of the surroundings of the vehicle captured by an in-vehicle camera (hereinafter, referred to as "camera 11" (see FIG. 4)). The in-vehicle device 10 also executes a predetermined program to estimate the mounting attitude of the camera 11 mounted on the vehicle, and gradually executes a calibration process based on the estimated attitude of the camera 11. The in-vehicle device 10 also executes a predetermined program to gradually change the content of image recognition of the image captured by the camera 11 according to the stage of the calibration process.

[0012] First, a method in which the in-vehicle device 10 estimates the mounting attitude of the camera 11 will be described with reference to Figs. 1 to 3. Figs. 1 to 3 are schematic explanatory diagrams (part 1) to (part 3) of the attitude estimation method according to the embodiment. Prior to describing the attitude estimation method according to the embodiment, an attitude estimation method according to a comparative example and problems therein will be described in more detail. Fig. 1 shows the contents.

[0013] In a comparative attitude estimation method, feature points on the road surface are extracted from a rectangular ROI (Region of Interest) set in a captured image, and the attitude of the vehicle-mounted camera is estimated based on the optical flow that indicates the movement of these feature points between frames.

[0014] When estimating the attitude of the camera 11 based on the optical flow of characteristic points on the road surface, the characteristic points on the road surface to be extracted include corners of road markings such as lanes.

[0015] However, as shown in Fig. 1, for example, lanes in a captured image are captured so as to converge toward a vanishing point according to the law of perspective. Therefore, when a rectangular ROI (hereinafter referred to as "rectangular ROI 30-1") is used, feature points such as three-dimensional objects other than the road surface are easily extracted in the upper left or upper right of the rectangular ROI 30-1.

[0016] FIG. 1 shows an example in which optical flows Op1 and Op2 are extracted based on feature points on a road surface, and an optical flow Op3 is extracted based on feature points of a three-dimensional object other than the road surface.

[0017] Here, for example, in the case of an algorithm that extracts a pair of parallel line segments in real space and estimates the posture of the camera 11 based on the pair of line segments, the pair of optical flows Op1 and Op2 is a correct combination in posture estimation (hereinafter referred to as a "correct flow"), whereas the pair of optical flows Op1 and Op3 is an incorrect combination (hereinafter referred to as an "incorrect flow").

[0018] Based on such erroneous flows, the posture of the camera 11 cannot be correctly estimated. The posture of the camera 11 is determined by estimating the rotation angles of the PAN, TILT, and ROLL axes for each pair of extracted optical flows, and determining the axis deviation based on the median value of the histogram. Therefore, the more erroneous flows there are, the lower the accuracy of the posture estimation of the camera 11 may be.

[0019] To deal with this, it is conceivable to set ROI 30 according to the shape of the road surface reflected in the captured image, instead of rectangular ROI 30-1. In this case, however, if the calibration values ​​(mounting position and PAN, TILT, ROLL) of the camera 11 are not known in the first place, it is not possible to set ROI 30 according to the shape of the road surface (hereinafter referred to as "road surface ROI 30-2").

[0020] Therefore, in the posture estimation method of the embodiment, the control unit 15 (see FIG. 4) provided in the in-vehicle device 10 executes a first posture estimation process using a rectangular ROI 30-1 set in a rectangular shape when the camera 11 is in the initial mounting stage, and executes a second posture estimation process using a superimposed ROI 30-S set according to the shape of the road surface when the camera 11 is not in the initial mounting stage.

[0021] Here, "when it is in the early stages of installation" refers to when the installation state of camera 11 is "in a first state." The "first state" is a state that allows estimation that camera 11 is in the early stages of installation. For example, the first state is a state in which the time that has elapsed since camera 11 was installed is less than a predetermined elapsed time. Also, for example, the first state is a state in which the number of calibrations since camera 11 was installed is less than a predetermined number. On the other hand, "when it is not in the early stages of installation" refers to when the installation state of camera 11 is "in a second state" that is different from the first state.

[0022] 2, in the posture estimation method according to the embodiment, the control unit 15 executes posture estimation processing using the optical flow of the rectangular ROI 30-1 when the camera 11 is initially attached (step S1). When the control unit 15 is not initially attached to the camera 11, the control unit 15 executes posture estimation processing using the optical flow of the road surface ROI 30-2 in the rectangular ROI 30-1 (step S2). The road surface ROI 30-2 in the rectangular ROI 30-1 refers to the overlapping ROI 30-S, which is the overlapping portion where the rectangular ROI 30-1 and the road surface ROI 30-2 overlap.

[0023] 2, when the optical flow of the superimposed ROI 30-S is used, the occurrence of erroneous flows is reduced. For example, the optical flows Op4, Op5, and Op6 included in the processing target in step S1 are not included in step S2.

[0024] Figure 3 shows a comparison between the case where rectangular ROI 30-1 is used and the case where overlapping ROI 30-S is used. When overlapping ROI 30-S is used, there are fewer false flows, fewer estimations, and higher estimation accuracy than when rectangular ROI 30-1 is used. However, the estimation time is slow and a calibration value is required.

[0025] However, the disadvantages regarding the estimation time and the calibration value are compensated for by performing the posture estimation process using the rectangular ROI 30-1 at the initial stage of mounting the camera 11 in step S1.

[0026] That is, according to the posture estimation method of the embodiment, when rectangular ROI 30-1 is used and when superimposed ROI 30-S is used, the advantages of each method can be used to compensate for their respective disadvantages, thereby improving the accuracy of posture estimation of camera 11.

[0027] In this manner, in the posture estimation method according to the embodiment, when the camera 11 is in the initial stage of installation, the control unit 15 executes a first posture estimation process using a rectangular ROI 30-1 set in a rectangular shape, and when the camera 11 is not in the initial stage of installation, the control unit 15 executes a second posture estimation process using a superimposed ROI 30-S set according to the shape of the road surface.

[0028] Therefore, according to the posture estimation method according to the embodiment, the accuracy of posture estimation of the camera 11 can be improved.

[0029] An example of the configuration of the in-vehicle device 10 to which the attitude estimation method according to the embodiment described above is applied will be described in more detail below.

[0030] Fig. 4 is a block diagram showing a configuration example of the in-vehicle device 10 according to the embodiment. Note that Fig. 4 and Fig. 7 shown later show only components necessary for explaining the features of the embodiment, and descriptions of general components are omitted.

[0031] In other words, each component shown in Figures 4 and 7 is a functional concept, and does not necessarily have to be physically configured as shown in the figures. For example, the specific form of distribution and integration of each block is not limited to that shown in the figures, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0032] In addition, in the description using FIG. 4 and FIG. 7, the description of components that have already been described may be simplified or omitted.

[0033] As shown in FIG. 4, the in-vehicle device 10 according to the embodiment includes a camera 11, a sensor unit 12, a notification device 13, a storage unit 14, and a control unit 15.

[0034] The camera 11 includes an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), and captures an image of a predetermined imaging range using the imaging element. The camera 11 is attached to various locations of the vehicle, such as the windshield or dashboard, so as to capture an image of a predetermined imaging range in front of the vehicle.

[0035] The sensor unit 12 is various sensors mounted on the vehicle, including, for example, a vehicle speed sensor, a G sensor, etc. The notification device 13 is a device that notifies information related to calibration. The notification device 13 is realized by, for example, a display, a speaker, etc.

[0036] The storage unit 14 is realized by a storage device such as a RAM (Random Access Memory), a flash memory, etc. In the example of Fig. 4, the storage unit 14 stores image information 14a and attachment information 14b.

[0037] The image information 14a stores images captured by the camera 11. Thus, when the vehicle in which the in-vehicle device 10 is mounted is involved in an accident, the in-vehicle device 10 can output the image information 14a, thereby making it possible to use the image information 14a to reproduce the accident situation and to clarify the cause of the accident.

[0038] The mounting information 14b is information related to the mounting of the camera 11. The mounting information 14b includes design values ​​related to the mounting position and posture of the camera 11, and the above-mentioned calibration values. The mounting information 14b may further include various information that can be used to determine whether the mounting of the camera 11 is still in its early stages, such as the date and time of mounting, the time elapsed since mounting, and the number of calibrations performed since mounting.

[0039] The control unit 15 is a controller, and is realized by a central processing unit (CPU), a micro processing unit (MPU), or the like executing a program according to an embodiment (not shown) stored in the storage unit 14 using a RAM as a working area. The control unit 15 can also be realized by an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0040] The control unit 15 has a mode setting unit 15a, a posture estimation unit 15b, and a calibration execution unit 15c, and realizes or executes the functions and actions of information processing described below.

[0041] Mode setting unit 15a sets the posture estimation mode, which is the execution mode of posture estimation unit 15b, to the first mode when camera 11 is in the initial attachment period. Also, mode setting unit 15a sets the posture estimation mode of posture estimation unit 15b to the second mode when camera 11 is not in the initial attachment period.

[0042] When the execution mode is set to the first mode, the posture estimation unit 15b executes a first posture estimation process using the optical flow of the rectangular ROI 30-1. When the execution mode is set to the second mode, the posture estimation unit 15b executes a second posture estimation process using the optical flow of the road surface ROI 30-2 (i.e., the superimposed ROI 30-S) in the rectangular ROI 30-1.

[0043] Here, the road surface ROI 30-2 and the superimposed ROI 30-S will be specifically described. Fig. 5 is an explanatory diagram (part 1) of the road surface ROI 30-2 and the superimposed ROI 30-S. Fig. 6 is an explanatory diagram (part 2) of the road surface ROI 30-2 and the superimposed ROI 30-S.

[0044] 5, the road surface ROI 30-2 is set as the ROI 30 according to the shape of the road surface reflected in the captured image. The road surface ROI 30-2 is set to be an area of ​​about half a lane to one lane on the left and right sides of the host vehicle's lane and about 20 m deep based on a known calibration value.

[0045] As shown in Fig. 5, the overlapping ROI 30-S is an overlapping portion where the rectangular ROI 30-1 and the road surface ROI 30-2 overlap. More abstractly, the overlapping ROI 30-S can be said to be a trapezoidal region obtained by removing the upper left region C-1 and the upper right region C-2 from the rectangular ROI 30-1 as shown in Fig. 6. By using the region obtained by removing the upper left region C-1 and the upper right region C-2 as the processing target region for pose estimation, the occurrence of erroneous flows can be reduced and the accuracy of pose estimation can be improved.

[0046] A configuration example of posture estimation unit 15b will be described in more detail. Fig. 7 is a block diagram showing a configuration example of posture estimation unit 15b. As shown in Fig. 7, posture estimation unit 15b has acquisition unit 15ba, feature point extraction unit 15bb, feature point tracking unit 15bc, line segment extraction unit 15bd, calculation unit 15be, noise removal unit 15bf, and determination unit 15bg.

[0047] The acquisition unit 15ba acquires the captured image of the camera 11 and stores it in the image information 14a. The feature point extraction unit 15bb sets each ROI 30 according to the execution mode of the posture estimation unit 15b for the captured image stored in the image information 14a. In addition, the feature point extraction unit 15bb extracts feature points included in the set ROI 30.

[0048] The feature point tracking unit 15bc tracks each feature point extracted by the feature point extraction unit 15bb across frames and extracts an optical flow for each feature point. The line segment extraction unit 15bd removes noise components from the optical flows extracted by the feature point tracking unit 15bc and extracts a group of line segment pairs based on each optical flow.

[0049] The calculation unit 15be calculates the rotation angles of the PAN, TILT, and ROLL axes for each pair of line segments extracted by the line segment extraction unit 15bd using the algorithm of Non-Patent Document 1.

[0050] The noise removal unit 15bf removes noise caused by low speed and steering angle from each angle calculated by the calculation unit 15be based on the sensor value of the sensor unit 12. The determination unit 15bg creates a histogram of each angle from which noise has been removed, and determines the estimated values ​​of the PAN, TILT, and ROLL angles based on the median. The determination unit 15bg also stores the determined angle estimates in the attachment information 14b.

[0051] Returning to the description of Fig. 4, the calibration execution unit 15c executes calibration based on the estimation result by the attitude estimation unit 15b. Specifically, the calibration execution unit 15c executes the calibration process by comparing the angle estimation value estimated by the attitude estimation unit 15b with the design value included in the attachment information 14b and correcting the error.

[0052] The calibration execution unit 15c executes a first calibration process based on the estimated posture of the camera 11 during a first predetermined period during which the mounting state of the camera 11 is in a first state, i.e., during a first predetermined period from the time the camera 11 is mounted.

[0053] Thereafter, the calibration execution unit 15c executes a second calibration process based on the attitude of the camera 11 estimated during a second predetermined period in which the mounting state of the camera 11 is in the second state.

[0054] That is, the calibration execution unit 15c executes the second calibration process, which is more detailed than the first calibration process, for a second predetermined period after the first calibration process is completed. In this manner, the control unit 15 executes the calibration process in stages.

[0055] Then, the calibration execution unit 15c notifies the external device 50 of the corrected calibration values, and causes the external device 50 to gradually change the content of the image recognition process according to the stage of the calibration process.

[0056] The external device 50 here is a device that provides driving support for the vehicle, for example, by performing image recognition processing on the image captured by the camera 11 to detect obstacles, detect parking spaces, automatically drive the vehicle, and automatically park the vehicle. The external device 50 is connected to an information management server 51 via a communication network 100 such as the Internet so as to be able to wirelessly communicate with the information management server 51.

[0057] According to the in-vehicle device 10, even if the calibration process of the camera 11 has not been completely completed, the external device 50 can be caused to execute image recognition processing according to the stage of the calibration process, thereby enabling driving assistance by the external device 50 to be started earlier.

[0058] Here, an example of an instruction related to the image recognition process that the in-car device 10 issues to the external device 50 depending on the stage of the calibration process will be described with reference to Fig. 8 and Fig. 9. Fig. 8 and Fig. 9 are explanatory diagrams showing an example of an instruction from the in-car device 10 according to the embodiment to the external device 50.

[0059] 8, the calibration execution unit 15c executes a first calibration process for a first predetermined period from the attachment of the camera 11. Then, the calibration execution unit 15c instructs the external device 50 to prohibit the execution of the image recognition process until the first calibration process is completed.

[0060] Accordingly, the external device 50 does not detect the detection target by image recognition processing, and does not notify or warn the user of the detection target (for example, an obstacle, etc.) In other words, the external device 50 does not perform driving assistance until the first predetermined period has elapsed since the camera 11 was attached, because even the first calibration process has not been completed and the detection accuracy of the target by image recognition processing is relatively low.

[0061] This allows the in-vehicle device 10 to prevent the external device 50, with low detection accuracy, from erroneously notifying the user of or erroneously warning the user about the presence of a detection target that does not actually exist.

[0062] After that, when the first calibration process is completed, the calibration execution unit 15c executes a second calibration process that is more detailed than the first calibration process for a second predetermined period from the completion of the first calibration process.

[0063] Then, the calibration execution unit 15c instructs the external device 50 to execute a first image recognition process until the second calibration process is completed. At this time, the calibration execution unit 15c causes the external device 50 to detect a detection target existing within a range of a first predetermined distance (for example, 5 m) from the camera 11, and causes the external device 50 to execute a first image recognition process that notifies (including a warning unit) the user of the detection result.

[0064] This allows the in-vehicle device 10 to notify the user of the presence of a detection target at a relatively short distance that is detected by the external device 50 with higher detection accuracy than when the first calibration has not been completed.

[0065] In other words, even if the calibration process has not been completely completed, the in-vehicle device 10 can start driving assistance early by having the external device 50 notify the driver of the detection result according to the detection accuracy of the detection target at that time.

[0066] Furthermore, the in-vehicle device 10 does not allow the external device 50, which has not yet completed the calibration process and does not have sufficient detection accuracy for distant detection targets, to detect a detection target that exists farther away than the first predetermined distance. This allows the in-vehicle device 10 to prevent the user from being erroneously notified or warned about the presence of a distant detection target that does not actually exist.

[0067] After the second calibration process is completed, the calibration execution unit 15c then instructs the external device 50 to execute a second image recognition process that has higher sensitivity than the first image recognition process.

[0068] At this time, the calibration execution unit 15c causes the external device 50 to detect a detection target present within a range of a second predetermined distance (e.g., 10 m) from the camera 11 that is longer than the first predetermined distance (e.g., 5 m), and causes the external device 50 to execute a second image recognition process to notify the user of the detection result (including a warning unit).

[0069] This allows the in-vehicle device 10 to appropriately notify the user of the presence of a detection target at a relatively long distance that is detected by the external device 50 after the calibration process has been completely completed.

[0070] 8 is an example of an instruction for the external device 50 to perform image recognition processing. The calibration execution unit 15c may be configured to issue an instruction for the external device 50 as shown in FIG.

[0071] For example, as shown in FIG. 9, the calibration execution unit 15c may provide instructions to the external device 50 similar to the instructions shown in FIG. 8 until a first predetermined period has elapsed since the installation of the camera 11, and then may provide instructions different from the instructions shown in FIG. 8 during a second predetermined period.

[0072] Specifically, the calibration execution unit 15c instructs the external device 50 to execute the first image recognition process during a second predetermined period from the completion of the first calibration process to the completion of the second calibration process. However, the calibration execution unit 15c prohibits the external device 50 from notifying the user of the detection result, and instructs the external device 50 to transmit the detection result to the information management server 51.

[0073] In this way, the in-vehicle device 10 can prevent the external device 50 with low detection accuracy from notifying the user of an uncertain detection result, and can use the detection result at that time to investigate the cause when an accident occurs by transmitting the detection result to the information management server 51. In this case, after completing the second calibration process, the calibration execution unit 15c issues instructions to the external device 50 similar to those shown in FIG.

[0074] Next, a process procedure executed by the in-vehicle device 10 will be described with reference to Fig. 10 and Fig. 11. Fig. 10 and Fig. 11 are flowcharts showing a process procedure executed by the in-vehicle device 10 according to the embodiment.

[0075] 10, the control unit 15 of the in-vehicle device 10 determines whether or not the camera 11 is in an initial installation state (step S101). If the camera 11 is in an initial installation state (step S101, Yes), the control unit 15 sets the attitude estimation mode to the first mode (step S102).

[0076] Then, the control unit 15 executes a posture estimation process using the optical flow of the rectangular ROI 30-1 (step S103). On the other hand, if the mounting is not in the initial stage (step S101, No), the control unit 15 sets the posture estimation mode to the second mode (step S104).

[0077] Then, the control unit 15 executes a posture estimation process using the optical flow of the road surface ROI 30-2 in the rectangular ROI 30-1 (step S105).Then, the control unit 15 determines whether or not there is a process end event (step S106).

[0078] The process end event is, for example, the arrival of a non-execution time period of the posture estimation process, the stopping of the engine, the turning off of the power, etc. If the process end event has not occurred (step S106, No), the control unit 15 repeats the process from step S101. If the process end event has occurred (step S106, Yes), the control unit 15 ends the process.

[0079] Furthermore, control unit 15 executes the process shown in Fig. 11 in parallel with the process shown in Fig. 10. As shown in Fig. 11, control unit 15 determines whether or not it is within a first predetermined period from the attachment of camera 11 (step S201). When control unit 15 determines that it is not within the first predetermined period from the attachment of camera 11 (step S201, No), it moves the process to step S205.

[0080] When it is determined that the first predetermined period has elapsed since the attachment of the camera 11 (Yes at step S201), the control unit 15 executes a first calibration process (step S202). Then, the control unit 15 issues an instruction to the external device 50 to prohibit the execution of the image recognition process (step S203).

[0081] Next, the control unit 15 determines whether the first calibration process is completed (step S204). When the control unit 15 determines that the first calibration process is not completed (step S204, No), the control unit 15 moves the process to step S202.

[0082] When the control unit 15 determines that the first calibration process is completed (step S204, Yes), it then determines whether or not it is within a second predetermined period (step S205). When the control unit 15 determines that it is not within the second predetermined period (step S205, No), it moves the process to step S209.

[0083] When it is determined that it is within the second predetermined period (step S205, Yes), the control unit 15 executes the second calibration process (step S206). Then, the control unit 15 instructs the external device 50 to execute the first image recognition process (step S207).

[0084] Next, the control unit 15 determines whether the second calibration process is completed (step S208). If the control unit 15 determines that the second calibration process is not completed (step S208, No), the control unit 15 moves the process to step S206.

[0085] When the control unit 15 determines that the second calibration process is completed (Yes at step S208), it instructs the external device 50 to perform a second image recognition process (step S209), and ends the process.

[0086] The programs according to the embodiments are recorded on a computer-readable recording medium such as a hard disk, a flexible disk (FD), a CD-ROM, a Magneto-Optical disk (MO), a Digital Versatile Disc (DVD), or a Universal Serial Bus (USB) memory, and can be executed by being read from the recording medium by a computer.

[0087] As an appendix, the features of the present invention are as follows: (1) A control unit is provided for gradually executing a calibration process of an on-board camera and gradually changing the content of an image recognition process of an image captured by the on-board camera according to a stage of the calibration process. Information processing device. (2) The control unit is A first calibration process is executed for a first predetermined period from the time when the vehicle-mounted camera is attached; prohibiting execution of the image recognition process until the first calibration process is completed; performing a second calibration process, the second calibration process being more detailed than the first calibration process, during a second predetermined period after the first calibration process is completed; executing a first image recognition process from the completion of the first calibration process to the completion of the second calibration process; After the second calibration process is completed, a second image recognition process having a higher sensitivity than the first image recognition process is executed. The information processing device according to (1). (3) The first image recognition process includes: A process of detecting a detection target present within a range of a first predetermined distance from the vehicle-mounted camera and notifying a user of the detection result, The second image recognition process includes: A process of detecting a detection target present within a range from the vehicle-mounted camera to a second predetermined distance longer than the first predetermined distance, and notifying a user of the detection result. The information processing device according to (2). (4) The first image recognition process includes: This is a process of detecting a detection target and prohibiting notification of the detection result to the user. The second image recognition process includes: A process of detecting a detection target and notifying a user of the detection result. The information processing device according to (2). (5) The control unit is transmitting results of the first image recognition processing and the second image recognition processing to a server; The information processing device according to (4). (6) The control unit is A storage unit is provided for storing images captured by the vehicle-mounted camera. The information processing device according to any one of (1) to (5). (7) An information processing method executed by an information processing device, A calibration process of the vehicle-mounted camera is executed in stages, and the content of the image recognition process of the image captured by the vehicle-mounted camera is changed in stages according to the stage of the calibration process; An information processing method comprising: (8) Calibration processing of the vehicle-mounted camera is performed in stages, and the content of image recognition processing of the image captured by the vehicle-mounted camera is changed in stages according to the stage of the calibration processing. program. (9) An information processing method executed by an information processing device, When a first calibration process performed when the vehicle-mounted camera is attached is completed, a first image recognition process is executed, and when a second calibration process is completed after the first calibration process is completed, a second image recognition process is executed in a range different from that of the first image recognition process; An information processing method comprising: (10) The information processing method according to (9), wherein the first calibration process is performed in a rectangular range, and the second calibration process is performed in a specified range according to a road surface shape.

[0088] Further advantages and modifications may readily occur to those skilled in the art. Thus, the invention in its broader aspects is not limited to the specific details and representative embodiments shown and described above. Accordingly, various modifications may be made without departing from the spirit or scope of the general inventive concept as defined by the appended claims and equivalents thereof. [Explanation of symbols]

[0089] 10 Onboard equipment 11 Camera 12 Sensor section 13 Notification Devices 14 Storage section 14a Image information 14b Installation Information 15 Control section 15a Mode setting section 15b Posture estimation section 15ba acquisition department 15bb Feature point extraction unit 15bc Feature point tracking unit 15bd Line segment extraction section 15be calculation unit 15bf Noise reduction section 15bg decision section 15c Calibration execution section 30 ROI 30-1 Rectangular ROI 30-2 Road surface ROI 30-S Overlapping ROI 50 External device 51 Information Management Server

Claims

1. A method for detecting a vehicle-mounted camera according to claim 1, comprising: setting a rectangular first processing target area in an image output from an installed vehicle-mounted camera; and executing a first calibration process for the vehicle-mounted camera based on an optical flow of a first feature point identified within the first processing target area. a second processing target area having a substantially trapezoidal shape is set by removing an area other than the road surface from the first processing target area, and a second calibration process is performed on the vehicle-mounted camera based on an optical flow of a second feature point identified on the road surface within the second processing target area; After the first calibration process is completed, a first image recognition process is started based on the captured image for driving assistance; After the second calibration process is completed, a second image recognition process is started based on the captured image for the driving assistance. An information processing device having a control unit.

2. The first calibration process is performed in a rectangular range, and the second calibration process is performed in a range according to a road surface shape. The information processing device according to claim 1 .

3. The first image recognition process is not executed until the first calibration process is completed. The information processing device according to claim 1 .

4. executing the first image recognition process while executing the second calibration process; The information processing device according to claim 1 .

5. Transmitting results of the first image recognition process and the second image recognition process to a server. The information processing device according to claim 1 .

6. storing images captured by the vehicle-mounted camera; The information processing device according to claim 1 .

7. A control unit of the information processing device A rectangular first processing target area is set in a captured image output from the mounted vehicle-mounted camera, and a first calibration process is performed on the vehicle-mounted camera based on an optical flow of a first feature point identified within the first processing target area. a second processing target area having a substantially trapezoidal shape is set by removing an area other than the road surface from the first processing target area, and a second calibration process is performed on the vehicle-mounted camera based on an optical flow of a second feature point identified on the road surface within the second processing target area; After the first calibration process is completed, a first image recognition process is started based on the captured image for driving assistance; after the second calibration process is completed, starting a second image recognition process based on the captured image for the driving assistance; An information processing method comprising:

8. A method for detecting a vehicle-mounted camera according to claim 1, comprising: setting a rectangular first processing target area in an image output from the mounted vehicle camera; and executing a first calibration process for the vehicle-mounted camera based on an optical flow of a first feature point identified within the first processing target area. a second processing target area having a substantially trapezoidal shape is set by removing an area other than the road surface from the first processing target area, and a second calibration process is performed on the vehicle-mounted camera based on an optical flow of a second feature point identified on the road surface within the second processing target area; After the first calibration process is completed, a first image recognition process is started based on the captured image for driving assistance; causing the computer to execute a procedure of starting a second image recognition process based on the captured image for the driving assistance after the second calibration process is completed; program.

Images