Information processing apparatus, information processing method, and storage medium

Abstract

The present disclosure relates to an information processing apparatus, an information processing method, and a storage medium. In a system that performs target detection based on images, the influence of a failure of a camera is suppressed. An information processing apparatus performs the following actions: acquires a plurality of images acquired by a plurality of cameras mounted on a moving body; performs detection processing that detects a prescribed target of the surroundings of the moving body based on the plurality of images; and determines, for each of the plurality of cameras, whether an abnormality has occurred, and in the case where it is determined that an abnormality has occurred in at least one of the plurality of cameras, the information processing apparatus generates virtual data corresponding to the camera for which an abnormality has been determined, and performs the detection processing using the virtual data.

Description

Technical Field

[0001] This disclosure relates to vehicle technology. Background Technology

[0002] There is a technology that simultaneously senses the road environment using an onboard camera and identifies road areas in real time. Relatedly, for example, Patent Document 1 discloses a system for sensing malfunctions in onboard cameras.

[0003] Existing technical documents Patent documents Patent Document 1: Japanese Patent Application Publication No. 2001-211466 Summary of the Invention

[0004] The problem that the invention aims to solve The purpose of this disclosure is to suppress the effects of camera malfunctions in image-based target detection systems.

[0005] Solution for solving the problem One aspect of this disclosure is an information processing apparatus having a control unit that performs the following actions: acquiring multiple images obtained by multiple cameras mounted on a mobile body; performing detection processing to detect predetermined targets around the mobile body based on the multiple images; and determining whether an anomaly has occurred for each of the multiple cameras, wherein if the control unit determines that at least one of the multiple cameras has an anomaly, it generates virtual data corresponding to the camera that has been determined to be abnormal, and uses the virtual data to perform the detection processing.

[0006] One aspect of this disclosure is an information processing method executed by an information processing device, comprising: acquiring multiple images obtained by multiple cameras mounted on a mobile body; performing detection processing to detect predetermined targets around the mobile body based on the multiple images; determining whether an anomaly has occurred for each of the multiple cameras; and, if an anomaly is determined to have occurred in at least one of the multiple cameras, generating virtual data corresponding to the camera that was determined to have an anomaly, and using the virtual data to perform the detection processing.

[0007] In addition, as another option, one could provide a program for causing a computer to perform the aforementioned information processing method, or a computer-readable storage medium that non-temporarily stores the program.

[0008] Invention Effects According to this disclosure, in an image-based target detection system, the effects of camera malfunctions can be suppressed. Attached Figure Description

[0009] Figure 1 This diagram illustrates the method of autonomous driving performed by vehicle 1.

[0010] Figure 2 This is a diagram used to illustrate the subject matter of this disclosure.

[0011] Figure 3 This diagram illustrates the configuration of vehicle 1 and onboard device 10.

[0012] Figure 4 This is a diagram illustrating the outline of the presumed model of the first embodiment.

[0013] Figure 5 This is a diagram illustrating the data flow of the first embodiment.

[0014] Figure 6 This is a flowchart of the process performed by the vehicle-mounted device 10 in the first embodiment.

[0015] Figure 7 This is a diagram illustrating the outline of the presumed model of the second embodiment.

[0016] Figure 8 This is a diagram illustrating the data flow of the second embodiment.

[0017] Figure 9 This is a flowchart of the process performed by the vehicle-mounted device 10 in the second embodiment.

[0018] Explanation of reference numerals in the attached figures: 1: Vehicle; 10: Vehicle-mounted devices; 11: Control Department; 12: Storage Department; 13: Ministry of Communications; 14: Input / output section. Detailed Implementation

[0019] In recent years, research has been conducted on autonomous driving systems that enable vehicles to drive autonomously along pre-set routes. In autonomous driving, vehicles determine their position and posture, for example, by comparing pre-stored road maps with results obtained from sensing the road environment.

[0020] For example, autonomous vehicles identify the position of a designated target (typically a lane boundary line) on the road based on images acquired by multiple onboard cameras, and infer the vehicle's position and generate a predetermined trajectory based on the identification results.

[0021] In such a system, it is preferable to implement fault prevention measures.

[0022] For example, if one of the multiple vehicle cameras malfunctions and outputs an incorrect image, lane boundary lines and other features might be identified based on that erroneous image. This could result in the generation of a problematic predetermined trajectory. However, immediately stopping the autonomous vehicle's driving system upon detecting a camera malfunction is not the safest approach.

[0023] To address this issue, it is preferable to continue processing for autonomous driving while minimizing the impact of a malfunctioning camera on the system. The information processing apparatus disclosed herein solves this problem.

[0024] One information processing apparatus disclosed herein includes a control unit that performs the following actions: acquiring multiple images obtained by multiple cameras mounted on a mobile body; performing detection processing to detect predetermined targets around the mobile body based on the multiple images; and determining whether an anomaly has occurred for each of the multiple cameras. If the control unit determines that at least one of the multiple cameras has an anomaly, it generates virtual data corresponding to the camera that has been determined to be abnormal and uses the virtual data to perform the detection processing.

[0025] The information processing apparatus disclosed herein is typically a device mounted on a mobile body (such as a car), but it can also be a server device capable of communicating with the mobile body.

[0026] The control unit performs processing to detect defined targets around the moving vehicle based on images acquired by multiple cameras. Typical targets include lane boundaries, but are not limited to these. The results of target detection can also be used to identify areas (road areas) suitable for the movement of the moving vehicle.

[0027] Furthermore, the control unit determines that at least one of the multiple cameras has malfunctioned. Malfunction determination can be performed using any method. For example, the control unit can determine that a camera has malfunctioned if it outputs an image containing specified features. An image containing specified features can be, for example, an image whose statistical values ​​(mean, variance, deviation, etc.) of brightness, chroma, hue, etc., meet a specified threshold. Alternatively, an image containing specified features can also refer to an image where no edges with an intensity greater than or equal to a specified value are detected.

[0028] Furthermore, anomaly detection can also be performed using methods other than image processing. For example, the control unit can determine that a camera has malfunctioned if it outputs a signal containing a specified characteristic. For instance, if a camera outputs a signal that would not be generated during normal operation, it can be presumed that the camera has malfunctioned. It should be noted that the signal can refer to an electrical signal or a data string derived from an electrical signal.

[0029] If the control unit determines that at least one of the multiple cameras has malfunctioned, it generates virtual data corresponding to the camera that has been identified as malfunctioning, and uses the virtual data to perform target detection processing.

[0030] For example, when performing target detection based on features obtained by transforming an image, the control unit can also generate virtual features for the image output by the camera that has been identified as abnormal, using these virtual features as virtual data. Virtual features can be, for example, data such as sets of null values ​​that do not contribute to target detection. This allows target detection to continue while suppressing the influence of images generated by cameras that have malfunctioned.

[0031] It should be noted that, when using a machine learning model for detection processing, other methods can also be used, wherein the machine learning model is a model that has been pre-learned by using multiple learning images to detect the specified target from the image.

[0032] For example, when using a machine learning model that has been trained to ensure that the contribution rate of a virtual image in object detection is less than or equal to a specified value, the image output by a camera suspected of being anomalous can be replaced with a virtual image. This method allows object detection to continue while suppressing the influence of anomalous cameras.

[0033] Hereinafter, embodiments of the present disclosure will be described based on the accompanying drawings. The configurations of the following embodiments are illustrative, and the present disclosure is not limited to the configurations of the embodiments.

[0034] (First Implementation) Reference Figure 1 An overview of the vehicle system according to the first embodiment will be described. The vehicle system of this embodiment is configured to include an on-board device 10 mounted on a vehicle 1 and a camera 20 mounted on a vehicle 1.

[0035] It should be noted that in this embodiment, vehicle 1 includes multiple vehicle-mounted cameras. It should also be noted that in the following description, the term "camera 20" will be used to refer to each of these multiple vehicle-mounted cameras.

[0036] The on-board unit 10 mounted on vehicle 1 identifies the road area surrounding the vehicle based on images captured by multiple cameras 20, and uses the identification results to perform control that enables vehicle 1 to drive autonomously. The on-board unit 10 stores data related to the route to the destination (route data), and drives along the identified road area according to the route data.

[0037] A road area refers to an area where a vehicle 1 can travel. A road area can be identified, for example, based on the detection of lines that divide lanes (lane boundary lines). However, the object of detection is not limited to lane boundary lines. In this embodiment, lane boundary lines are shown as an example of the object of detection; however, road edges, lane center lines, road center lines, road signs, etc., can also be objects of identification.

[0038] exist Figure 1 In the diagram, solid lines represent lane boundary lines, and dashed lines represent road areas. The onboard device 10 identifies road areas based on images acquired by multiple cameras 20 and generates a predetermined trajectory for the vehicle by traveling along the road areas.

[0039] Figure 2 This diagram illustrates the problem to be solved by the vehicle system of the first embodiment. Here, it is assumed that vehicle 1 has three cameras: camera 20A located at the front center of the vehicle, camera 20B located at the front right of the vehicle, and camera 20C located at the front left of the vehicle. Figure 2 (A) is entering Figure 1 An example of images captured by the cameras of vehicle 1 immediately before the curve shown. Figure 2 As shown in (A), under normal circumstances, each camera outputs an image captured from a different location for the same object.

[0040] Figure 2 (B) means Figure 2 The diagram in (A) illustrates an example of a malfunction in one of the cameras (camera 20C). In this example, assume that camera 20C malfunctions and no longer outputs a normal image. Figure 2 In the example shown in (B), camera 20C outputs an abnormal image with longitudinal edge interference. In this case, when identifying lane boundary lines based on these three images, the vehicle-mounted device 10 cannot accurately identify the location of the road area. For example, if the curvature of the road is identified as lower than the actual curvature due to the influence of the image output by camera 20C, vehicle 1 may be forced to leave the road area.

[0041] To address this issue, the vehicle-mounted device 10 of this embodiment determines whether any abnormalities have occurred among the multiple cameras present in the vehicle, and performs prescribed processing based on the determination results, so as to prevent the output of the cameras that have malfunctioned from affecting autonomous driving.

[0042] [Device Composition] Figure 3 This is a diagram illustrating an example of the configuration of vehicle 1. Vehicle 1 is configured to have an onboard device 10 and multiple cameras 20.

[0043] The vehicle-mounted device 10 can be configured as a computer with a processor (CPU (Central Processing Unit), GPU (Graphics Processing Unit), etc.), main storage (RAM (Random Access Memory), ROM (Read Only Memory), etc.), and auxiliary storage (EPROM (Erasable Programmable Read Only Memory), hard disk drive, removable media, etc.). The auxiliary storage device stores an operating system (OS), various programs, various tables, etc., and can achieve the various functions (software modules) as described below for the specified purpose by executing the programs stored in the auxiliary storage device. However, some or all of the functions can also be implemented as hardware modules, for example, through hardware circuits such as ASIC (Application Specific Integrated Circuit) and FPGA (Field-Programmable Gate Array).

[0044] The vehicle-mounted device 10 is configured to include a control unit 11, a storage unit 12, a communication unit 13, and an input / output unit 14.

[0045] The control unit 11 is a computing component that executes prescribed programs to implement various functions of the vehicle-mounted device 10. The control unit 11 can be implemented, for example, by a hardware processor such as a CPU. Furthermore, the control unit 11 may also be configured to include RAM, ROM (Read Only Memory), cache memory, etc.

[0046] The control unit 11 is configured to have three software modules: an identification unit 111, an anomaly determination unit 112, and a driving control unit 113. Each software module can be implemented by the control unit 11 (CPU, etc.) executing a program stored in the storage unit 12, which will be described later.

[0047] The recognition unit 111 acquires images from multiple cameras 20 and identifies road areas based on the acquired images. In this embodiment, the recognition unit 111 converts the acquired images into feature quantities and inputs the obtained feature quantities into a machine learning model stored in the storage unit 12. The machine learning model is a model specifically designed for estimating the position of lane boundary lines (hereinafter referred to as the estimation model). The recognition unit 111 generates data representing the position information of lane boundary lines (hereinafter referred to as road area data) based on the output of the estimation model. The road area data is sent to the driving control unit 113.

[0048] Furthermore, the identification unit 111 obtains information related to camera anomalies from the anomaly determination unit 112 (described later), and performs prescribed processing to suppress the impact of the malfunctioning camera if any one of the multiple cameras 20 malfunctions. Details of the processing will be described later.

[0049] Furthermore, the identification unit 111 identifies targets (e.g., other vehicles, pedestrians, traffic lights, traffic signs, etc.) located outside the lane boundary lines surrounding the vehicle, for example, based on images captured by the camera 20. The identification results are sent to the driving control unit 113 in the same manner as the road area identification results for autonomous driving.

[0050] The anomaly determination unit 112 determines that any one of the multiple cameras 20 has an anomaly by monitoring the output of the multiple cameras 20, and notifies the identification unit 111. The anomaly determination unit 112 determines that any one of the multiple cameras 20 has an anomaly by any of the following. In the following description, the camera that is determined to have an anomaly is referred to as the "abnormal camera".

[0051] (1) Image-based determination The anomaly determination unit 112 can determine the presence of an abnormal camera by analyzing the images (hereinafter referred to as camera images) output by the multiple cameras 20 respectively. For example, the anomaly determination unit 112 acquires statistical values ​​(e.g., average values) related to the brightness, chroma, and hue of the target camera image, and determines that the corresponding camera is abnormal if the acquired values ​​meet predetermined conditions. For example, if the average brightness is too high (or too low) or the hue is deviated, it is presumed that the camera is abnormal.

[0052] Alternatively, the anomaly determination unit 112 may perform prescribed image processing on the camera image of the object and determine that the corresponding camera has an anomaly based on the result. For example, the anomaly determination unit 112 may perform edge detection processing from the camera image of the object, and if no edge with an intensity greater than or equal to a predetermined value is detected, it may determine that the corresponding camera has an anomaly (failed to accurately capture the lane boundary line).

[0053] (2) Signal-based determination Furthermore, the anomaly determination unit 112 can determine anomalies, for example, by analyzing the signals output by the multiple cameras 20. For instance, the anomaly determination unit 112 can determine that an anomaly has occurred in the corresponding camera if the target camera outputs electrical signals or data strings that do not conform to the prescribed format or sequence.

[0054] The driving control unit 113 controls the autonomous driving of the vehicle based on the road area identification results performed by the identification unit 111. For example, the driving control unit 113 uses the road area identification results and obstacle detection results generated by the identification unit 111 to control the autonomous driving of the vehicle. For instance, the driving control unit 113 drives along the route data and ensures that obstacles do not enter a predetermined safe area centered on the vehicle. Known methods can be used to enable the vehicle to drive autonomously. It should be noted that the route data refers to data used for route guidance, typically including information on left and right turns leading to the destination, and information on interchanges to be used.

[0055] Storage unit 12 is a unit for storing information and is composed of storage media such as RAM, disk, and flash memory. Storage unit 12 stores programs executed by control unit 11, data used by those programs, etc. Furthermore, storage unit 12 stores the aforementioned route data and estimation models.

[0056] An inference model is a machine learning model that takes features obtained from multiple camera images as input and outputs the position information of lane boundary lines.

[0057] Figure 4 (A) is a diagram illustrating the learning method of the inference model. For example... Figure 4As shown in (A), the estimation model is obtained by using feature quantities obtained by transforming a learning image (e.g., an image taken by a vehicle traveling on an actual road) as input data, and using the positional information of lane boundary lines contained in the learning image (e.g., accurate positional information of lane boundary lines obtained from a high-resolution map of the road) as correct answer data. It should be noted that the accurate positional information of the lane boundary lines can be represented, for example, by coordinates in a three-dimensional coordinate system (vehicle coordinate system) with vehicle 1 as the reference.

[0058] During the learning process, the comparison between the inference results obtained through the inference model and the correct answer data is fed back to the inference model, and the parameters are updated in a way that the difference between the inference results and the correct answer data decreases.

[0059] It should be noted that in this example, after converting the images acquired by multiple cameras 20 into feature values, these feature values ​​are integrated, and the integrated feature values ​​are input into the estimation model. This is because the multiple cameras 20 are installed at different locations on the vehicle 1, thus requiring processing to eliminate parallax.

[0060] Figure 4 (B) is a diagram illustrating the method of making inferences using a learned inference model. For example... Figure 4 As shown in (B), the positional information of the lane boundary line can be obtained by inputting feature quantities based on multiple images acquired by the vehicle-mounted camera into the estimation model. During the estimation phase, the feature quantities are integrated in the same way as during the learning phase.

[0061] The communication unit 13 is a wireless communication interface for connecting the vehicle-mounted device 10 to the in-vehicle network.

[0062] The input / output unit 14 is a component that receives input operations from the occupants of the vehicle and presents information to them. In this embodiment, the input / output unit 14 is composed of a touch panel display. That is, it consists of a liquid crystal display and its control unit, a touch panel and its control unit.

[0063] It should be noted that, regarding the specific hardware configuration of the vehicle-mounted device 10, constituent elements can be appropriately omitted, substituted, or added depending on the implementation method. For example, the control unit 11 may include multiple hardware processors. The hardware processors may be composed of microprocessors, FPGAs, GPUs, etc. In addition, input / output devices other than those shown in the examples (such as optical drivers) may also be added. Furthermore, the vehicle-mounted device 10 may be composed of multiple computers. In this case, the hardware configuration of each computer may be the same or different.

[0064] Camera 20 refers to multiple cameras installed on vehicle 1. The multiple cameras 20 are installed at different locations on vehicle 1, for example, in a manner that allows them to capture the entire surroundings of vehicle 1.

[0065] [Details of the processing performed by the onboard device] Next, details of the processing performed by the identification unit 111 and the anomaly determination unit 112 will be explained.

[0066] Figure 5 This is a diagram showing the data flow during the process (estimation phase) in which the vehicle-mounted device 10 identifies the road area.

[0067] First, the anomaly detection unit 112 acquires camera images output from multiple cameras 20, and determines the presence or absence of anomalies for each camera using the method described above. The detection result is sent to the recognition unit 111. Figure 5 In the example shown, assume that camera image A is output from camera 20A, camera image B is output from camera 20B, and camera image C is output from camera 20C. Furthermore, assume that an anomaly occurs in camera 20C, and consequently, an anomaly is also found in camera image C. In this case, the anomaly determination unit 112 sends a determination result that an anomaly has occurred in camera 20C to the recognition unit 111.

[0068] Multiple camera images are input into the recognition unit 111.

[0069] The recognition unit 111 performs prescribed processing on the acquired camera images, converting the images into feature quantities. In this embodiment, a feature quantity refers to a set of two-dimensional coordinates of pixels (e.g., white pixels) that are presumed to represent lane boundary lines. These coordinates are presented in a two-dimensional coordinate system (camera coordinate system) based on each camera.

[0070] Furthermore, the recognition unit 111 integrates the feature quantities corresponding to each camera. Feature quantity integration can be performed, for example, by projecting the feature quantities (two-dimensional coordinates) corresponding to each camera onto a common three-dimensional space. The space of the projection destination is a space within a three-dimensional coordinate system (vehicle coordinate system) based on vehicle 1. This space is called the feature quantity space. The integrated feature quantities become a set of three-dimensional coordinates representing the location of the lane boundary line in the feature quantity space.

[0071] In addition, the recognition unit 111 inputs the integrated feature quantity (a set of three-dimensional coordinates in the feature quantity space) into the estimation model, which is used as a machine learning model for recognizing road areas.

[0072] The estimation model takes a set of three-dimensional coordinates in the feature space as input and outputs the estimated position information of the lane boundary lines in the vehicle coordinate system. The data containing the position information of the estimated lane boundary lines is called "road area data".

[0073] The identification unit 111 sends road area data output from the estimation model to the driving control unit 113, and the driving control unit 113 uses the data to control the autonomous driving of the vehicle 1.

[0074] It should be noted that when the identification unit 111 receives a notification from the anomaly determination unit 112 that an abnormal camera exists, it replaces the feature quantity corresponding to the image acquired by the abnormal camera with a virtual feature quantity. A virtual feature quantity is a feature quantity that does not affect the processing performed by the estimation model. For example, a virtual feature quantity can be a set of null values ​​or other feature quantities that are not projected into the feature quantity space. By generating such a feature quantity, the influence of the image output by the abnormal camera can be eliminated in the estimation processing performed by the estimation model.

[0075] [Processing Flowchart] Next, the process of the processing performed by the vehicle-mounted device 10 will be explained. Figure 6 This is a flowchart of the process performed by the on-board unit 10. Figure 6 The process shown is performed periodically during the driving of vehicle 1.

[0076] First, in step S11, the anomaly determination unit 112 acquires images from multiple cameras 20.

[0077] Next, in step S12, the anomaly determination unit 112 performs image analysis to determine whether a camera exhibiting an anomaly exists. In this step, the anomaly determination unit 112 analyzes each of the multiple images from a predetermined viewpoint to determine whether a camera outputting an image that is normally unavailable. For example, if the range or average value of brightness, chroma, hue, etc., deviates from the normal range, it can be determined that the corresponding camera has exhibited an anomaly. The analysis result is then sent to the recognition unit 111. Furthermore, the anomaly determination unit 112 also sends multiple images to the recognition unit 111.

[0078] It should be noted that, in addition to the image analysis method described above, the presence or absence of camera anomalies can also be determined by analyzing the signal output by camera 20. In this case, the anomaly determination unit 112 can send the results obtained from analyzing electrical signals, data strings, etc., to the recognition unit 111.

[0079] Next, in step S13, the recognition unit 111 converts the multiple images into feature quantities. In this step, for example, pixels that are presumed to represent lane boundary lines can be extracted from each image, and the set of coordinates of these pixels can be used as feature quantities.

[0080] Next, in step S14, the identification unit 111 determines the presence or absence of an abnormal camera (abnormal camera) based on the parsing result received from the anomaly determination unit 112. If an abnormal camera exists, the feature value corresponding to the image output by the abnormal camera is replaced with a virtual feature value (step S15). If no abnormal camera exists, the process proceeds to step S16.

[0081] In step S16, the recognition unit 111 projects the generated feature quantities (including virtual feature quantities) onto a predetermined feature quantity space. The feature quantity space is a three-dimensional space with a coordinate system based on the vehicle. This step projects the positions of the lane boundary lines captured by each camera onto the feature quantity space. It should be noted that, at this time, the virtual feature quantities do not affect the projection result.

[0082] Next, in step S17, the recognition unit 111 estimates the position of the lane boundary line based on the feature quantities projected onto the feature quantity space, and identifies the road region. In this step, the recognition unit 111 inputs a set of three-dimensional coordinates representing the position of the lane boundary line in the feature quantity space into the estimation model, and obtains the estimated position information of the lane boundary line as output. Furthermore, the recognition unit 111 generates road region data containing the obtained position information.

[0083] The generated road area data is sent from the identification unit 111 to the driving control unit 113 for use in the autonomous driving control of the vehicle 1.

[0084] As explained above, the vehicle-mounted device 10 of the first embodiment determines that any one of the multiple cameras 20 has malfunctioned based on images acquired by the multiple cameras 20 mounted on the vehicle. Furthermore, it performs lane boundary line recognition processing by replacing the image output by the camera determined to have malfunctioned with virtual feature values.

[0085] Based on this configuration, the influence of images output by abnormal cameras can be eliminated in the lane boundary line recognition process. Therefore, even in the event of a camera malfunction, safe and continued vehicle operation can be ensured.

[0086] It should be noted that, for safety reasons, in the event of a camera malfunction, it is permissible to isolate only the affected camera from the system and continue operation. However, if the malfunctioning camera is isolated from the system, data can no longer be obtained from it; therefore, fault diagnosis is impossible unless the camera is stored at the factory.

[0087] On the other hand, in this embodiment, images from the malfunctioning camera continue to be acquired, and replacement with virtual feature quantities is performed. Therefore, images output by the malfunctioning camera can be accumulated, or images output by the malfunctioning camera can be sent to the diagnostic device in real time via the network. That is, rapid fault diagnosis can be expected.

[0088] (Second Implementation) In the first embodiment, the influence of the image output by the abnormal camera is eliminated by replacing it with a virtual feature value. In contrast, the second embodiment replaces the image output by the abnormal camera itself with a virtual image.

[0089] In the first embodiment, the feature values ​​obtained by converting the image are replaced with virtual ones and input into the estimation model. However, if an input image (virtual image) that does not affect the output is defined in advance and the estimation model learns the features of the virtual image, the same effect as the first embodiment can be obtained by replacing the input image itself.

[0090] Figure 7 This diagram illustrates the learning phase of the estimation model in the second embodiment. In this embodiment, the estimation model is learned by replacing a portion of the learning image with a virtual image. The virtual image may be, for example, a fully filled image or an image containing a prescribed pattern. The virtual image is independent of the position of the lane boundary lines; therefore, by learning from a dataset containing virtual images, the contribution of the virtual images to the output approaches zero as the learning progresses.

[0091] It should be noted that this example illustrates how a portion of the input data is replaced with a virtual image in a typical learning process. However, it's also possible to use only the virtual image as input and perform learning using the correct answer data indicating "no lane boundary lines exist within the image." Thus, it can be learned that "virtual images do not contribute to the recognition of lane boundary lines."

[0092] In the second embodiment, the model is learned by ensuring that the contribution rate of the virtual image to the output is less than or equal to a predetermined value. It should be noted that the virtual image can also be an image that does not affect the projection results of the feature quantities.

[0093] Figure 8This is a diagram showing the data flow during the process (estimation phase) in which the vehicle-mounted device 10 identifies the road area in the second embodiment.

[0094] In the second embodiment, the recognition unit 111, which receives the determination result from the anomaly determination unit 112, replaces the image output from the abnormal camera with a virtual image. The virtual image is converted into feature values ​​in the same way as the normal camera image and becomes the object of integration.

[0095] However, in this embodiment, although the features based on the virtual image are integrated, they do not affect the estimation results obtained through the estimation model. Therefore, even in the case of an abnormal camera, the influence of the image output by the abnormal camera can be eliminated in the same way as in the first embodiment.

[0096] Figure 9 This is a flowchart of the process performed by the vehicle-mounted device 10 in the second embodiment. The steps shown by the dashed lines refer to the same processes as in the first embodiment.

[0097] In this embodiment, after image parsing in step S12, before converting the image into feature quantities, a step of determining the presence or absence of an abnormal camera is performed (step S14). If an abnormal camera is present, the process transfers to step S21, where the image output by the abnormal camera is replaced with a virtual image, and then the conversion from image to feature quantities is performed (step S13). If no abnormal camera is present, the process transfers from step S14 to step S13, and processing continues.

[0098] When step S13 is completed, the process moves to step S16 and performs the same process as in the first embodiment.

[0099] In this way, the same effect as the first implementation can be achieved by replacing the image output by the abnormal camera with a virtual image.

[0100] (Modified example) The above-described implementation is merely an example, and this disclosure may be implemented with appropriate modifications without departing from its spirit.

[0101] For example, as long as there is no technical contradiction, the processing and units described in this disclosure can be freely combined to implement the process.

[0102] Furthermore, while the description of the implementation provides an example of the vehicle-mounted device 10 identifying road areas, road area identification can also be performed by a server device located in a different location from the vehicle 1. In this case, camera images can be sent from the vehicle-mounted device 10 to the server device, and the results of road area identification (road area data) can be sent from the server device to the vehicle-mounted device.

[0103] Furthermore, while the description of the implementation shows a specific example of the vehicle-mounted device 10 recognizing lane boundary lines, any target on the road (traffic lights, traffic signs, etc.) or a target moving on the road (other vehicles, etc.) can also be used as the recognition target. When performing the process of recognizing targets outside the lane boundary lines, the virtual data described in the implementation can also be utilized.

[0104] Furthermore, a process described as being performed by one device can also be executed by multiple devices. Alternatively, a process described as being performed by different devices can also be executed by one device. In a computer system, the hardware configuration (server configuration) used to implement each function can be flexibly changed.

[0105] This disclosure can also be implemented by supplying a computer program that implements the functions described in the above embodiments to a computer, which then reads and executes the program using one or more processors. Such a computer program can be provided to the computer via a non-transitory computer-readable storage medium that can be connected to the computer's system bus, or via a network. Non-transitory computer-readable storage media include, for example, any type of optical disk (such as a floppy disk, hard disk drive, etc.), optical disk (such as CD-ROM, DVD / Blu-ray disc, etc.), read-only memory (ROM), random access memory (RAM), EPROM, EEPROM (Electrically Erasable Programmable Read Only Memory), magnetic cards, flash memory, optical cards, and any type of medium suitable for storing electronic instructions.

Claims

1. An information processing apparatus comprising a control unit, the control unit performing the following operations: Acquire multiple images obtained by multiple cameras mounted on a moving body; A detection process is performed to detect specified targets around the moving object based on the multiple images; as well as For each of the multiple cameras, determine whether an anomaly has occurred. When the control unit determines that at least one of the plurality of cameras has malfunctioned, it generates virtual data corresponding to the camera that has been determined to be malfunctioning, and uses the virtual data to perform the detection process.

2. The information processing apparatus according to claim 1, wherein, If a camera is found to be outputting an image or signal containing specified characteristics, the control unit determines that the camera has malfunctioned.

3. The information processing apparatus according to claim 1, wherein, The detection process is based on the feature values ​​obtained by transforming each of the plurality of images.

4. The information processing apparatus according to claim 3, wherein, The control unit generates virtual feature quantities for the images output by the cameras that are determined to be abnormal, and uses these virtual data as the virtual data.

5. The information processing apparatus according to claim 4, wherein, The virtual feature quantity is the feature quantity that does not contribute to the detection of the specified target.

6. The information processing apparatus according to claim 1, wherein, The detection process is performed using a machine learning model, which is a model that has been pre-learned using multiple learning images to detect the specified target from the images.

7. The information processing apparatus according to claim 6, wherein, The machine learning model is a model that learns by ensuring that the contribution rate of a specified virtual image in object detection is less than or equal to a specified value.

8. The information processing apparatus according to claim 7, wherein, The control unit replaces the image output by the camera that was determined to be abnormal with the virtual image.

9. An information processing method, which is executed by an information processing device. The information processing method includes: Acquire multiple images obtained by multiple cameras mounted on a moving body; A detection process is performed to detect specified targets around the moving object based on the multiple images; For each of the plurality of cameras, determine whether an anomaly has occurred; as well as If it is determined that at least one of the plurality of cameras has malfunctioned, virtual data corresponding to the camera that was determined to be malfunctioning is generated, and the detection process is performed using the virtual data.

10. The information processing method according to claim 9, wherein, If a camera is found to be outputting an image or signal containing the specified characteristics, it is determined that the camera has malfunctioned.

11. The information processing method according to claim 9, wherein, The detection process is based on the feature values ​​obtained by transforming each of the plurality of images.

12. The information processing method according to claim 11, wherein, Virtual feature quantities are generated for the images output by the cameras that are identified as abnormal, and these are used as the virtual data.

13. The information processing method according to claim 12, wherein, The virtual feature quantity is the feature quantity that does not contribute to the detection of the specified target.

14. The information processing method according to claim 9, wherein, The detection process is performed using a machine learning model, which is a model that has been pre-learned using multiple learning images to detect the specified target from the images.

15. The information processing method according to claim 14, wherein, The machine learning model is a model that learns by ensuring that the contribution rate of a specified virtual image in object detection is less than or equal to a specified value.

16. The information processing method according to claim 15, wherein, The image output by the camera that was determined to be abnormal is replaced with the virtual image.

17. A non-transitory storage medium having a program recorded thereon, the program being used to cause a computer to perform the information processing method as described in claim 9.

Images