Information processing method, program, information processing device, and camera unit

Abstract

An information processing method, a program, an information processing device, and a camera unit are provided that suitably recognize an object. [Solution] The information processing method acquires an image captured by a camera 11 mounted on a vehicle C, recognizes the object based on the output when a reduced image is input into a learning model trained to output a recognition result of an object contained in the image when the image is input, acquires an extracted image by extracting a partial area of ​​the image, and causes an information processing device 2 to execute a process of recognizing the object based on the output when the acquired extracted image is input into the learning model.

Description

[Technical Field]

[0001] The present invention relates to an information processing method, a program, an information processing device, and a camera unit. [Background technology]

[0002] There are technologies for detecting objects in images. For example, Patent Document 1 discloses an image processing system that can perform image analysis of a predetermined area in a moving image with higher accuracy while reducing the amount of data to be processed. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-211760 Summary of the Invention [Problem to be solved by the invention]

[0004] In one aspect, an object is to provide an information processing method and the like that can suitably recognize an object. [Means for solving the problem]

[0005] An information processing method according to one aspect of the present disclosure acquires an image captured by a camera mounted on a vehicle, recognizes the object based on the output when a reduced version of the image is input into a learning model trained to output a recognition result of an object contained in the image when the image is input, acquires an extracted image by extracting a partial area of ​​the image, and causes a computer to execute a process of recognizing the object based on the output when the acquired extracted image is input into the learning model. [Effects of the Invention]

[0006] According to one aspect of the present disclosure, it is possible to provide an information processing method or the like that appropriately recognizes an object. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a schematic diagram illustrating the configuration of a camera unit including an information processing device (camera ECU) according to a first embodiment. [Figure 2] FIG. 2 is a block diagram illustrating a physical configuration of an information processing apparatus. [Figure 3] FIG. 1 is an explanatory diagram regarding a generation process of a learning model (object detection model). [Figure 4] 10 is a flowchart illustrating a process of a control unit of the information processing device. [Figure 5] FIG. 10 is an explanatory diagram showing a recognition result in a reduced image. [Figure 6] FIG. 10 is an explanatory diagram showing a recognition result in an extracted image. [Figure 7] 10 is a flowchart illustrating processing by a control unit of an information processing device according to a second embodiment (reliability score). [Figure 8] 10 is a flowchart illustrating the processing of a control unit of an information processing device according to a third embodiment (merging point). [Figure 9] FIG. 10 is an explanatory diagram showing a process of changing (shifting to the right) an extraction area according to a junction point. [Figure 10] FIG. 10 is an explanatory diagram showing a process of changing (shifting to the left) an extraction area according to a junction point. [Figure 11] 10 is a flowchart illustrating the processing of a control unit of an information processing device according to a fourth embodiment (extracted image). DETAILED DESCRIPTION OF THE INVENTION

[0008] (Embodiment 1) Hereinafter, embodiments will be described with reference to the drawings. FIG. 1 is a schematic diagram illustrating the configuration of a camera unit 1 including an information processing device 2 (camera ECU) according to the first embodiment. FIG. 2 is a block diagram illustrating the physical configuration of the information processing device 2. The camera unit 1 includes a camera 11 mounted on a vehicle C, a display unit 12 (an example of an output device), and the information processing device 2. The information processing device 2 performs image processing on an image captured by the camera 11 and outputs the image, including the image processing results, to the display unit 12, thereby displaying the image on the display unit 12. Such a camera unit 1 may function as, for example, a door mirror camera device that replaces a conventional door mirror.

[0009] The camera 11 is, for example, a CMOS camera, and includes door mirror cameras 11 arranged on the left and right outside the front doors of the vehicle C, or a rearview camera 11 arranged on the rear window. FIGS. 1 and 2 show an example in which one camera 11 is arranged on each side. However, any number of cameras 11 may be used in this embodiment as long as there is at least one. The camera 11 is communicably connected to the information processing device 2 via, for example, a serial cable, and outputs captured images such as video to the information processing device 2 periodically or in real time. Note that the term "real time" in the various processes in this specification does not mean strictly immediate processing or simultaneous processing, but means that processing is performed as quickly as possible.

[0010] The display unit 12 is configured with, for example, a liquid crystal display or the like, and includes a door mirror display arranged inside a pillar inside the vehicle, or a display for car navigation. Alternatively, an information terminal such as a smartphone wirelessly connected to the information processing device 2 via Bluetooth (registered trademark) or the like may function as the display unit 12. The display unit 12 is communicably connected to the information processing device 2 via, for example, a serial cable, and displays images output from the information processing device 2. FIGS. 1 and 2 show an example in which one display unit 12 is provided in each of the driver's seat and the passenger seat. However, the number of display units 12 according to this embodiment may be any number greater than one. For example, only one display unit 12 may be provided near the driver's seat.

[0011] The information processing device 2 includes a control unit 20, a storage unit 23, an input / output I / F 21, and a communication unit 22, and functions as a camera ECU that performs image processing on an image captured by the camera 11 and outputs the image together with the image processing results to the display unit 12. The control unit 20 is configured with a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), etc., and performs various control processes, arithmetic processes, etc. by reading and executing a control program P (program product) and data stored in advance in the storage unit 23.

[0012] The storage unit 23 is configured by a volatile memory element such as a RAM (Random Access Memory) or a non-volatile memory element such as a ROM (Read Only Memory), an EEPROM (Electrically Erasable Programmable ROM), or a flash memory. The storage unit 23 pre-stores a control program P and data that the control unit 20 references when processing various calculations. Furthermore, the storage unit 23 may also store an entity file of the learning model 201.

[0013] Alternatively, the learning model 201 may be implemented in an AI chip, and the control unit 20 may input images to the learning model 201 and obtain recognition results for objects from the learning model 201 by performing input / output processing on the AI ​​chip via an internal bus or the like.

[0014] The input / output I / F 21 is, for example, a communication interface for serial communication. The information processing device 2 is connected to the camera 11, the display unit 12, etc. via the input / output I / F 21 so as to be able to communicate with them.

[0015] The communication unit 22 is an input / output interface that uses a communication protocol such as CAN or Ethernet (registered trademark). The control unit 20 may communicate with other ECUs connected to the network 4 via the communication unit 22. For example, the information processing device 2 according to this embodiment may communicate with an ECU 3 via the network 4 as shown in FIGS. 1 and 2.

[0016] The ECU 3 is a control device that performs control related to the vehicle C. Typically, the ECU 3 is an on-board device. Like the information processing device 2, the ECU 3 includes a control unit (CPU), a storage unit, and a communication unit. A GPS device 31 is connected to the ECU 3, and the ECU 3 derives the current traveling location of the vehicle C (host vehicle) by periodically acquiring a GPS signal via the GPS device 31. The ECU 3 periodically transmits information related to the derived traveling location to the information processing device 2 via the network 4. Furthermore, map information is stored in the storage unit of the ECU 3, and the ECU 3 may also periodically transmit the map information to the information processing device 2 via the network 4. Note that in this embodiment, the network 4, the ECU 3, and the GPS device 31 are not essential components.

[0017] FIG. 3 is an explanatory diagram of a generation process of a learning model 201 (object detection model). The learning model 201 is implemented, for example, in an AI chip or the like. Alternatively, the learning model 201 may be configured as a software function unit controlled by the control unit 20. In this embodiment, the information processing device 2 generates the learning model 201. However, this is not limited to this, and the learning model 201 may be generated by a model generation device such as a server device. The control unit 20 of the information processing device 2 may train a pre-learning machine learning model (e.g., a neural network such as YOLO or R-CNN) using existing training data or training data prepared for the learning model 201, thereby generating a learning model that recognizes (detects) objects included in an input image when an image is input, and use this as the learning model 201. In this way, the learning model 201 functions as an object detection model.

[0018] The recognition result for an object output by the learning model 201 includes at least whether or not the object was recognized. If an object was recognized, the recognition result includes the area (bounding box) and type (class) of the object in the image. The recognition result may also include a confidence score (predicted probability of each class) of the recognized object.

[0019] The training data includes question data and answer data, and the image (scene image) acquired from the camera 11 corresponds to the question data, and the recognition result of the object corresponds to the answer data. The area and type of the object, which is the answer data, may be set by annotating (adding) it to the image. The data set of question data and answer data included in the training data for learning the learning model 201 and the data set of input data and output data when the learning model 201 is used are synonymous, and if it is defined in one data set, it naturally applies to the other data set as well.

[0020] The neural network (learning model 201) trained using training data is expected to be used as a program module that is part of artificial intelligence software. The learning model 201 is used in the information processing device 2 that includes the control unit 20 (CPU, etc.) and storage unit 23 as described above, and a neural network system is configured by being executed by the information processing device 2 that has such calculation processing capabilities. That is, the control unit 20 of the information processing device 2 performs calculations to extract feature quantities of an image input to an input layer in accordance with instructions from the learning model 201 stored in the storage unit 23, and outputs a recognition result for the object.

[0021] The learning model 201 is configured using, for example, YOLO or R-CNN (Region Convolutional Neural Network), and includes an input layer that accepts image input, a middle layer that extracts features of the image, and an output layer that outputs recognition results for the object. The input layer has multiple neurons that accept image input and passes the input values ​​to the middle layer. The middle layer is defined using an activation function such as a ReLU function or a sigmoid function, has multiple neurons that extract features of each input value, and passes the extracted features to the output layer. Parameters of the activation function, such as weighting coefficients and bias values, are optimized using backpropagation. The output layer is configured, for example, by a fully connected layer and outputs recognition results for the object based on the features output from the middle layer. The output layer may further include, for example, a softmax layer, which also outputs the probability (confidence score) of the object.

[0022] In this embodiment, the learning model 201 is R-CNN or the like, but is not limited thereto. The learning model 201 may be constructed using other machine learning algorithms, such as neural networks other than R-CNN, transformers, BERT, GPT, recurrent neural networks (RNNs), long-short term models (LSTMs), support vector machines (SVMs), Bayesian networks, linear regression, regression trees, multiple regression, random forests, and ensembles. Alternatively, the learning model 201 may be constructed using an artificial intelligence chatbot based on an LLM, such as ChatGPT. In this case, ChatGPT may be fine-tuned to efficiently output recognition results for the target object. Alternatively, a question (prompt) generated using an external database, such as a WebDB, may be input together with an image to a prompt input interface, which is the input interface of ChatGPT.

[0023] 4 is a flowchart illustrating the processing of the control unit 20 of the information processing device 2. The control unit 20 of the information processing device 2 performs the following processing in real time, for example, when the vehicle C is in an activated state.

[0024] The control unit 20 of the information processing device 2 acquires images captured by the camera 11 (S101). The vehicle C is equipped with a plurality of cameras 11, including cameras 11 that capture images of the rear of the vehicle C, such as cameras 11 installed outside the left and right front doors of the vehicle C, i.e., in locations where conventional door mirrors would be installed. The control unit 20 of the information processing device 2 periodically or in real time acquires images captured and output by the cameras 11. The images from the cameras 11 may be still images captured at a predetermined cycle, or may be video images captured at a predetermined frame rate. In this case, the video images include a plurality of frame groups (still image groups) corresponding to the frame rate, and the control unit 20 of the information processing device 2 may perform the following processing on each of the frames (still images) included in the frame groups.

[0025] The control unit 20 of the information processing device 2 reduces the acquired image (S102). The control unit 20 of the information processing device 2 reduces the image acquired from the camera 11, i.e., the original image, according to the number of pixels supported by the learning model 201 (i.e., input size). The "number of pixels supported" refers to the number of pixels of an input image at which the learning model 201 can output a meaningful recognition result, and / or the number of pixels of an image that can be input to the learning model 201, which is predetermined according to the number of pixels. The number of pixels supported by the learning model 201 may be predetermined according to the specifications and specifications of the learning model 201. The specific value of the number of pixels supported is not particularly limited, and may be, for example, 640px x 384px or 640px x 480px. Information related to the number of pixels supported, as well as various information or parameters used in the calculation process, are pre-stored in the storage unit 23 of the information processing device 2. In the present embodiment, as an example, the model size of the learning model 201 is 640px x 384px.

[0026] The control unit 20 of the information processing device 2 performs, for example, image compression processing on the image (original image) acquired from the camera 11 to reduce the number of pixels and generate a reduced image. The number of pixels in the reduced image is equal to the corresponding number of pixels in the learning model 201. If the number of pixels in the original image is 1920px x 1536px and the image is reduced according to the model size (640px x 384px) of the learning model 201, the reduction ratio is calculated by dividing the number of pixels in the reduced image by the number of pixels in the original image (640px / 1920px). In this way, the reduced image will have fewer pixels than the original image, which is the image before reduction, but will include all areas included in the original image.

[0027] The control unit 20 of the information processing device 2 inputs the reduced image to the learning model 201 (S103). The learning model 201 receives the reduced image and outputs a recognition result of the object.

[0028] 5 is an explanatory diagram showing the recognition result for a reduced image. Based on the input reduced image, the learning model 201 determines or estimates whether the reduced image contains an object such as a car, bicycle, or pedestrian. If an object is contained, the learning model 201 outputs the range (bounding box), type (class), and reliability score (predicted probability of each class) of the object as the recognition result. In this case, the learning model 201 may output the recognition result by superimposing it on the input reduced image.

[0029] The control unit 20 of the information processing device 2 acquires the recognition result of the object from the learning model 201 (S104). The control unit 20 of the information processing device 2 acquires the recognition result of the object from the learning model 201 by inputting a reduced image (reduced image) to the learning model 201. The recognition result of the object includes, for example, the area occupied by an object such as a car, a pedestrian, or a vehicle included in the reduced image, the type of the object, and a reliability score. If the learning model 201 recognizes multiple objects, the area and type (and reliability score) of each recognized object are output. If the learning model 201 does not recognize any objects, the area, etc. indicating the recognition result are not output.

[0030] The control unit 20 of the information processing device 2 determines whether the object has been recognized (S105). The control unit 20 of the information processing device 2 determines whether the object has been recognized according to the recognition result output from the learning model 201. For example, if the learning model 201 outputs information itself indicating "whether or not an object has been recognized," the control unit 20 may determine whether the object has been recognized according to the information. Also, for example, if the learning model 201 outputs the area, etc. of the object only when it has recognized the object, the control unit 20 determines that the object has been recognized (detected) if the area, etc. of the object is included in the recognition result, and determines that the object has not been recognized (detected) if the area, etc. is not included.

[0031] If an object is recognized (S105: YES), the control unit 20 of the information processing device 2 identifies the object closest to the camera 11 (i.e., to the vehicle C) (S1051). The method for identifying the closest object in S1051 is not particularly limited. For example, the control unit 20 of the information processing device 2 may identify the closest object based on the Y coordinate value in the image coordinate system of the object's region (bounding box) in the reduced image. For example, if the positive direction of the Y axis indicates the downward direction of the vehicle C, the object with the largest Y coordinate value of the bottom side of the bounding box may be identified as the closest object. Alternatively, if the positive direction of the Y axis indicates the upward direction of the vehicle C, the object with the smallest Y coordinate value of the bottom side of the bounding box may be identified as the closest object. Alternatively, the control unit 20 of the information processing device 2 may calculate the depth of each region of the recognized object using the function or imaging characteristics of the camera 11, and identify the object in the region with the smallest depth as the closest object. Furthermore, the control unit 20 may determine the proximity of an object based on the size of the area of ​​the recognized object and the class of the object. For example, images may be captured in advance while varying the distance between the camera 11 and an object (such as a car or a person) that can be recognized by the learning model 201, and a data table that associates the distance with the size (on the image) of the object at that distance may be created for each class, and the data table may be stored in the storage unit 23. Then, in the process of S105, the data table corresponding to the class of each object may be referenced, and the distance corresponding to the size of the area of ​​each recognized object may be considered to be the distance between the camera 11 (i.e., vehicle C) and the object. The object with the closest distance may then be determined to be the closest object.

[0032] If the object is not recognized (S105: NO), the control unit 20 of the information processing device 2 acquires an extracted image by extracting a partial area from the acquired image (S106). If the object is not recognized, the control unit 20 of the information processing device 2 may generate and acquire the extracted image by extracting a predetermined area from the acquired image, i.e., the original image.

[0033] The predetermined area, i.e., the extraction area, may be set to include the farthest view in the original image. Alternatively, the extraction area may be set to include the distant view of the adjacent lane (or road shoulder) when the vehicle C is traveling in a straight line. In this embodiment, the extraction area is determined in advance according to at least one of the position and angle of the camera 11 on the vehicle C that captures the original image, and the angle of view of the camera 11. Therefore, for example, when left and right cameras 11 are installed in a location where conventional door mirrors are installed, the position of the extraction area in the image captured by the left camera 11 (original image) (e.g., the coordinate position of the extraction image in the original image) and the position of the extraction area in the image captured by the right camera 11 (original image) are determined in advance.

[0034] The control unit 20 of the information processing device 2 may extract an extracted image so that the number of pixels is equal to or less than the corresponding number of pixels of the learning model 201. In this case, it is more desirable that the control unit 20 extract an extracted image so that the number of pixels of the extracted image is equal to the corresponding number of pixels. Alternatively, the control unit 20 may extract an extracted image having a size exceeding the corresponding number of pixels, and then reduce the extracted image to match the corresponding number of pixels of the learning model 201, and input the reduced extracted image (reduced extracted image) to the learning model 201. In other words, when inputting an extracted image to the learning model 201, the control unit 20 of the information processing device 2 may not require reduction processing as performed on the original image, or, even if reduction processing is performed, may generate the extracted image so that the reduction rate is greater than the reduction rate for the original image.

[0035] The control unit 20 of the information processing device 2 inputs the extracted image to the learning model 201 (S107). The learning model 201 receives the extracted image and outputs a recognition result of the object.

[0036] FIG. 6 is an explanatory diagram showing the recognition result for an extracted image. Based on the input extracted image, the learning model 201 determines or estimates whether the extracted image contains an object such as a car, bicycle, or pedestrian. If an object is included, the learning model 201 outputs the object's bounding box, type (class), and reliability score (predicted probability of each class) as a recognition result. In this case, the learning model 201 may output the recognition result by superimposing it on the input extracted image. Note that the recognizable objects and / or identifiable classes may be determined appropriately depending on the specifications and specifications of the learning model 201, the contents of the training data, and the like. For example, the learning model 201 may be capable of recognizing only automobiles as objects. Furthermore, automobiles may be recognized into more detailed classes, such as passenger cars and trucks.

[0037] The control unit 20 of the information processing device 2 acquires the recognition result of the object from the learning model 201 (S108). The control unit 20 of the information processing device 2 determines whether the object has been recognized (S109). The control unit 20 of the information processing device 2 acquires the recognition result of the object from the learning model 201 by inputting the extracted image to the learning model 201. The control unit 20 of the information processing device 2 determines whether the object has been recognized based on the recognition result acquired from the learning model 201. The control unit 20 of the information processing device 2 performs the processes of S107 to S109 on the extracted image, similar to S103 to S105.

[0038] If the object is not recognized (S109: NO), the control unit 20 repeats the process from S101 again, as described below. On the other hand, if the object is recognized (S109: YES), the control unit 20 of the information processing device 2 identifies the nearest object (S1091). If the object is recognized based on the extracted image, the control unit 20 of the information processing device 2 identifies the object that is nearest to the vehicle C from among one or more objects included in the extracted image. The control unit 20 of the information processing device 2 performs the process of S1091 on one or more objects included in the extracted image, similar to S1051.

[0039] After S1051 or S1091 is executed, the control unit 20 of the information processing device 2 determines whether an object is approaching based on information about the object detected in multiple images arranged in time series (S1052). After executing S1051 or S1091, the control unit 20 of the information processing device 2 determines whether the object recognized in the reduced image or the extracted image is approaching the vehicle C. At this time, the control unit 20 of the information processing device 2 performs object detection and identification processing using the learning model 201 for each of consecutive frames (original images) in a video, for example, and stores the processing results and intermediate data during the processing together with the original images in the storage unit 23. The control unit 20 of the information processing device 2 determines whether the object closest to the vehicle C is approaching the vehicle C using information about the object detected in multiple images arranged in time series from the past to the present stored in the storage unit 23 (for example, information indicating the position of the object or information indicating the transition of the position). Specifically, the control unit 20 may determine whether the identified object is approaching the vehicle C by using a method such as frame difference or optical flow.

[0040] When it is determined that an object is approaching (S1052: YES), the control unit 20 of the information processing device 2 outputs warning information indicating that an object is approaching (S1053). Note that the manner in which the warning information is output is not particularly limited. For example, when it is determined that an object is approaching the vehicle C, the control unit 20 of the information processing device 2 outputs the warning information indicating that an object is approaching by popping up a warning image or warning message on the image on the display unit 12 or by outputting a buzzer sound from a speaker (an example of an output device).

[0041] After executing the process of S109 or S1053, the control unit 20 of the information processing device 2 repeats the process from S101. In this way, the control unit 20 of the information processing device 2 performs the same process on the next image acquired from the camera 11 (for example, the next or subsequent frame image of a video) as on the previous image, thereby detecting the approach of an object in real time and outputting warning information.

[0042] If no object is recognized in S109 (S109: NO), or if it is determined in S1052 that no object is approaching (S1052: NO), the control unit 20 may determine that there is no approaching object. The control unit 20 may store this determination result in the storage unit 23 in association with information that indicates the time when the determination was made, such as the time of the determination.

[0043] According to this embodiment, the information processing device 2 reduces the image acquired from the camera 11 and inputs the reduced image to the learning model 201. Therefore, even if the image captured by the camera 11 has a large number of pixels, the reduced image with a relatively small number of pixels is input to the learning model 201, thereby reducing the computational load on the learning model 201 and shortening the time required for recognition processing. Then, the information processing device 2 extracts a partial area from the image acquired from the camera 11 to obtain an extracted image. The extracted image has a smaller number of pixels than the original image (image acquired from the camera 11). Even when the extracted image is input to the learning model 201, the computational load on the learning model 201 and the processing time required for recognition processing can be reduced. In this way, even if the image acquired from the camera 11 (original image) has a large number of pixels, inputting the reduced image (reduced image) or the extracted image to the learning model 201 reduces the computational load on the learning model 201 and shortens the processing time required for recognition processing. Furthermore, when constructing the learning model 201 using a neural network selected from existing neural networks, this embodiment allows for the use of a neural network with a smaller amount of calculation (lighter processing operation). If a model with a smaller amount of calculation is used, the processing time of the learning model 201 is also reduced, and as a result, the time required for the recognition process can be reduced.

[0044] According to this embodiment, the reduction process of an image performed when inputting the image to the learning model 201 is a process of reducing the number of pixels. In this reduction process, the original image and / or the extracted image are reduced in size according to the number of pixels corresponding to the learning model 201. When inputting an image to the learning model 201, the information processing device 2 performs the reduction process on the image (original image) acquired from the camera 11 and the extracted image, thereby reducing the number of pixels of these images. Alternatively, the information processing device 2 may perform the reduction process only on the image (original image) acquired from the camera 11. In this case, the information processing device 2 performs the reduction process so that the reduction ratio for the extracted image is greater than the reduction ratio for the reduced image (reduced image). The reduction ratio is calculated by dividing the number of pixels after reduction by the number of pixels before reduction. Therefore, if the reduction process is not performed on the extracted image, the reduction ratio for the extracted image is 1, which is the maximum. By increasing the reduction ratio of the extracted image, the recognition accuracy by the learning model 201 can be improved. Therefore, even if the learning model 201 is unable to recognize an object in a reduced image (reduced image), it can recognize the object contained in the extracted image by using the extracted image.

[0045] According to this embodiment, the information processing device 2 inputs a reduced image (reduced image) to the learning model 201, thereby acquiring the recognition result of the object by the learning model 201. When the learning model 201 recognizes an object, it outputs the range (bounding box), type (class), and confidence score (predicted probability of each class) of the object in the reduced image. When the learning model 201 recognizes an object based on the reduced image, the information processing device 2 does not generate an extracted image or input the extracted image to the learning model 201. In other words, the information processing device 2 inputs the extracted image to the learning model 201 only when the learning model 201 does not detect an object based on the reduced image. In this way, the information processing device 2 generates an extracted image and inputs the extracted image to the learning model 201 only when the learning model 201 does not detect an object based on the reduced image (reduced image). This reduces the computational load on the information processing device 2 and shortens the processing time required for recognition processing.

[0046] According to this embodiment, the region of the extracted image (extraction region) in the original image (image acquired from the camera 11) is determined in advance. For example, the extraction region in the original image is defined by two points in an image coordinate system and stored in the storage unit 23 of the information processing device 2. The number of pixels in the extraction region may be defined, for example, in a range from 100% to 110% of the model size of the learning model 201. The region of the extracted image (extraction region) in the original image (image acquired from the camera 11) may also include a region including the most distant scene in the original image. In other words, the extraction region may be set in the extracted image extracted from the original image so as to include the point in the original image that is farthest from the vehicle C. The extraction region set in this manner is determined according to the position of the camera 11 that captures the original image, i.e., it differs depending on the position or location of the camera 11 arranged on the vehicle C. Therefore, when the left and right cameras 11 are placed near the left and right doors of the vehicle C (where conventional door mirrors are installed), the extracted image regions (extracted regions) in the image (original image) captured by the left camera 11 and the image (original image) captured by the right camera 11 are defined to be different. This makes it possible to efficiently extract extracted images that include regions that are at a large distance (depth) from the camera 11, i.e., the vehicle C, and to recognize objects that are located relatively far away.

[0047] According to this embodiment, when the learning model 201 detects multiple objects based on a reduced image or an extracted image, the information processing device 2 identifies the object closest to the vehicle C among the multiple objects. When identifying the closest object, the information processing device 2 may perform depth calculation based on the Y coordinate value in the image coordinate system of the area (bounding box) of the object in the image (reduced image or extracted image) or using the specifications or imaging characteristics of the camera 11. Then, the information processing device 2 performs the process on multiple images arranged in time series, i.e., consecutive frames in a video, to determine whether the detected object is approaching the vehicle C by using a method such as frame difference or optical flow. When the information processing device 2 determines that an object is approaching the vehicle C, the information processing device 2 can efficiently notify the operator of the vehicle C of warning information regarding an object approaching the vehicle C (host vehicle) by, for example, displaying a warning icon on the image as a pop-up or outputting a buzzer sound, indicating that the object is approaching.

[0048] (Embodiment 2) The information processing device 2 according to the present disclosure may output warning information when there is an object that has a reliability score equal to or greater than a threshold and is approaching the vehicle C, among the objects recognized by the learning model 201. Hereinafter, the information processing device 2 according to the second embodiment will be described with reference to Fig. 7. Note that in the following embodiments, the same parts as those in the first embodiment will not be described repeatedly. 7 is a flowchart illustrating the processing of the control unit 20 of the information processing device 2 according to the second embodiment. For example, when the vehicle C is in an activated state, the control unit 20 of the information processing device 2 performs the following processing in real time. The control unit 20 of the information processing device 2 performs the processing from S201 to S205, similar to the processing from S101 to S105 in the first embodiment. Note that in this embodiment, when the learning model 201 recognizes one or more objects, it outputs a recognition result including a reliability score for each of the objects.

[0049] If the object is not recognized (S205: NO), the control unit 20 performs the process of S206 in the same manner as S106. On the other hand, if it is determined that the object is recognized (S205: YES), the control unit 20 of the information processing device 2 determines whether or not there is an object whose reliability score is equal to or greater than a threshold (S2051). Specifically, the control unit 20 of the information processing device 2 identifies the object that has the highest (largest) reliability score among the reliability scores of each object. The control unit 20 of the information processing device 2 determines whether or not the value of the identified highest reliability score is equal to or greater than a threshold (for example, 0.5 or greater).

[0050] If the reliability score is not equal to or greater than the threshold (S2051: NO), the control unit 20 of the information processing device 2 generates and acquires an extracted image by extracting a region including the object with the highest reliability score among the recognized objects (S2052). If a single object is identified in S2051 as having the highest reliability score (i.e., the highest reliability score that is less than the threshold in S2051), the control unit 20 generates and acquires an extracted image including (the region of) the object with the highest reliability score. On the other hand, if there are multiple objects with the highest reliability scores, the control unit 20 identifies the object closest to the vehicle C among these multiple objects by performing, for example, processing similar to S1051 in the first embodiment. Alternatively, if there are multiple objects with the highest reliability scores, the control unit 20 of the information processing device 2 may identify the object closest to a predetermined extracted region from among these multiple objects. The control unit 20 of the information processing device 2 extracts an extracted image so as to include the region of the identified object. At this time, the control unit 20 of the information processing device 2 may extract an extracted image centered on the center of gravity of the object.

[0051] When the control unit 20 of the information processing device 2 has performed the process of S206 or S2052, it performs the process of S207, similar to the process S107 in the first embodiment. At this time, when the control unit 20 of the information processing device 2 has performed the process of S206, similar to the process S106 in the first embodiment, an extracted image generated by extracting a predetermined area in the original image is input to the learning model 201. When the control unit 20 of the information processing device 2 has performed the process of S2052, an extracted image including an area of ​​an object recognized in the reduced image and having a reliability score less than a predetermined threshold (e.g., 0.5) is input to the learning model 201. Furthermore, the control unit 20 of the information processing device 2 performs the processes from S208 to S209, similar to the processes S108 to S109 in the first embodiment.

[0052] If the reliability score is equal to or greater than the threshold (S2051: YES), the control unit 20 of the information processing device 2 identifies the closest object among the objects whose reliability scores are equal to or greater than the threshold (S2053). The control unit 20 of the information processing device 2 performs the process of S2053, similar to the process of S1051 in the first embodiment. Furthermore, the control unit 20 of the information processing device 2 performs the process of S2054, similar to the process of S1052 in the first embodiment. Note that if there is only one object whose reliability score is equal to or greater than the threshold, the control unit 20 may perform the determination of S2054 for that one object without performing the process of S2053. If it is determined in S2054 that an object is approaching, the control unit 20 outputs warning information, similar to S1053 (S2055). If the result of S209 is NO, or after performing the process of S2055, the control unit 20 repeats the process from S201.

[0053] According to this embodiment, the information processing device 2 inputs a reduced image (reduced image) into the learning model 201 to obtain a recognition result of the object by the learning model 201. When the learning model 201 recognizes an object, it outputs the range, type, and reliability score of the object in the reduced image. If the reliability score is less than a predetermined threshold (less than 0.5), the information processing device 2 extracts an extracted image based on the position of the object whose reliability score is less than the threshold. At this time, the information processing device 2 may extract the extracted image, for example, centered on the center of gravity of the object. Since the information processing device 2 inputs the extracted image into the learning model 201, it can perform further recognition processing on the object whose reliability score is less than the threshold. At this time, by using the extracted image, it is possible to improve the recognition accuracy of the learning model 201.

[0054] (Embodiment 3) When an object is not recognized by the learning model 201, the information processing device 2 according to the present disclosure may refer to the position information and map information of the vehicle C, and may change the area to be extracted from the original image as an extracted image if the position information and map information (specifically, the shape of the road) satisfy specific conditions. Hereinafter, the information processing device 2 according to the third embodiment will be described with reference to Figs. 8 to 10. 8 is a flowchart illustrating the processing of the control unit 20 of the information processing device 2 according to the third embodiment (merging point). The control unit 20 of the information processing device 2 performs the following processing in real time, for example, when the vehicle C is in an activated state. The control unit 20 of the information processing device 2 performs the processing from S301 to S305, similar to the processing from S101 to S105 in the first embodiment.

[0055] If the object is not recognized (S305: NO), the control unit 20 of the information processing device 2 acquires position information and map information (S306). The information processing device 2 is communicably connected to the ECU 3, for example, via the network 4. The ECU 3 is connected to a GPS device 31 including a GPS antenna and the like, and periodically receives GPS signals to output the current position of the vehicle C, i.e., the traveling point at the current time, as position information. Furthermore, the ECU 3 may store map information and output the traveling position and map information of the surrounding area including the traveling position.

[0056] The information processing device 2 acquires the traveling position and surrounding map information including the traveling position from the ECU 3 connected to the GPS device 31. The map information includes various information related to the road, such as information related to the junction of a main road and a branch road. Alternatively, the GPS device 31 may be connected to the information processing device 2, and the map information may be stored in the storage unit 23 of the information processing device 2.

[0057] The control unit 20 of the information processing device 2 determines whether the traveling point is a junction point (S307). Based on the acquired position information (the traveling point at the current time) and map information (information on surrounding roads including the traveling point), the control unit 20 of the information processing device 2 determines whether the traveling point is a junction point, i.e., whether the traveling point is located within a predetermined distance (for example, within 100 m) from the junction point where a main line and a branch line merge. Alternatively, the control unit 20 of the information processing device 2 may determine whether the traveling point at the current time is located within a predetermined distance from the junction point where a main line and a branch line merge, based on an image captured a predetermined time before the current time, such as 5 minutes.

[0058] If the traveling point is a merging point (S307: YES), that is, if the current traveling point is located within a predetermined distance, the control unit 20 of the information processing device 2 acquires an extracted image in which the extraction area has been changed in accordance with the merging point (S308). If the current traveling point is located within the predetermined distance, the control unit 20 of the information processing device 2 changes a predetermined area (extraction area) by shifting (moving) the area toward the outside with respect to the vehicle C in accordance with the merging point, and extracts the changed area to generate and acquire an extracted image. The control unit 20 of the information processing device 2 may derive whether the merging point is located to the left or right with respect to the vehicle C based on the acquired traveling point and map information, and shift (move) the predetermined area (extraction area) toward the outside with respect to the vehicle C in accordance with the derived direction of the merging point.

[0059] FIG. 9 is an explanatory diagram showing the process of changing the extraction area in accordance with the merging point (shifting to the right). FIG. 10 is an explanatory diagram showing the process of changing the extraction area in accordance with the merging point (shifting to the left). When the merging point is on the left side of the vehicle C's moving direction, for example, a predetermined area (extraction area) may be shifted (moved) outward (to the left) from the vehicle C in an image (original image) captured by the camera 11 arranged on the left front door. Alternatively, when the merging point is on the right side of the vehicle C's moving direction, for example, a predetermined area (extraction area) may be shifted (moved) outward (to the right) from the vehicle C in an image (original image) captured by the camera 11 arranged on the right front door.

[0060] In this way, when vehicle C is near a merging point, the extraction region used to generate an extracted image is changed by moving (shifting) it outward relative to vehicle C. This allows, for example, when vehicle C is traveling on the main line, to efficiently identify other vehicles traveling on a branch line that will merge at the merging point, or when vehicle C is traveling on the branch line, to efficiently identify other vehicles traveling on the main line.

[0061] Alternatively, if the current traveling point is located within a predetermined distance, the control unit 20 of the information processing device 2 may, for example, calculate a coordinate difference or a merging angle between the traveling point and the merging point, and derive the area of ​​the merging point included in the original image based on the coordinate difference or the merging angle. In this case, the control unit 20 of the information processing device 2 may determine whether the merging point is located to the left or right with respect to the vehicle C based on the calculated coordinate difference or the merging angle, and derive the area of ​​the merging point included in the original image by shifting it left or right with respect to a predetermined extracted area based on the left / right determination result. The control unit 20 of the information processing device 2 may, for example, acquire the extracted image by extracting the extracted image including the merging point, with the center of gravity of the derived area of ​​the merging point as the center.

[0062] If the traveling point is not a junction point (S307: NO), the control unit 20 of the information processing device 2 acquires an extracted image including a predetermined area (S3071). If the traveling point is not a junction point, that is, if the current traveling point is not located within a predetermined distance from the junction point, the control unit 20 of the information processing device 2 acquires an extracted image including a predetermined area, similar to process S106 in the first embodiment.

[0063] After the processing of S308 or S3071 is executed, the control unit 20 of the information processing device 2 inputs the extracted image to the learning model 201 (S309). When S308 is executed, the extracted image including the junction point is input to the learning model 201. When S3071 is executed, the extracted image including a predetermined area is input to the learning model 201, similar to the processing of S106 in the first embodiment.

[0064] The control unit 20 of the information processing device 2 performs the processes of S310 and S311, similar to the processes of S108 and S109 in embodiment 1. If the answer is YES in S311, the control unit 20 performs the processes of S3111, S3052, and S3053, similar to the processes of S1091, S1052, and S1053 in embodiment 1. Furthermore, if the answer is YES in S305, the control unit 20 of the information processing device 2 performs the processes of S3051 to S3053, similar to the processes of S1051 to S1053 in embodiment 1. If the answer is NO in S311, or after performing the process of S3053, the control unit 20 repeats the process from S301.

[0065] In the present embodiment, when an object is not recognized based on the reduced image, the control unit 20 of the information processing device 2 acquires position information and map information and makes a determination regarding the junction, etc. However, this is not limited to this. Even when an object is recognized based on the reduced image, the control unit 20 of the information processing device 2 may acquire position information and map information, make a determination regarding the junction, etc., and generate an extracted image including the junction.

[0066] According to this embodiment, the information processing device 2 inputs a reduced image (reduced image) to the learning model 201, thereby acquiring a recognition result of the object by the learning model 201. If the learning model 201 does not detect an object based on the reduced image, the information processing device 2 inputs an extracted image to the learning model 201. At that time, if the current traveling point of vehicle C is a merging point on a road, the information processing device 2 changes a predetermined extraction area so that an area indicating the merging point is included, i.e., moves (shifts) the extraction area outward relative to vehicle C, and extracts the extracted image. When determining whether the current traveling point of vehicle C is a merging point on a road, the information processing device 2 may acquire, from another EUC mounted on vehicle C, position information (current traveling point of vehicle C) acquired from a GPS mounted on vehicle C and map information including the traveling point, and determine whether a merging point exists within a predetermined distance around vehicle C. In this way, the extracted image includes merging points located within a specified distance, such as behind vehicle C, and therefore, objects such as other cars or bicycles from the merging point can be efficiently recognized using learning model 201.

[0067] (Embodiment 4) The information processing device 2 according to the present disclosure may alternately or in parallel perform object recognition processing using a reduced image and object recognition processing using an extracted image. Furthermore, when an object is recognized by at least one of these two types of object recognition processing, the information processing device 2 may identify the object closest to the vehicle C among the recognized objects and determine whether the identified object is approaching the vehicle C. Then, when it is determined that the identified object is approaching the vehicle C, the information processing device 2 may cause the output device to output warning information. 11 is a flowchart illustrating the processing of the control unit 20 of the information processing device 2 according to the fourth embodiment (constant extraction of an extracted image). The control unit 20 of the information processing device 2 performs the following processing in real time, for example, when the vehicle C is in an activated state. The control unit 20 of the information processing device 2 performs the processing from S401 to S404, similar to the processing from S101 to S104 in the first embodiment. The control unit 20 of the information processing device 2 performs the processing from S405 to S407, similar to the processing from S106 to S108 in the first embodiment. Note that the processing from S402 to S404 and the processing from S405 to S407 are performed in no particular order.

[0068] The control unit 20 of the information processing device 2 determines whether one or more objects have been recognized from the recognition results of S404 and S407 (S408). The control unit 20 of the information processing device 2 acquires the recognition result for the object output by the learning model 201 based on the reduced image and the recognition result for the object output by the learning model 201 based on the extracted image. The control unit 20 of the information processing device 2 determines whether an object has been recognized from these two recognition results, i.e., from either the reduced image or the extracted image.

[0069] If one or more objects are recognized (S408: YES), the control unit 20 of the information processing device 2 identifies the nearest object (S4081). The control unit 20 of the information processing device 2 performs the processes from S4081 to S4083, similar to the processes from S1051 to S1053 in the first embodiment.

[0070] If no object is recognized (S408: NO), or after performing the process of S4083, the control unit 20 repeats the process from S401.

[0071] (Variation 1) In each of the above-described embodiments, the information processing device 2 may also function as the ECU 3. In this case, the control unit 20 also functions as the control unit of the ECU 3, and the communication unit 22 also functions as the communication unit of the ECU 3. In this case, the GPS device 31 is connected to the information processing device 2 via the network 4. The information processing device 2, the ECU 3, and the control program P may be devices outside the vehicle.

[0072] (Variation 2) In each of the above embodiments, the control unit 20 of the information processing device 2 determines whether or not an object is approaching for one object that is closest to the vehicle C among the objects that can be recognized by the learning model 201. However, the information processing device 2 according to the present disclosure may determine whether or not an object is approaching for multiple objects that can be recognized by the learning model 201. Then, if it is determined that one or more objects are approaching, warning information may be output to the display unit 12 or the like.

[0073] For example, in FIG. 4, if the answer is YES in S105, the determination process in S1052 may be performed for each of the objects recognized by the learning model 201 without performing the identification process in S1051. Similarly, if the answer is YES in S109, the determination process in S1052 may be performed for each of the objects recognized by the learning model 201 without performing the identification process in S1091. Alternatively, in FIG. 4, if the answer is YES in S105, a predetermined number of objects may be identified in S1051 in order of proximity to vehicle C, and the process in S1052 may be performed for each of the objects. Similarly, if the answer is YES in S109, a predetermined number of objects may be identified in S1091 in order of proximity to vehicle C, and the process in S1052 may be performed for each of the objects. The same applies to steps in other embodiments that perform processes similar to these steps.

[0074] (Variation 3) In the present disclosure, a plurality of camera ECUs may be mounted on the vehicle C. For example, if cameras 11 are arranged on the left and right sides of the vehicle C, a camera ECU for the left camera and a camera ECU for the right camera may be mounted on the vehicle C. When a plurality of camera ECUs are used in this manner, any one of the camera ECUs may be configured to function as the information processing device 2 in each of the above-described embodiments, or the processing executed by the information processing device 2 may be distributed among the plurality of camera ECUs.

[0075] The embodiments disclosed herein should be considered to be illustrative in all respects and not restrictive. The scope of the present disclosure is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims.

[0076] Multiple claims may be combined with each other regardless of the form of reference. Multiple dependent claims may be included in the claims, depending on multiple claims. Multiple dependent claims may be included in a multiple dependent claim. If multiple dependent claims are not included in a multiple dependent claim, this does not limit the number of multiple dependent claims that are included in a multiple dependent claim. [Explanation of symbols]

[0077] C vehicle 1 camera unit 11 Camera 12 Display section 2. Information processing unit (camera ECU) 20 Control Unit 21 Input / Output Interface 22 Communications Department 23 Memory section P Control Program (Program Product) 201 Learning model (object detection model) 3 ECU 31 GPS device 4 Network

Claims

1. Acquire images captured by a camera mounted on the vehicle, A learning model is trained to output a recognition result of an object contained in an image when the image is input, and recognizes the object based on the output when the reduced image is input. obtaining an extracted image by extracting a partial region from the image; The object is recognized based on the output when the acquired extracted image is input to the learning model. An information processing method that causes a computer to execute a process.

2. In the image reduction process performed when the image is input to the learning model, the reduction ratio of the extracted image is greater than the reduction ratio of the reduced image. The information processing method according to claim 1 .

3. and reducing the number of pixels of the image and / or the extracted image so that the number of pixels falls within the number of pixels corresponding to the learning model.

3. The information processing method according to claim 1.

4. If the learning model does not recognize the object when the reduced image is input, the extracted image extracted from the image or a reduced extracted image obtained by reducing the extracted image is input to the learning model. The information processing method according to claim 1 .

5. an area to be extracted as the extracted image from the image captured by the camera is determined in advance; the camera mounted on the vehicle includes a plurality of cameras arranged at different positions on the vehicle; The area extracted from the image as the extracted image differs depending on each of the plurality of cameras. The information processing method according to claim 1 .

6. obtaining a confidence score when the learning model recognizes an object based on the reduced image; If the confidence score is less than a predetermined threshold, an extracted image is obtained by extracting an area determined from the position of the object in the image; The extracted image is input into the learning model. The information processing method according to claim 1 .

7. determining whether the current traveling point of the vehicle is a merging point on a road based on the position information of the vehicle acquired from a GPS mounted on the vehicle; If the current traveling point of the vehicle is the junction, the area extracted from the image as the extracted image is changed in accordance with the junction. The information processing method according to claim 1 .

8. When a plurality of objects are recognized by the recognition result of the learning model, the object that is closest to the vehicle is identified among the plurality of objects; Determine whether the identified object is approaching the vehicle. The information processing method according to claim 1 .

9. The images include a plurality of images arranged in time series, When determining whether the specified object is approaching the vehicle, information about the specified object in each of the plurality of images is used to determine whether the object is approaching the vehicle. The information processing method according to claim 8.

10. When it is determined that the specified object is approaching the vehicle, an output device is caused to output warning information indicating that the specified object is approaching. The information processing method according to claim 9.

11. Acquire images captured by a camera mounted on the vehicle, A learning model is trained to output a recognition result of an object contained in an image when the image is input, and recognizes the object based on the output when the reduced image is input. obtaining an extracted image by extracting a partial region from the image; The object is recognized based on the output when the acquired extracted image is input to the learning model. A program that causes a computer to perform a process.

12. An information processing device mounted on a vehicle and including a control unit, The control unit Acquire images captured by a camera mounted on the vehicle, A learning model is trained to output a recognition result of an object contained in an image when the image is input, and recognizes the object based on the output when the reduced image is input. obtaining an extracted image by extracting a partial region from the image; The object is recognized based on the output when the acquired extracted image is input to the learning model. Information processing device.

13. a camera mounted on a vehicle and having an imaging range including the rear of the vehicle; a camera ECU communicably connected to the camera; A camera unit including a display unit communicably connected to the camera ECU, The camera ECU Acquire an image captured by the camera; A learning model is trained to output a recognition result of an object contained in an image when the image is input, and recognizes the object based on the output when the reduced image is input. obtaining an extracted image by extracting a partial region from the image; Recognizing an object based on an output when the acquired extracted image is input to the learning model; The acquired image is output to the display unit together with information about the object recognized based on the reduced image or the object recognized based on the extracted image. Camera unit.

Images