Information processing device, information processing method, and recording medium

Abstract

Provided are an information processing device, an information processing method, and a recording medium capable of improving object detection performance while suppressing an increase in computing resources. The information processing device (140) is configured to obtain sensing data from an optical sensor (110), obtain position information of an object (100) equipped with the optical sensor (110), obtain map information, determine a specific sensing data area in the sensing data corresponding to a specific area in the sensing area of ​​the optical sensor (110) using the obtained position information and map information, determine input information of an object detection model according to the specific sensing data area, and use the input information to cause the object detection model to perform object detection processing.

Description

Technical Field

[0001] The present disclosure relates to an information processing device, an information processing method, and a recording medium. Background Art

[0002] In recent years, many studies and commercializations of self-driving cars (robot cars) that are mechanically driven instead of drivers have been conducted. The self-driving car is equipped with an object detection device for detecting objects around the self-driving car. Patent document 1 proposes an object recognition device that sets a priority recognition area near other detected vehicles and recognizes pedestrians in the priority recognition area.

[0003] (Prior art literature)

[0004] (Patent Document)

[0005] Patent Document 1: Japanese Patent Application Publication No. 2017-167608

[0006] However, in the prior art, there is a possibility that the computational resources used for object detection will increase. For example, in Patent Document 1, pedestrian detection is performed after vehicle detection, so at least two detection processes are performed. Therefore, more computational resources are used than a single detection process. Summary of the invention

[0007] Therefore, an object of the present disclosure is to provide an information processing device, an information processing method, and a program that can improve object detection performance while suppressing an increase in computational resources used for object detection.

[0008] An information processing device involved in one of the embodiments of the present disclosure is configured to obtain sensor data from an optical sensor, obtain position information of an object equipped with the optical sensor, obtain map information, determine a specific sensor data area in the sensor data corresponding to a specific area in the sensor area of ​​the optical sensor using the position information and the map information, determine input information of an object detection model according to the specific sensor data area, and use the input information to enable the object detection model to perform object detection processing.

[0009] Furthermore, one of the embodiments of the present disclosure involves an information processing method, which utilizes a processor to obtain sensor data from an optical sensor, obtain position information of an object equipped with the optical sensor, and obtain map information, and utilizes the position information and the map information to determine a specific sensor data area in the sensor data corresponding to a specific area in the sensor area of ​​the optical sensor, determines input information of an object detection model according to the specific sensor data area, and utilizes the input information to enable the object detection model to perform object detection processing.

[0010] Furthermore, a recording medium computer having a program recorded thereon according to one embodiment of the present disclosure is a recording medium having a program recorded thereon for causing a computer to execute the information processing method.

[0011] Furthermore, their general or specific forms may be implemented as systems, methods, integrated circuits, computer programs, or computer-readable recording media such as CD-ROMs, or may be implemented as any combination of systems, methods, integrated circuits, computer programs, and recording media.

[0012] An information processing device or the like according to one embodiment of the present disclosure can improve object detection performance while suppressing an increase in computational resources used for object detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is a diagram showing a schematic configuration of an information processing device according to Embodiment 1.

[0014] Figure 2A This is a diagram showing an example of a general template rectangular group according to the first embodiment.

[0015] Figure 2B This is a diagram showing an example of a sidewalk template rectangular group according to the first embodiment.

[0016] Figure 3 This is a flowchart showing an example of the operation of the information processing device according to the first embodiment.

[0017] Figure 4 This is a flowchart showing an example of the operation of the region of interest determination unit according to the first embodiment.

[0018] Figure 5 This is a diagram showing an example of an image obtained by the region of interest determination unit according to the first embodiment.

[0019] Figure 6 This is a diagram showing an example of map information obtained by the region of interest determination unit according to the first embodiment.

[0020] Figure 7 is Figure 5 Figure 3. Image overlap of the region of interest.

[0021] Figure 8 This is a diagram in which the detection result according to the first embodiment is superimposed on an image.

[0022] Fig. 9 This is a diagram in which distance information is superimposed on an image according to the first embodiment.

[0023] Fig.10This is a flowchart showing an example of the operation of the information processing device involved in the second embodiment.

[0024] Fig.11 This is a flowchart showing the operation of generating an input image by the object detection unit according to the second embodiment.

[0025] Fig. 12A FIG. 1 is a diagram showing an image before reduction when a region of interest is included according to the second embodiment.

[0026] Fig. 12B FIG. 1 is a diagram showing a reduced image including a region of interest according to the second embodiment.

[0027] Fig.13A FIG. 1 is a diagram showing an image before reduction when the region of interest is not included according to the second embodiment.

[0028] Fig. 13B FIG. 1 is a diagram showing a reduced image when the region of interest is not included according to the second embodiment.

[0029] Explanation of symbols

[0030] 100 moving objects

[0031] 110 Optical Sensor

[0032] 120 Positioning Department

[0033] 121 Position Acquisition Unit

[0034] 122 Position Acquisition Department

[0035] 130 Map Storage Department

[0036] 140 Information processing device

[0037] 141 Focus Area Determination Department (Determination Department)

[0038] 142 Template rectangle setting unit

[0039] 142a General Template Rectangular Group

[0040] 142b Sidewalk template rectangular group

[0041] 143 Input image setting unit

[0042] 144 Object detection unit (model control unit)

[0043] 150 Vehicle Control Unit

[0044] H1 Object detection rectangle

[0045] M1 Map Information

[0046] P1, P2, P11, P22 images

[0047] R1, R2 sidewalk section

[0048] R11, R22 clipping area

[0049] R33 All Areas DETAILED DESCRIPTION

[0050] (Knowledge that is the basis of this disclosure)

[0051] As described above, in the prior art, there is a possibility that computing resources for object detection will increase. In addition, if the object detection unit using artificial intelligence such as neural networks or pattern matching technology, which is a component of an autonomous vehicle (an example of a mobile body), wants to improve its detection capability, it will cause problems such as lack of real-time detection processing and increased power consumption. Furthermore, in the prior art, the priority recognition area is limited to the vicinity of the vehicle, so the detection capability of people in the area where the vehicle does not exist is not necessarily improved.

[0052] In this regard, one of the embodiments of the present disclosure involves an information processing device that is configured to obtain sensor data from an optical sensor, obtain position information of an object equipped with the optical sensor, and obtain map information; use the position information and the map information to determine a specific sensor data area in the sensor data corresponding to a specific area in the sensor area of ​​the optical sensor; determine input information of an object detection model according to the specific sensor data area; and use the input information to enable the object detection model to perform object detection processing.

[0053] According to this, the input information of the object detection model is determined according to the specific sensor data area, so the processing amount or processing content of the object detection model can be controlled according to the specific sensor data area and other areas. Therefore, the increase of computing resources used for object detection can be suppressed while improving the object detection performance.

[0054] For example, when the specific sensing data area is a sidewalk, the input information corresponding to the sidewalk is input to the object detection model, thereby improving the detection performance of vulnerable traffic persons such as children and wheelchair users who are more likely to be on the sidewalk than in other areas. In addition, by allocating computing resources according to the sidewalk and other areas, it is possible to suppress the increase in computing resources required for object detection processing. Therefore, the safety of vulnerable traffic persons can be improved with the limited computing resources of the information processing device.

[0055] Furthermore, for example, in the object detection model, an object template rectangle used for object detection is used to select an object for performing object detection processing, and the input information includes the object template rectangle.

[0056] According to this, the object to be processed by the object detection model is selected according to a specific sensing data area, so that the object detection performance can be improved while suppressing the increase in computing resources required for processing the object detection model.

[0057] And, for example, the object template rectangle is set according to the attribute of the specific sensor data area.

[0058] According to this, the object to be processed by the object detection model is selected according to the attribute of the specific sensor data area. Here, there is a typical camera method for each type of object (for example, a person, a car, etc.). In addition, according to each attribute of the sensor data area, there are different types of objects that may exist in the sensor data area. Therefore, as in this structure, the object to be processed by the object detection model is selected according to the attribute of the specific sensor data area, thereby effectively achieving both the suppression of the increase in computing resources and the improvement of object detection performance.

[0059] And, for example, distance information about the sensing data is obtained, and the size of the object template rectangle is set according to the obtained distance information.

[0060] According to this, the size of the object template rectangle is set according to the distance information, so that the object detection performance can be improved while suppressing the increase of calculation resources for object detection.

[0061] And, for example, the optical sensor includes an image sensor, the sensor data includes a sensor image, which is an image obtained from the image sensor, the input information includes an image based on the sensor image, and the resolution of other areas of the image outside the specific sensor data area is lower than the resolution of the specific sensor data area.

[0062] According to this, the computational resources that were allocated to other regions other than the region of interest can be allocated to the region of interest, thereby improving the object detection performance while suppressing an increase in computational resources.

[0063] Furthermore, for example, the optical sensor includes an image sensor, the sensor data includes a sensor image, which is an image obtained from the image sensor, the input information includes a composite image of an image of the specific sensor data area cut out from the sensor image and an overall image, the composite image is an image based on the sensor image, the overall image is an image including both the specific sensor data area and other areas outside the specific sensor data area, and the resolution of the overall image is lower than that of the image of the specific sensor data area.

[0064] According to this, since both the image of the target region and the entire image are input to the object detection model, it is possible to suppress the occurrence of erroneous detection or missed detection of objects due to clipping of the target region.

[0065] Furthermore, for example, one of the embodiments of the present disclosure involves an information processing method that utilizes a processor to obtain sensing data from an optical sensor, obtain position information of an object equipped with the optical sensor, obtain map information, utilize the position information and the map information to determine a specific sensing data area in the sensing data corresponding to a specific area in the sensing area of ​​the optical sensor, determine input information of an object detection model according to the specific sensing data area, and utilize the input information to enable the object detection model to perform object detection processing.

[0066] According to this, the input information of the object detection model is determined according to the specific sensor data area, so the processing amount or processing content of the object detection model can be controlled according to the specific sensor data area and other areas. Therefore, the increase of computing resources used for object detection can be suppressed while improving the object detection performance.

[0067] Furthermore, for example, a program according to one embodiment of the present disclosure may be a program for causing a computer to execute the information processing method.

[0068] Accordingly, the computer can execute the information processing method according to the program.

[0069] Furthermore, their general or specific forms may be implemented as systems, devices, methods, integrated circuits, computer programs, or non-temporary recording media such as computer-readable CD-ROMs, and may be implemented as any combination of systems, devices, methods, integrated circuits, computer programs, and recording media.

[0070] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Moreover, the embodiments described below all illustrate a specific example of the present disclosure. The numerical values, shapes, constituent elements, steps, the order of steps, etc. shown in the following embodiments are an example and do not limit the purpose of the present disclosure. Therefore, the constituent elements of the following embodiments that are not recorded in the implementation scheme showing the highest concept are described as arbitrary constituent elements. Moreover, in all embodiments, the various contents can also be combined.

[0071] Furthermore, each drawing is a schematic diagram and is not necessarily a strict representation.

[0072] Furthermore, in this specification, terms such as equal or parallel that indicate the relationship between elements, terms such as rectangle that indicate the shape of an element, and numerical values ​​and numerical ranges do not express only strict meanings, but also mean that they also include substantially equivalent ranges, such as differences of about several percent.

[0073] (Implementation Method 1)

[0074] Hereinafter, the information processing device according to the present embodiment will be described with reference to Figures 1 to 9 Provide explanation.

[0075] [1-1. Configuration of information processing device]

[0076] First, regarding the structure of the mobile body 100 equipped with the information processing device 140, refer to Figure 1 Provide explanation.

[0077] Figure 1 1 is a diagram showing a schematic configuration of a mobile object 100 having an information processing device 140 according to the present embodiment. In the present embodiment, the mobile object 100 is a vehicle. The vehicle is, for example, an automatic driving car that controls the driving of the vehicle without requiring operation by a driver, but may also be a vehicle that can be switched to automatic driving or manual driving.

[0078] like Figure 1 The moving object 100 is shown to include an optical sensor 110, a positioning unit 120, a map storage unit 130, an information processing device 140, and a vehicle control unit 150. The moving object 100 is an example of an object on which the optical sensor 110 is mounted.

[0079] The optical sensor 110 is a device that detects the conditions around the mobile body 100. The optical sensor 110 detects, for example, the positions of other vehicles and pedestrians around the mobile body 100. The optical sensor 110 is implemented by a sensor device that generates a captured image of the surroundings of the mobile body 100, such as a camera or LIDAR (Light Detection and Ranging) having an image sensor. The following description takes the case where the optical sensor 110 is a camera as an example. The optical sensor 110 outputs the captured image (in this embodiment, the camera image) to the information processing device 140. In addition, the sensor provided to the mobile body 100 is not limited to an optical sensor. The sensor may be, for example, a sensor that detects temperature, pressure, magnetism, sound, light, gas, ultrasonic waves, and electromagnetic waves, and outputs sensor data showing the detection results. In addition, when the optical sensor 110 has an image sensor, the sensor data includes a sensor image that is an image obtained by the image sensor.

[0080] The positioning unit 120 measures the position and direction of the moving object 100. The positioning unit 120 includes a position obtaining unit 121 and an orientation obtaining unit 122.

[0081] The position acquisition unit 121 acquires from the outside the current position of the mobile object 100. The position acquisition unit 121 is implemented by, for example, a GNSS receiving device that receives radio waves transmitted from a navigation satellite (artificial satellite) such as a Global Navigation Satellite System (GNSS).

[0082] The orientation acquisition unit 122 acquires the orientation (e.g., the traveling direction) of the mobile body 100. The orientation acquisition unit 122, for example, has a geomagnetic sensor (e.g., a compass), and measures the orientation of the mobile body 100 based on the geomagnetism detected by the geomagnetic sensor, thereby acquiring the orientation of the mobile body 100. Moreover, the method by which the orientation acquisition unit 122 acquires the orientation is not limited to the above. The orientation acquisition unit 122 may also, for example, have a gyro sensor, and acquire the orientation of the mobile body 100 using the angular velocity output by the gyro sensor.

[0083] The positioning unit 120 outputs position information (for example, latitude and longitude) indicating the location of the moving object 100 and direction information indicating the direction in which the moving object 100 is traveling to the information processing device 140 using the information obtained by the position obtaining unit 121 and the direction obtaining unit 122 .

[0084] The map storage unit 130 outputs the map information of the surroundings of the mobile body 100 to the information processing device 140. The map information may be the map information stored in the map storage unit 130 or the map information obtained by the map storage unit 130 from the outside. The map information includes road information and building information of the land. Moreover, the map information may be frequently changed according to events (for example, the time for students to go to school or go home at a nearby school, the occurrence of an accident, or traffic restrictions). For example, the map information may be dynamic map information that associates so-called static map information with event information that changes moment by moment or in real time. If the map information is dynamic map information, the area of ​​interest described later can be set more appropriately.

[0085] The information processing device 140 is a device that performs information processing. For example, the information processing device 140 is a computer that uses various information obtained from the optical sensor 110, the positioning unit 120, and the map storage unit 130 to detect objects around the moving body 100 and output the detection results. The information processing device 140 is a device that sets the inference method of the object detection unit 144 to obtain a detection result that is more in line with the intention of the user (for example, the driver). Figure 1 As shown, the information processing device 140 includes a focus region determination unit 141 , a template rectangle setting unit 142 , an input image setting unit 143 , and an object detection unit 144 .

[0086] The region of interest determination unit 141 determines the region of interest in the image, which is the sensor data obtained from the optical sensor 110, based on the position information of the mobile body 100 and the map information. The region of interest is a specific sensor data area in the image corresponding to a specific area (e.g., an area to be focused on) in the sensor area of ​​the optical sensor 110. In the present embodiment, the region of interest is an area in the image corresponding to the sidewalk area as a specific area in the sensor area of ​​the optical sensor 110. That is, in the present embodiment, the region of interest is the sidewalk portion in the image. Moreover, the specific area is not limited to the sidewalk, but is appropriately determined according to the travel path of the mobile body 100, etc.

[0087] The region of interest determination unit 141 obtains an image from the optical sensor 110. In other words, the region of interest determination unit 141 functions as a sensor data acquisition unit that obtains an image from the optical sensor 110. Furthermore, the region of interest determination unit 141 obtains map information from the map storage unit 130. In other words, the region of interest determination unit 141 functions as a map information acquisition unit that obtains map information. Furthermore, the region of interest determination unit 141 obtains position information and direction information from the positioning unit 120. In other words, the region of interest determination unit 141 functions as a position information acquisition unit that obtains position information of the mobile body 100 equipped with the optical sensor 110. Furthermore, the region of interest determination unit 141 obtains road information in the sensing direction of the optical sensor 110 based on the position information, direction information, and map information. The region of interest determination unit 141 determines a region of interest (e.g., a sidewalk portion) that is a specific sensor data region in the image using the road information. In other words, the region of interest determination unit 141 functions as a determination unit that determines a region of interest in the image. The present disclosure has a characteristic that the attention area determination unit 141 determines the attention area using map information. The attention area is an example of a desired area. In the following, an example in which the attention area is a sidewalk is described.

[0088] The template rectangle setting unit 142, as an example of an object detection parameter setting unit, sets the parameters of the object detection unit 144 using the region of interest. Specifically, the template rectangle setting unit 142 determines the object template rectangle (often referred to as the anchor box) to be used by the object detection unit 144 according to the shape of the region of interest on the sensing data, i.e., the image. The template rectangle setting unit 142 outputs information related to the object template rectangle (hereinafter also simply referred to as the template rectangle) to be used to the object detection unit 144. Here, the template rectangle is referred to as Figure 2A as well as Figure 2B Provide explanation.

[0089] Figure 2A 1 is a diagram showing an example of the general template rectangular group 142 a according to the present embodiment. Figure 2B 1 is a diagram showing an example of the sidewalk template rectangular group 142 b according to the present embodiment. Figure 2BThe sidewalk template rectangle group 142b shown is an example of a template rectangle group for a region of interest. The general template rectangle group 142a is used, for example, to detect objects in a region other than the region of interest. The sidewalk template rectangle group 142b is used to detect objects in the sidewalk portion of the region of interest. Moreover, the template rectangle is information on the aspect ratio of the vertical and horizontal directions. The template rectangle is calculated using various algorithms (for example, k-means clustering) based on the data set (image data and annotation data, etc.) used when training the object detection unit 144. For example, the object detection unit 144 may also learn the template rectangles included in the general template rectangle group 142a and the sidewalk template rectangle group 142b, and the learned template rectangles are stored in a storage unit (not shown) possessed by the information processing device 140.

[0090] like Figure 2A as well as Figure 2B In this case, the template rectangle setting unit 142 outputs information indicating that five types of template rectangles are used as the general template rectangle group 142a and the sidewalk template rectangle group 142b, respectively, to the object detection unit 144.

[0091] The general template rectangle group 142a includes template rectangles for detecting objects existing on the road on which the moving body 100 travels. The general template rectangle group 142a includes template rectangles for detecting people, vehicles, bicycles, etc. on an average basis. Furthermore, the sidewalk template rectangle group 142b includes template rectangles that are easier to detect people than the general template rectangle group 142a. For example, the sidewalk template rectangle group 142b mainly includes template rectangles for detecting pedestrians. In the focus area template rectangle group, a template rectangle corresponding to the focus area is set.

[0092] The template rectangles included in the general template rectangle group 142a and the sidewalk template rectangle group 142b may be partially identical. Figure 2A as well as Figure 2B , an example in which three of the five template rectangles are the same. For example, the sidewalk template rectangle group 142b may also include template rectangles that can detect other objects such as vehicles. In this way, it is possible to detect vehicles on the sidewalk, such as vehicles crossing the sidewalk to move from a parking lot to a driveway.

[0093] Furthermore, the number of template rectangles included in the general template rectangle group 142a and the template rectangle group 142b for sidewalks is not limited thereto. The number of template rectangles included in the general template rectangle group 142a and the template rectangle group 142b for sidewalks, if the same, may be less than 4 types, 9 types, 12 types, or more. Furthermore, the template rectangle setting unit 142 may also set a template rectangle group for an area of ​​interest including a template rectangle corresponding to the attribute of the area of ​​interest (e.g., sidewalk, lane, etc.) for each area of ​​interest when there are multiple types of areas of interest. For example, when the area of ​​interest is the sky, the template rectangle group for the area of ​​interest used for the area of ​​interest may also include a template rectangle corresponding to a flying object such as a drone. Accordingly, a flying object flying at a low altitude can be detected with higher accuracy. Furthermore, the map information may also include information on the traffic arrangement of the sky, such as the flight path of a flying object such as a drone and the stop position of the flying object in the high altitude.

[0094] Furthermore, the template rectangle may be a template rectangle corresponding to the road on which the moving body 100 is traveling. For example, the template rectangle may be a template rectangle dedicated to a car. When the moving body 100 is traveling on a highway (for example, traveling in a convoy), the car-dedicated template rectangle is used, thereby enabling efficient detection of the car.

[0095] Refer again Figure 1 The template rectangle setting unit 142, for example, in the case where there is a focus area in the image, instructs the object detection unit 144 to use Figure 2B The template rectangle group 142b for the sidewalk is shown. Furthermore, the template rectangle setting unit 142 instructs the object detection unit 144 to use the Figure 2A A generic template rectangle group 142a is shown.

[0096] The input image setting unit 143 outputs the content of the processing performed on the image data obtained from the attention area determination unit 141 to the object detection unit 144. For example, the input image setting unit 143 outputs information indicating that a process of cutting out a part of the image according to the attention area is performed to the object detection unit 144. Specifically, when the attention area exists, the input image setting unit 143 instructs the object detection unit 144 to cut out the image including the attention area from the original image (for example, the camera image obtained from the optical sensor 110), and generate an image (for example, a composite image) including the cut image and the original image according to the information of the image size set in advance. In addition, in the present embodiment, the input image setting unit 143 may set the image obtained from the attention area determination unit 141 as the input image and instruct the object detection unit 144. That is, the input image setting unit 143 may output information indicating that the image size is not changed to the object detection unit 144. In addition, the input image is an image used by the object detection unit 144 to detect the object. In addition, the original image is an example of the entire image.

[0097] The object detection unit 144 detects an object based on the input sensor data, i.e., an image, etc. The object detection unit 144 is an information processing unit that controls the processing for the object detection model, and can be composed of a circuit. The object detection model is a mathematical model (inferencer) of a neural network type having multiple layers, and is also a mathematical model for performing object detection processing. The object detection unit 144 is an example of a model control unit.

[0098] Basically, the object detection model has multiple processing layers consisting of an input layer, an intermediate layer, and an output layer. The object data of the object detection processing is input to the input layer as input data (input information). And, from the output layer, the processing result data of the object detection processing is output as output data (output information). The input layer and the output layer are different processing layers, and the processing layer between the input layer and the output layer is called an intermediate layer. The object detection model can also have multiple intermediate layers.

[0099] For example, an object detection model learned using learning data may be installed inside object detection unit 144. Specifically, an object detection model learned by a device different from information processing device 140 may be installed inside object detection unit 144. Alternatively, object detection unit 144 may learn an object detection model installed inside object detection unit 144, and the learned object detection model may be installed inside object detection unit 144.

[0100] Specifically, the learning data includes input data and output data that is accurate for the input data. Furthermore, in learning the object detection model, the input data included in the learning data is input to the input layer of the object detection model. Furthermore, the object detection model is updated in such a way that the output data from the output layer of the object detection model is equal to the output data included in the learning data, thereby learning the object detection model.

[0101] Furthermore, for example, the object detection unit 144 inputs information such as the sensor data obtained by the focus region determination unit 141 to the input layer of the object detection model. Furthermore, the object detection unit 144 performs processing up to the first intermediate layer of the object detection model. The first intermediate layer may be predetermined or selected from a plurality of layers of the object detection model.

[0102] Furthermore, the object detection unit 144 inputs the intermediate input data generated based on the first intermediate output data from the first intermediate layer of the object detection model to the layer after the first intermediate layer. The layer after the first intermediate layer is, specifically, the next layer of the first intermediate layer. Furthermore, the object detection unit 144 performs processing up to the output layer of the object detection model.

[0103] Furthermore, the object detection unit 144 outputs the output data (detection result) from the output layer of the object detection model to the outside. The object detection unit 144 may also output the detection result to the vehicle control unit 150, for example. Accordingly, the vehicle control unit 150 can control the operation of the mobile body 100 based on the output data from the output layer of the object detection model. Furthermore, the object detection unit 144 may also display the detection result on a display unit (not shown) provided in the mobile body 100. Accordingly, the detection result can be notified to the driver.

[0104] Furthermore, the object detection model may be installed in other devices. The object detection unit 144 may control the input and output of the object detection model, and the substantial processing of the object detection model may be performed by other devices.

[0105] The object detection unit 144 uses Figure 2A as well as Figure 2B In the present embodiment, the object detection unit 144 detects an object using the template rectangles of both the general template rectangle group 142a and the sidewalk template rectangle group 142b. For example, the object detection unit 144 learns all kinds of template rectangles for object detection, and the template rectangle specified by the learned template rectangles of all kinds is selected by the instruction from the template rectangle setting unit 142, thereby realizing the object detection unit 144 capable of changing the template rectangle.

[0106] Furthermore, the multiple components of the information processing device 140 may be composed of dedicated hardware, general-purpose hardware that executes the program, etc., or a combination thereof. Furthermore, the general-purpose hardware may be composed of a memory storing a program, and a general-purpose processor that reads the program from the memory and executes it. Here, the memory may be a semiconductor memory or a hard disk, etc., and the general-purpose processor may be a CPU, etc. Furthermore, the dedicated hardware may be composed of a memory and a dedicated processor, etc.

[0107] The vehicle control unit 150 controls the operation (driving) of the moving body 100 based on the detection result of the object detection unit 144. In addition, the vehicle control unit 150 can also control the operation of the moving body 100 based on the detection result and the map information. For example, even if no pedestrian is detected on the sidewalk according to the detection result, the vehicle control unit 150 can also perform control such as reducing the speed of the moving body 100 when the map information includes an event where people gather around the driving road.

[0108] Furthermore, in the above description, an example is described in which the information processing device 140 includes both the template rectangle setting unit 142 and the input image setting unit 143 . However, in the present embodiment, it is sufficient to include at least the template rectangle setting unit 142 .

[0109] As described above, the information processing device 140 includes: a sensor data acquisition unit that acquires the image P1 (an example of sensor data) from the optical sensor 110; a position information acquisition unit that acquires position information of the moving body 100 (an example of an object) equipped with the optical sensor 110; a map information acquisition unit that acquires map information; a focus area determination unit 141 (an example of a determination unit) that determines a focus area (an example of a specific sensor data area) of the image P1 corresponding to a pedestrian area (an example of a specific area) of the sensor area of ​​the optical sensor 110 using the position information and the map information; and an object detection unit 144 (an example of a model control unit) that determines input information of an object detection model according to the focus area and causes the object detection model to perform object detection processing using the input information. In addition, the focus area determination unit 141 has the functions of the sensor data acquisition unit, the position information acquisition unit, and the map information acquisition unit.

[0110] [1-2. Operation of information processing device]

[0111] Next, regarding the operation of the information processing device 140, refer to Figures 3 to 9 Provide explanation.

[0112] Figure 3This is a flowchart showing an example of the operation of the information processing device 140 according to the present embodiment.

[0113] like Figure 3 As shown, first, the information processing device 140 obtains image data from the optical sensor 110 ( S10 ). Specifically, the region of interest determination unit 141 obtains the image data. Figure 5 : is a diagram showing an example of an image P1 obtained by the attention area determination unit 141 according to the present embodiment. It is assumed that the attention area determination unit 141 obtains, for example, an image P1 indicated by image data. Figure 5 The range of the captured image P1 shown is an example of a sensing area. Furthermore, the image P1 is an example of sensing data obtained from the optical sensor 110. Furthermore, when the optical sensor 110 is a LIDAR, the range where the laser is emitted is an example of the sensing area.

[0114] Next, the information processing device 140 determines the region of interest on the image P1 based on the map information and the positioning information (S20). Specifically, the region of interest determination unit 141 determines the region of interest. Here, the operation of the region of interest determination unit 141 to determine the region of interest is described in detail in detail. Figures 4 to 7 Provide explanation.

[0115] Figure 4 : is a flowchart showing an example of the operation of the region of interest determination unit 141 according to the present embodiment.

[0116] like Figure 4 As shown, first, the region of interest determination unit 141 obtains positioning information and map information (S21). Specifically, the region of interest determination unit 141 obtains positioning information including the position and direction (traveling direction) of the mobile object 100 as positioning information from the positioning unit 120, and obtains map information from the map storage unit 130.

[0117] Figure 6 FIG. 1 is a diagram showing an example of map information M1 obtained by the region of interest determination unit 141 according to the present embodiment. The map information M1 is information corresponding to the image P1. For example, based on the map information M1 and the positioning information, Figure 6 Determine each area of ​​the image. And, Figure 6 The "driveway", "sidewalk", "park", and "private land" shown are examples of attributes of the region of interest. In addition, each region may be determined using a region segmentation method (eg, semantic segmentation) instead of or in addition to the above method.

[0118] Furthermore, the region of interest determination unit 141 determines the region of interest using the map information M1 (S22). Specifically, the region of interest determination unit 141 determines the position (e.g., coordinate data) of the region of interest on the image P1 using the map information M1. Figure 5 In the case of the image P1 shown, the sidewalk portion in the image P1 is determined as the focus area using the road information included in the map information M1 corresponding to the image P1.

[0119] Figure 7 is Figure 5 The image P1 overlaps the region of interest. Figure 7 The regions of interest in the image P1 shown are the sidewalk portions R1 and R2. Figure 7 It is shown that the region of interest (sidewalk portions R1 and R2) is not written in the image P1 itself, but is handled in the form of coordinate data of the image P1 and the region of interest associated therewith.

[0120] Refer again Figure 4 , then, it is determined whether there is a region of interest in the image P1 (S23). The region of interest determination unit 141, when the region of interest exists ("Yes" in S23), outputs the image data corresponding to the image P1 and the information of the region of interest to the template rectangle setting unit 142 and the input image setting unit 143 (S24). The region of interest determination unit 141 may also, for example, output the information of the region of interest to the template rectangle setting unit 142, and output the image data and the information of the region of interest to the input image setting unit 143. The information of the region of interest includes, for example, coordinate data of the region of interest associated with the image P1. The information of the region of interest may also include attributes of the region of interest. Furthermore, when the region of interest is of one type, the information of the region of interest output to the template rectangle setting unit 142 may include the coordinate data of the region of interest, if the information indicating the existence of the region of interest is included.

[0121] Furthermore, when the attention region does not exist (No in S23), the attention region determination unit 141 outputs the image data to the input image setting unit 143 (S25). Furthermore, when the attention region determination unit 141 is No in step S23, it may output information indicating that the attention region does not exist to the template rectangle setting unit 142.

[0122] The template rectangle setting unit 142, when obtaining the information of the region of interest, outputs the information of the template rectangle to be used to the object detection unit 144 in a manner of using the template rectangle corresponding to the obtained information of the region of interest (for example, the template rectangle included in at least one of the general template rectangle group 142a and the sidewalk template rectangle group 142b). Furthermore, the input image setting unit 143, when obtaining the image data and the information of the region of interest, outputs the obtained image data and the information of the region of interest to the object detection unit 144.

[0123] Refer again Figure 3 , and the processing after step S30 is performed. Moreover, the processing after step S30 is the processing performed by the object detection unit 144. For example, in the processing of steps S30 to S80, the object detection unit 144 inputs the input information including the input image and the template rectangle for object detection into the object detection model to perform. Moreover, in this embodiment, the detection result of the object detection unit 144 is recorded as a detection rectangle, but the detection result is not limited to a rectangular area. The detection result can also be, for example, a polygonal area. Moreover, steps S30 to S80 are an example of a step of causing the object detection model to perform object detection processing.

[0124] Furthermore, the object detection unit 144 may perform predetermined image processing on the image data (camera image captured by the optical sensor 110) obtained from the input image setting unit 143 between steps S20 and S30. The object detection unit 144 may also perform processing to change the image data to an image with a resolution used for object detection. The object detection unit 144 may also, for example, perform processing to reduce the image size in order to reduce the amount of processing. Furthermore, the processing may be performed, for example, based on instructions from the input image setting unit 143. The image with changed resolution is an example of an input image. Furthermore, the processing may not be performed. In this case, the image obtained from the input image setting unit 143 is an example of an input image.

[0125] Furthermore, the object detection unit 144 detects object candidate rectangles from the input image (S30). The object detection unit 144 selects candidate regions by, for example, grouping similar regions in pixel units. That is, the object detection unit 144 combines regions having similar features to extract an object candidate as an object candidate rectangle. Specifically, the object detection unit 144 combines regions having similar features to extract an object candidate as an object candidate rectangle. The object detection unit 144 extracts object candidates as rectangle information from the input image. The object detection unit 144 extracts, for example, tens of thousands of object candidate rectangles from one input image.

[0126] Next, when the region of interest exists (Yes in S40), the object detection unit 144 performs a process of selecting object candidate rectangles using the general template rectangle group 142a and the sidewalk template rectangle group 142b (S50). In other words, the object detection unit 144 performs object candidate rectangle suggestion using the general template rectangle group 142a and the sidewalk template rectangle group 142b. Here, object candidate rectangle suggestion refers to extracting object candidate rectangles for executing object detection processing from the object candidate rectangles detected in step S30 using the template rectangle. In other words, object candidate rectangle suggestion refers to extracting objects for processing after step S70 from the object candidate rectangles detected in step S30. Object candidate rectangle suggestion refers to extracting object candidate rectangles that are more likely to be the desired object from the object candidate group by inferring object candidates based on the object candidate and the template rectangle multiple of the constant. Specifically, in step S50, object candidate rectangles that are close in size and aspect ratio to the template rectangle are extracted.

[0127] The object detection unit 144 extracts an object candidate rectangle using the sidewalk template rectangle group 142b in the sidewalk portions R1 and R2 as the target area. That is, in order to detect pedestrians more accurately in the sidewalk portions R1 and R2 as the target area, the sidewalk template rectangle group 142b that can easily detect people is used. When at least part of the coordinate data of the object candidate rectangle among the plurality of object candidate rectangles includes the coordinate data of the sidewalk portions R1 and R2, the object candidate rectangle is extracted using the sidewalk template rectangle group 142b.

[0128] Furthermore, the object detection unit 144 extracts object candidate rectangles using the general template rectangle group 142a in the area not in the focus area (for example, the area other than the sidewalk portions R1 and R2). That is, in order to detect pedestrians and vehicles evenly in the area not in the focus area, the general template rectangle group 142a is used, which can detect a variety of objects more easily than the sidewalk template rectangle group 142b.

[0129] Then, the object detection unit 144 selects a template rectangle for the sidewalk portions R1 and R2 and a template rectangle for other areas according to the instruction of the template rectangle setting unit 142 .

[0130] In step S50, the object detection unit 144 extracts thousands (for example, two thousand) object candidate rectangles from tens of thousands (for example, 10,000) of object candidate rectangles. In step S50, the object candidate rectangles are not limited to being selected as thousands, but may be selected as the number of object candidate rectangles that the object detection unit 144 can perform object detection processing at a predetermined processing speed. The predetermined processing speed is a processing speed that does not cause safety problems when performing autonomous driving, for example, 30 fps (frame per second).

[0131] If the target region does not exist (No in S40 ), the object detection unit 144 performs a process of selecting object candidate rectangles using the general template rectangle group 142 a ( S60 ).

[0132] Next, the object detection unit 144 classifies the extracted object candidate rectangles (S70). Specifically, the object detection unit 144 infers the reliability of the estimated level (person, vehicle, bicycle, etc.) for the extracted object candidate rectangles and classifies them. The object detection unit 144 infers the reliability of the object candidate rectangles for the object candidate rectangles existing in the sidewalk parts R1 and R2 using the sidewalk template rectangle group 142b and classifies them. The inference of reliability means, for example, calculating the possibility (probability, etc.) of each estimated level. The classification means, for example, calculating the probability of each estimated level in such a way that the total of all estimated levels becomes 100% based on the inference result of the reliability. The object detection unit 144 classifies, for example, 60% for people, 25% for vehicles, and 15% for bicycles, which total 100%. Furthermore, the object detection unit 144 infers the reliability of the object candidate rectangles for the object candidate rectangles existing outside the sidewalk parts R1 and R2 using the general template rectangle group 142a and classifies them.

[0133] Furthermore, the object detection unit 144 performs threshold processing on the reliability of the object candidate rectangle and detects the object detection rectangle (S80). In other words, the object detection unit 144 sets the rectangle having a reliability greater than a predetermined threshold value from among the object candidate rectangles as the object detection rectangle. In the example described above, if the predetermined threshold value is 50%, the object candidate indicated by the object candidate rectangle is detected as a person. Furthermore, the predetermined threshold value is not limited to 50%, and can be set, for example, from a value greater than 50%.

[0134] Then, the object detection unit 144 outputs the detection result (S90). The object detection unit 144, for example, performs processing to adapt the object detection rectangle to the original camera image (for example, image P1), and outputs the processed camera image. The object detection unit 144, for example, outputs Figure 8 Image P1 is shown. Figure 8 This is a diagram in which the detection result is superimposed on the image P1 according to the present embodiment.

[0135] like Figure 8 As shown in FIG. 1 , as a process adapted to the image P1 (camera image), the object detection rectangle H1 is superimposed on the image P1. The object detection unit 144 may also Figure 8 The image P1 shown is displayed on, for example, a display unit provided in the moving object 100 .

[0136] Furthermore, in this embodiment, the number of template rectangles included in the general template rectangle group 142a is the same as the number of template rectangles included in the sidewalk template rectangle group 142b. Therefore, the processing amount of the object detection unit 144 in steps S50, S70, and S80 is equal to the processing amount of the object detection unit 144 in steps S60, S70, and S80.

[0137] Furthermore, in the above-described embodiment, an example is described in which the range of constant multiples of the template rectangle used for processing object candidate suggestions from the object candidate group is not logically limited, but the present invention is not limited thereto. Specifically, the range of constant multiples of the template rectangle for the object candidate group may be dynamically determined according to the distance from the optical sensor 110. For example, in the case of Fig. 9 It is shown that when the distance can be measured based on the map information M1 (for example, the distance from the optical sensor 110), the object detection unit 144 may also change the range of a constant multiple of the template rectangle used in the object candidate rectangle suggestion process using the measured distance. For example, the object detection unit 144 sets the size of the object template rectangle according to the distance information obtained from the map information. Moreover, here, the setting of the size means changing the size without changing the aspect ratio of the template rectangle. Furthermore, it is not limited to obtaining the distance information from the map information. For example, the distance information may also be obtained from various sensors possessed by the mobile body 100. For example, in the case where the optical sensor 110 is implemented by LIDAR, the distance information may also be information obtained by the optical sensor 110. Fig. 9 This is a diagram in which distance information is superimposed on image data according to the present embodiment.

[0138] like Fig. 9 As shown, a constant multiple of the maximum value may be set according to the distance to be measured. When the distance to the optical sensor 110 is 3 m, a constant multiple greater than that of the maximum value in the case of 6 m may be used.

[0139] Furthermore, for example, a dedicated processor may refer to a memory storing information of an object detection model and execute the information processing method.

[0140] Moreover, when the object detection model is a neural network type inferencer, the structure of the detection neural network may be switched according to at least one of the number of regions of interest and the aspect ratio of the regions of interest.

[0141] As described above, the information processing device 140 according to the present embodiment includes the attention region determination unit 141 and the object detection unit 144. The attention region determination unit 141 obtains the image P1 captured by the optical sensor 110, the position information of the moving object 100 equipped with the optical sensor 110, and the map information. The attention region determination unit 141 determines the attention region in the image P1 using the position information and the map information. Furthermore, the object detection unit 144 determines the input information of the object detection model according to the attention region, and uses the input information to make the object detection model perform object detection processing. In other words, the information processing device 140 involved in this embodiment is configured to, for example, obtain image P1 (an example of sensor data) from the optical sensor 110, obtain position information of the moving body 100 (an example of an object) equipped with the optical sensor 110, obtain map information, and use the obtained position information and map information to determine a focus area in image P1 (an example of a specific sensor data area) corresponding to a sidewalk area (an example of a specific area) in the range where image P1 is captured (an example of the sensor area of ​​the optical sensor 110), determine input information of the object detection model according to the focus area, and use the input information to cause the object detection model to perform object detection processing.

[0142] According to this structure, the input information of the object detection model is determined according to the region of interest. In this embodiment, the template rectangle input to the object detection model is determined according to the region of interest. The object detection model uses the template rectangle corresponding to the region of interest to perform object detection processing, thereby improving the object detection performance of the region of interest. Therefore, it is possible to improve the object detection performance while suppressing the increase in computing resources used for object detection. Furthermore, the increase in the processing amount of the object detection unit 144 is suppressed, so it is possible to suppress the reduction in the real-time performance of real-time object detection based on the image used by the autonomous driving car, and the increase in power consumption.

[0143] (Implementation Method 2)

[0144] Hereinafter, the information processing device according to the present embodiment will be described with reference to Figures 10 to 13B. In addition, the structure of the information processing device involved in this embodiment is the same as that of the information processing device 140 involved in Embodiment 1, and the description thereof is omitted. In addition, in this embodiment, the information processing device 140 only needs to include the template rectangle setting unit 142 and the input image setting unit 143. In addition, in the following, in the same processing as that of the information processing device 140 in Embodiment 1, the description thereof may be omitted or simplified.

[0145] [2-1. Operation of information processing device]

[0146] For the operation of the information processing device 140 according to this embodiment, refer to Figures 10 to 13B Provide explanation. Fig.10 This is a flowchart showing an example of the operation of the information processing device according to the present embodiment.

[0147] like Fig.10 As shown, first, the information processing device 140 obtains image data from the optical sensor 110 (S110). Figure 3 The step S10 shown is the same and the description thereof is omitted.

[0148] Next, the information processing device 140 determines the region of interest on the image based on the map information and the positioning information (S120). Specifically, the region of interest determination unit 141 determines the region of interest. The operation of the region of interest determination unit 141 to determine the region of interest is similar to that of the first embodiment. Figure 3 Step S20 (specifically, Figure 4 The same is true for steps S21 to S25, and the description is omitted.

[0149] The template rectangle setting unit 142 outputs information on the template rectangle to be used to the object detection unit 144 in a manner that uses a predetermined template rectangle (a predetermined template rectangle, for example, the general template rectangle group 142a) regardless of whether the image P1 includes the region of interest. Furthermore, if the information on the region of interest is obtained, the template rectangle setting unit 142 may output information on the template rectangle to be used to the object detection unit 144 in a manner that uses a template rectangle corresponding to the obtained information on the region of interest (for example, at least one of the general template rectangle group 142a and the template rectangle group 142b for sidewalks). The following describes an example in which the template rectangle setting unit 142 instructs the object detection unit 144 to use the general template rectangle group 142a regardless of whether the image P1 includes the region of interest.

[0150] The input image setting unit 143 instructs the object detection unit 144 to cut out the rectangle including the sidewalk portion as the target area, and to generate the input image of the entire area of ​​the application image and the target area.

[0151] The object detection unit 144 generates an input image according to the instruction from the input image setting unit 143 (S130). The object detection unit 144 performs a predetermined image processing on the image P1 to generate the input image. The image processing of step S130 includes a process of reducing the image size in order to reduce the processing amount of the object detection unit 144. In addition, the processing after step S130 is the processing performed by the object detection unit 144.

[0152] Here, regarding the generation of the input image by the object detection unit 144, refer to Fig.11 Provide explanation. Fig.11 1 is a flowchart showing the operation of generating an input image by the object detection unit 144 according to the present embodiment.

[0153] like Fig.11 It is shown that when there is a region of interest in the image ("Yes" in S131), the image data (image) is reduced (S132). In step S132, the region of interest (in this embodiment, the sidewalk portion) and the area outside the region of interest (hereinafter also referred to as other areas) included in the image are reduced at different reduction rates. The object detection unit 144 may also, for example, reduce the region of interest included in the image and the entire area including the region of interest and other areas at different reduction rates. Specifically, the object detection unit 144 sets the reduction rate of the region of interest to be smaller than the reduction rate of the entire area or other areas.

[0154] For the image processing in step S132, refer to Fig. 12A as well as Fig. 12B Provide explanation. Fig. 12A FIG. 1 is a diagram showing an image P1 before reduction when the region of interest is included according to the present embodiment. Fig. 12B is a diagram showing a reduced image P11 including a region of interest according to the present embodiment. The image P11 is an example of an input image. Fig. 12A as well as Fig. 12B FIG. 4 shows an example of reducing an image P1 having an image size of 1280 pixels×720 pixels to an image P11 having an image size of 600 pixels×600 pixels. Fig. 12A The image P1 shown is generated Fig. 12B The image P11 is shown, thereby adjusting the image size.

[0155] like Fig. 12A as well as Fig. 12BIt is shown that the object detection unit 144 cuts out the cutout region R11 including the sidewalk portion R1 as the target region and the cutout region R22 including the sidewalk portion R2 from the image P1, and generates the image P11 based on the cutout regions R11 and R22 and the entire region R33 (for example, the image P1) including the cutout regions R11 and R22. That is, the image P11 is a composite image of the image including the target region (the cutout regions R11 and R22) and the image including the entire region R33. The cutout region R11 is not limited to a rectangle if it includes the sidewalk portion R1, and may be, for example, a polygon. The cutout region R22 is not limited to a rectangle if it includes the sidewalk portion R2, and may be, for example, a polygon. Moreover, the entire region R33 is an example of a sensing region.

[0156] The object detection unit 144, for example, reduces the entire region R33 to an image size of 600 pixels × 300 pixels, and reduces the cutout regions R11 and R22 to an image size of 600 pixels × 300 pixels. Moreover, as for the image size, if the reduction rate of the cutout regions R11 and R22 is smaller than the reduction rate of the entire region R33, it is not limited to the above-mentioned size. Moreover, the image size may be determined so that the reduction rate of the cutout regions R11 and R22 is smaller than the reduction rate of the entire region R33. The portion showing the entire region R33 in the image P11 described above has a lower resolution than the portion showing the cutout regions R11 and R22. That is, the image quality of the region of interest is better than that of other regions. Moreover, the images of both the cutout regions R11 and R22 (an example of a specific sensor data region) and other regions other than the cutout regions R11 and R22 are examples of the entire image. For example, the image of the entire region R33 is an example of the entire image.

[0157] Furthermore, the image size of the image P11 is different from the image obtained by reducing the image P1 when the target area does not exist in the image P1 (for example, Fig. 13B is equal to image P22).

[0158] Refer again Fig.11 If there is no region of interest in the image P1 (No in S131), the image data is uniformly reduced (S133). In step S133, the image data is reduced at a uniform reduction rate.

[0159] For the image processing in step S133, refer to Fig.13A as well as Fig. 13B Provide explanation. Fig.13A FIG. 2 is a diagram showing an image P2 before reduction when the region of interest is not included according to the present embodiment. Fig. 13Bis a diagram showing a reduced image P22 when the region of interest is not included according to the present embodiment. The image P22 is an example of an input image. Fig.13A as well as Fig. 13B An example is shown in which the image P2 having an image size of 1280 pixels×720 pixels is changed to an image P22 having an image size of 600 pixels×600 pixels. Fig.13A The range of the captured image P2 shown is an example of a sensing area.

[0160] like Fig.13A as well as Fig. 13B It is shown that the object detection unit 144 generates the image P22 by reducing the entire image P2. The image P22 has the same image quality throughout.

[0161] Refer again Fig.11 , and the processing after step S140 is performed. Moreover, the processing after step S140 is the processing performed by the object detection unit 144. For example, the object detection unit 144 inputs the input information including the input image and the template rectangle for object detection processing into the object detection model to perform the processing of steps S140 to S170. In this embodiment, the input image has features. The object detection unit 144 generates an image (for example, image P11) with different resolutions for the area of ​​interest and the area outside the area of ​​interest, and inputs the image as an input image to the object detection model. The input image may also be, for example, a synthesized image.

[0162] The object detection unit 144 detects object candidate rectangles from the input image (e.g., image P11 or P22) (S140). The object detection unit 144 detects object candidate rectangles from, for example, image P11 which is a composite image. Moreover, the object detection unit 144 is not limited to extracting object candidate rectangles from composite images. The object detection unit 144 may extract object candidate rectangles from an image whose resolution of the area outside the focus area is lower than that of the focus area. Furthermore, step S140 is different from that of the first embodiment. Figure 3 The same is true for step S30, and the description thereof is omitted.

[0163] Next, the object detection unit 144 performs the process of selecting a candidate rectangle of an object using the template rectangle (S150) to detecting an object detection moment (S170). When the object detection unit 144 obtains information indicating the use of the general template rectangle group 142a from the template rectangle setting unit 142, it performs the processes of steps S150 to S170 using the template rectangle included in the general template rectangle group 142a. For example, the image P11 has a high resolution of the region of interest compared to the case where the image P1 including the region of interest is uniformly reduced as in step S133. In addition, the image sizes of the images P11 and P22 are equal. Therefore, by using the image P11, it is possible to improve the object detection performance of the region of interest while suppressing the increase in the processing amount of the object detection unit 144. In this embodiment, a person in the region of interest can be detected with high accuracy.

[0164] Furthermore, when the object detection unit 144 receives an instruction from the template rectangle setting unit 142 to use the sidewalk template rectangle group 142b for the region of interest (sidewalk portions R1 and R2) and to use the general template rectangle group 142a for other regions or the entire region, the object detection unit 144 performs the processing of steps S150 to S170 using the general template rectangle group 142a and the sidewalk template rectangle group 142b. Thus, the object detection performance of the region of interest can be improved while suppressing an increase in the processing amount of the object detection unit 144. In this embodiment, a person in the region of interest can be detected with high accuracy.

[0165] Furthermore, in this embodiment, the image size of the input image is the same whether the sensing area includes the region of interest or not. Therefore, the amount of processing by the object detection unit 144 in steps S150 to S170 is the same whether the sensing area includes the region of interest or not.

[0166] Then, the object detection unit 144 outputs the detection result (S180). The object detection unit 144, for example, performs processing to adapt the object detection rectangle to the original camera image (for example, image P1), and outputs the processed camera image. Specifically, the object detection unit 144 restores the deviation of the coordinates of the focus area and the entire area of ​​the input image to the coordinate position of the original image data (image P1), and then outputs the object detection rectangle to the image P1. The object detection unit 144, for example, outputs Figure 8 Image P1 is shown.

[0167] As described above, the region of interest determination unit 141 of the information processing device 140 according to the present embodiment obtains a sensor image from the optical sensor 110 including an image sensor. Furthermore, the input information includes an image based on the sensor image, and the resolution of other regions other than the region of interest of the image is lower than the resolution of the region of interest. For example, the input information may include a composite image of an image of the region of interest cut out from the measured image based on the sensor image and the entire image, and the resolution of the entire image is lower than the resolution of the image of the region of interest.

[0168] According to this structure, the information processing device 140 determines the input information of the object detection model according to the region of interest. In this embodiment, the input image input to the object detection model is determined according to the region of interest. The object detection model uses the high-resolution image of the region of interest to perform object detection processing, thereby improving the object detection performance of the region of interest. Therefore, it is possible to improve the object detection performance while suppressing the increase in computing resources for detecting objects. Furthermore, when both the image of the region of interest and the entire image are input to the object detection model, it is possible to suppress the occurrence of erroneous detection or detection omission of objects caused by clipping of the region of interest.

[0169] (Other Embodiments)

[0170] As mentioned above, although the information processing device etc. which concern on embodiment were demonstrated based on embodiment, this disclosure is not limited to this embodiment.

[0171] Therefore, the components described in the drawings and detailed descriptions include not only the components necessary to solve the problem, but also components that are not necessary to solve the problem and are used to illustrate examples of the technology. Therefore, it should not be considered that these unnecessary components are necessary when they are described in the drawings and detailed descriptions.

[0172] In addition, various modifications that can be conceived by those skilled in the art to the embodiments, or embodiments achieved by arbitrarily combining components and functions of the embodiments within the scope not departing from the spirit of the present disclosure are also included in the present disclosure.

[0173] For example, in the above-described embodiment, an example in which the information processing device is mounted on a mobile body is described, but the present invention is not limited thereto. The information processing device may also be a server device installed outside the mobile body. The information processing device may also receive image data, location information, etc. from the mobile body by wireless communication, detect an object based on the received information and map information, and send the detection result to the mobile body.

[0174] Furthermore, in the above-described embodiment, an example in which the object detection unit is a neural network type detector is described, but the present invention is not limited thereto. For example, the object detection unit may be a pattern matching type detector.

[0175] Furthermore, the template rectangle setting unit of the embodiment may set the template rectangle to be used for each specified area of ​​the image. The template rectangle setting unit may also, when the area of ​​interest is the sky and it is known from the configuration and posture of the optical sensor that there is a sky in the upper area of ​​the image as the area of ​​interest, for example, use a template rectangle corresponding to the sky in the upper half of the image (for example, a template rectangle that can easily detect a flying object such as a drone).

[0176] Furthermore, in the above-described embodiment, the template rectangles (e.g., the general template rectangle group and the sidewalk template rectangle group) are determined according to the region of interest, but may also be determined according to the external environment of the mobile body. The external environment includes at least one of weather information and time information of the road on which the mobile body is traveling. For example, the sidewalk template rectangle group may include a sidewalk template rectangle group for sunny days and a sidewalk template rectangle group for rainy days, and the template rectangle setting unit may determine which sidewalk template rectangle group to use based on the sensing results of various sensors (e.g., wiper sensors that detect the operation of wipers) possessed by the mobile body. Furthermore, the sidewalk template rectangle group for rainy days includes, for example, a template rectangle that can easily identify a person holding an umbrella.

[0177] Furthermore, in the above-described embodiment, an example is described in which the number of template rectangles included in the general template rectangle group and the number of template rectangles included in the sidewalk template rectangle group is equal, but the present invention is not limited thereto. The number of template rectangles included in the sidewalk template rectangle group may be less than the number of template rectangles included in the general template rectangle group. The sidewalk template rectangle group may not include, for example, template rectangles other than people for detecting vehicles, etc. In this way, the processing amount of the object detection unit can be further reduced.

[0178] Furthermore, in the above-described embodiment, it is described that the information processing device involved in the embodiment is applicable to an example of an autonomous driving vehicle, but the present invention is not limited thereto. For example, the information processing device may also be applicable to indoor robots and flying objects. Furthermore, the information processing device may detect objects using indoor positioning methods (short-range wireless communications such as Wi-Fi (registered trademark), BLE (Bluetooth (registered trademark) Low Energy), beacons, etc.) instead of using optical sensors to detect objects.

[0179] Furthermore, the order of the plurality of processes described in the above embodiment is an example, and the order of the plurality of processes may be changed, or the plurality of processes may be executed in parallel.

[0180] Furthermore, each component described in the embodiment can also be implemented as software or as a typical integrated circuit LSI. They can be monolithic respectively or in a way that includes part or all of them. Multiple chips can also be concentrated in one device or be provided in multiple devices. Here, for LSI, there will be cases called system LSI, super LSI or extra-large LSI depending on the degree of integration. Moreover, the method of integrated circuitization is not limited to LSI, and can also be implemented as a dedicated circuit or a general-purpose processor. It is also possible to use an FPGA (Field Programmable Gate Array) that can be programmed after manufacturing LSI, or a reconfigurable processor that can reconfigure the connection and setting of the circuit unit inside the LSI. Furthermore, if an integrated circuit technology that replaces LSI appears due to the advancement of semiconductor technology or other derived technologies, it is of course possible to use this technology to integrate functional blocks.

[0181] Furthermore, the division of the functional blocks in the block diagram is an example, and multiple functional blocks may be implemented as one functional block, one functional block may be divided into multiple blocks, or a part of the functions may be transferred to other functional blocks. Furthermore, the functions of multiple functional blocks having similar functions may be processed in parallel or in time division by a single hardware or software.

[0182] Furthermore, the technology disclosed in the present invention may also be the program, or a non-temporary computer-readable recording medium recording the program. Furthermore, of course, the program may also be distributed via a transmission medium such as the Internet. For example, the program and the digital signal composed of the program may also be transmitted via an electrical communication line, a wireless or wired communication line, a network represented by the Internet, data broadcasting, etc. Furthermore, the program and the digital signal composed of the program may also be recorded on a recording medium for transmission, or transmitted via a network, etc., so as to be executed by another independent computer system.

[0183] Furthermore, the numbers such as ordinal numbers and quantities used are examples shown to specifically describe the technology of the present disclosure, and the present disclosure is not limited to the numbers shown as examples. Furthermore, the connection relationship between the constituent elements is an example shown to specifically describe the technology of the present disclosure, and the connection relationship for realizing the functions of the present disclosure is not limited to this.

[0184] The present disclosure can be used in information processing devices that perform object detection processing on sensor data, and can also be applied to image recognition systems, automatic operation systems, and the like.

Claims

1. An information processing device, the information processing device being configured as follows: Get sensor data from optical sensors, obtaining position information of an object equipped with the optical sensor, Get map information, determining a specific sensing data area in the sensing data corresponding to a specific area in the sensing area of ​​the optical sensor using the position information and the map information, determining whether the specific sensor data area exists in the sensor data, In a case where the specific sensor data area exists in the sensor data, first input information of an object detection model including a first object template rectangle and a second object template rectangle is determined, and the object detection model is caused to perform object detection processing using the first input information, wherein the first object template rectangle is used to perform object detection corresponding to the specific sensor data area, and the second object template rectangle is used to perform object detection corresponding to an area other than the specific sensor data area and not including the specific sensor data area. When the specific sensor data area is not included in the sensor data, the second input information of the object detection model is determined to include only the first object template rectangle and the second object template rectangle in the second object template rectangle, and the object detection model is enabled to perform object detection processing using the second input information.

2. The information processing device according to claim 1, When the sensor data includes the specific sensor data area, in the object detection model, the object to be subjected to object detection processing is selected by using the first object template rectangle and the second object template rectangle for object detection. When the specific sensing data area does not exist in the sensing data, the object detection model selects an object for performing object detection processing using the second object template rectangle used for object detection.

3. The information processing device according to claim 2, The first object template rectangle is set according to the attribute of the specific sensor data area.

4. The information processing device according to claim 2 or 3, obtaining distance information about the sensing data, The sizes of the first object template rectangle and the second object template rectangle are set according to the obtained distance information.

5. The information processing device according to claim 1, The optical sensor comprises an image sensor, The sensor data includes a sensor image, which is an image obtained from the image sensor. The first input information and the second input information include an image based on the sensor image, and a resolution of an area other than the specific sensor data area of ​​the image is lower than a resolution of the specific sensor data area.

6. The information processing device according to claim 1, The optical sensor comprises an image sensor, The sensor data includes a sensor image, which is an image obtained from the image sensor. The first input information and the second input information include a composite image of an image of the specific sensing data area cut out from the sensing image and a whole image, wherein the composite image is an image based on the sensing image, and the whole image is an image including both the specific sensing data area and other areas other than the specific sensing data area. The resolution of the entire image is lower than that of the image of the specific sensing data area.

7. An information processing method, Using the processor, Get sensor data from optical sensors, obtaining position information of an object equipped with the optical sensor, Get map information, determining a specific sensing data area in the sensing data corresponding to a specific area in the sensing area of ​​the optical sensor using the position information and the map information, determining whether the specific sensor data area exists in the sensor data, When the sensor data includes the specific sensor data area, determining first input information of an object detection model including a first object template rectangle and a second object template rectangle, The object detection model is caused to perform object detection processing using the first input information, wherein the first object template rectangle is used to perform object detection corresponding to the specific sensing data area, and the second object template rectangle is used to perform object detection corresponding to an area other than the specific sensing data area and excluding the specific sensing data area. In the case where the specific sensor data area is absent in the sensor data, determining second input information of the object detection model including only the first object template rectangle and the second object template rectangle of the second object template rectangle, The object detection model is caused to perform object detection processing using the second input information.

8. A recording medium, A program for causing a computer to execute the information processing method according to claim 7 is recorded.

Images