Information processing device, information processing method, and program product

Abstract

An information processing apparatus includes an input device and a controller. A captured image captured by a camera is input to the input device, the captured image including distance information for each pixel. The controller generates a transformed captured image by transforming coordinate pairs of individual pixels of the captured image based on the amount of movement of the camera or a mobile object on which the camera is mounted. Furthermore, the controller associates the coordinate pairs of pixels of the transformed captured image with the coordinate pairs of pixels of a post-movement captured image captured by the camera at the position after the movement, and identifies unassociated pixels included in the post-movement captured image that are not associated with pixels of the transformed captured image.

Description

Technical Field

[0001] The present technology relates to an information processing device, an information processing method, and a program applied to recognize an object in a captured image. Background Art

[0002] There are technologies for detecting a predetermined object region from an image.

[0003] Patent document 1 indicated below discloses an obstacle detector that detects obstacles located around a moving vehicle based on a difference image, wherein the difference image is based on the difference between a previous frame image and a reference frame image among frame images around the vehicle, the reference frame image being acquired at a reference time point, and the previous frame image being acquired at a time point before the reference time point.

[0004] Patent document 2 indicated below discloses an object detector that uses at least one reference image and a target image among a plurality of captured images to detect motion vectors of portions of a target image, calculates a difference image based on a difference between two images among the plurality of captured images, and detects an object region in which an object exists based on the motion vector and the difference image.

[0005] Reference List

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Publication No. 2018-97777

[0008] Patent Document 2: Japanese Patent Application Publication No. 2015-138319 Summary of the Invention

[0009] Technical issues

[0010] However, in each of the techniques disclosed in Patent Documents 1 and 2, objects are detected based on differences between entire images, which results in an increase in the amount of calculation. In addition, processing is often performed on an image similar to a previous image, which results in redundant processing.

[0011] In view of the above circumstances, an object of the present technology is to provide an information processing apparatus, an information processing method, and a program that enable elimination of redundant processing performed with respect to captured images sequentially acquired during movement and reduction of the amount of calculation.

[0012] Solution to the problem

[0013] To achieve the above-mentioned purpose, an information processing apparatus according to an embodiment of the present technology includes an input device and a controller. A captured image captured by a camera is input to the input device, and the captured image includes distance information for each pixel. The controller generates a transformed captured image obtained by transforming the coordinate pairs of each pixel of the captured image based on the amount of movement of the camera or a mobile body on which the camera is mounted. In addition, the controller associates the coordinate pairs of the pixels of the transformed captured image with the coordinate pairs of the pixels of the post-movement captured image captured by the camera at the position after the movement, and the controller identifies unassociated pixels included in the post-movement captured image that are not associated with the pixels of the transformed captured image.

[0014] Therefore, the information processing device recognizes pixels of the captured image after movement that are not associated with the pixels of the captured image, and this makes it unnecessary to perform new processing with respect to the associated pixels. This makes it possible to eliminate redundant processing of the captured images sequentially acquired during movement and reduce the amount of calculation.

[0015] The controller can perform recognition processing to identify the attributes of unassociated pixels in the captured image after movement, and can project the results of the recognition processing onto associated pixels or an area including associated pixels in the captured image after movement, and the recognition processing is performed relative to pixels included in the captured image corresponding to the associated pixels or the area including the associated pixels.

[0016] Therefore, the information processing device can project the results of the recognition processing performed on the captured image before the movement onto the captured image after the movement, with respect to the associated pixels of the captured image after the movement. This makes it possible to omit the recognition processing performed on the associated pixels and reduce the amount of calculation.

[0017] The controller may generate a map obtained by associating coordinate pairs of pixels of the captured image after the movement with coordinate pairs of pixels of the captured image in order to perform the projection.

[0018] Therefore, the information processing apparatus can easily project the result of recognition performed with respect to the captured image before the movement onto the captured image after the movement by using the generated map.

[0019] The controller can transform the captured image into three-dimensional point cloud data based on distance information of each pixel, can generate moving point cloud data obtained by performing a transformation relative to the three-dimensional point cloud data based on the movement amount, and can project the moving point cloud data onto the image plane to generate a transformed captured image.

[0020] Therefore, the information processing device converts the captured image into three-dimensional point cloud data based on the distance information, and converts the three-dimensional point cloud data into a plane image after the shift. Therefore, the information processing device can accurately identify the corresponding pixels.

[0021] The controller may set a priority for performing the recognition process according to positions of unassociated pixels in the captured image after the movement.

[0022] Therefore, the information processing device sets the frequency of executing the recognition process according to the position of the region, such as setting the frequency of executing the recognition process for the region in the center of the captured image to be higher than that for the region in the end portion of the captured image. This makes it possible to reduce the amount of calculation.

[0023] The controller may set a priority for executing the recognition process for each unassociated pixel according to a position of the unassociated pixel in the captured image after movement and according to a moving speed of the moving object.

[0024] Therefore, the information processing device can cope with changes in important areas due to changes in movement speed, such as, during high-speed movement, setting the frequency of performing recognition processing for the area in the central part of the image to be higher than that for the area in the end part of the image, and during low-speed movement, setting the frequency of performing recognition processing for the area in the central part of the image to be lower than that for the area in the end part of the image.

[0025] The controller may set a priority for performing the recognition process for each unassociated pixel according to the distance information of the unassociated pixel.

[0026] Therefore, the information processing device sets the frequency of executing the recognition process according to the distance, such as setting the frequency of executing the recognition process higher for an area close to the camera than for an area located far from the camera. This makes it possible to reduce the amount of calculation.

[0027] An image processing method according to another embodiment of the present technology includes:

[0028] Acquire a captured image captured by a camera, the captured image including distance information of each pixel;

[0029] generating a transformed captured image obtained by transforming coordinate pairs of respective pixels of the captured image based on an amount of movement of the camera or a mobile object on which the camera is mounted;

[0030] associating coordinate pairs of pixels of the transformed captured image with coordinate pairs of pixels of the moved captured image captured by the camera at the moved position; and

[0031] Unassociated pixels included in the moved captured image that are not associated with pixels of the transformed captured image are identified.

[0032] A program according to another embodiment of the present technology causes an information processing apparatus to execute a process including the following steps:

[0033] Acquire a captured image captured by a camera, the captured image including distance information of each pixel;

[0034] generating a transformed captured image obtained by transforming coordinate pairs of respective pixels of the captured image based on an amount of movement of the camera or a mobile object on which the camera is mounted;

[0035] associating coordinate pairs of pixels of the transformed captured image with coordinate pairs of pixels of the moved captured image captured by the camera at the moved position; and

[0036] Unassociated pixels included in the moved captured image that are not associated with pixels of the transformed captured image are identified.

[0037] Advantageous Effects of the Invention

[0038] As described above, the present technology enables elimination of redundant processing performed with respect to captured images sequentially acquired during movement and reduction of the amount of calculation. However, the present technology is not limited to this effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] [ Figure 1 ] Figure 1 is a block diagram depicting an example of a schematic configuration of a vehicle control system according to an embodiment of the present technology.

[0040] [ Figure 2 ] Figure 2 1 is a diagram of assistance in explaining an example of installation positions of a vehicle exterior information detection portion and an imaging portion included in a vehicle control system.

[0041] [ Figure 3 ] Figure 3 The diagram illustrates the configuration of functional blocks included in the integrated control unit of the vehicle control system.

[0042] [ Figure 4 ] Figure 4 is a flowchart illustrating the flow of image recognition processing performed by the vehicle control system.

[0043] [ Figure 5 ] Figure 5 is a diagram for describing processing performed by a projection map generator and a semantic segmentation projection section included in the integrated control unit.

[0044] [ Figure 6 ] Figure 6 is a diagram for describing processing performed by the unobserved region setting section included in the integrated control unit.

[0045] [ Figure 7 ] Figure 7 Details of the processing performed by the projection map generator are illustrated.

[0046] [ Figure 8 ] Figure 8 is a flowchart illustrating the flow of processing performed by the projection map generator.

[0047] [ Figure 9 ] Figure 9 Details of the processing performed by the unobserved region setting section are illustrated.

[0048] [ Figure 10 ] Figure 10 : is a flowchart illustrating the flow of processing performed by the unobserved region setting section.

[0049] [ Figure 11 ] Figure 11 is a diagram for describing processing performed by the area attribute relationship determination section and the update priority map generator included in the integrated control unit.

[0050] [ Figure 12 ] Figure 12 is a diagram for describing a map integration process performed by the update priority map generator.

[0051] [ Figure 13 ] Figure 13 is a diagram for describing processing performed by a region semantic segmentation section included in the integrated control unit.

[0052] [ Figure 14 ] Figure 14 An example of setting the update frequency and the update area in the image recognition process performed by the vehicle control system according to the modification example of the present technology is illustrated.

[0053] [ Figure 15 ] Figure 15 An example of setting the update frequency and the update area in the image recognition process performed by the vehicle control system according to the modification example of the present technology is illustrated.

[0054] [ Figure 16 ] Figure 16 An example of setting the update frequency and the update area in the image recognition process performed by the vehicle control system according to the modification example of the present technology is illustrated.

[0055] [ Figure 17 ] Figure 17 An example of update region setting performed by the region semantic segmentation section in the vehicle control system according to a modification example of the present technology is illustrated.

[0056] [ Figure 18 ] Figure 18 is a diagram for describing processing performed by a region semantic segmentation section in a vehicle control system according to a modification example of the present technology. DETAILED DESCRIPTION

[0057] Now, embodiments of the present technology will be described below with reference to the drawings.

[0058] [Configuration of vehicle control system]

[0059] Figure 1 7000 is a block diagram illustrating an example of a schematic configuration of a vehicle control system 7000 as an example of a mobile body control system to which the technology according to an embodiment of the present disclosure can be applied. The vehicle control system 7000 includes a plurality of electric control units connected to each other via a communication network 7010. Figure 1 In the example depicted in FIG, a vehicle control system 7000 includes a drive system control unit 7100, a body system control unit 7200, a battery control unit 7300, a vehicle exterior information detection unit 7400, a vehicle interior information detection unit 7500, and an integrated control unit 7600. For example, a communication network 7010 connecting the plurality of control units to each other may be an in-vehicle communication network compliant with any standard such as a controller area network (CAN), a local interconnect network (LIN), a local area network (LAN), FlexRay (registered trademark), or the like.

[0060] Each control unit includes a microcomputer that performs arithmetic processing according to various programs; a storage unit that stores programs executed by the microcomputer, parameters for various operations, and the like; and a drive circuit that drives various control target devices. Each control unit also includes a network interface (I / F) for communicating with other control units via the communication network 7010; and a communication I / F for communicating with devices, sensors, and the like inside and outside the vehicle via wired or radio communication. Figure 1 The functional configuration of the integrated control unit 7600 illustrated in FIG. 7 includes a microcomputer 7610, a general-purpose communication I / F 7620, a dedicated communication I / F 7630, a positioning unit 7640, a beacon receiving unit 7650, a vehicle interior device I / F 7660, a sound / image output unit 7670, an in-vehicle network I / F 7680, and a storage unit 7690. Other control units similarly include a microcomputer, a communication I / F, a storage unit, and the like.

[0061] The drive system control unit 7100 controls the operation of devices related to the vehicle's drive system according to various programs. For example, the drive system control unit 7100 functions as a control device for generating driving force for the vehicle, such as an internal combustion engine or a drive motor, a driving force transmission mechanism for transmitting driving force to the wheels, a steering mechanism for adjusting the vehicle's steering angle, and a braking device for generating braking force for the vehicle. The drive system control unit 7100 may also function as a control device for an anti-lock braking system (ABS), an electronic stability control (ESC), and the like.

[0062] Drive system control unit 7100 is connected to vehicle state detection unit 7110. For example, vehicle state detection unit 7110 includes at least one of a gyro sensor for detecting the angular velocity of the axial rotational movement of the vehicle body, an acceleration sensor for detecting vehicle acceleration, and sensors for detecting the amount of accelerator pedal operation, the amount of brake pedal operation, the steering angle of the steering wheel, the engine speed, the wheel rotation speed, and the like. Drive system control unit 7100 uses the signals input from vehicle state detection unit 7110 to perform arithmetic processing and control the internal combustion engine, drive motor, electric power steering system, braking system, and the like.

[0063] The body system control unit 7200 controls the operation of various devices provided to the vehicle body according to various programs. For example, the body system control unit 7200 functions as a control device for a keyless entry system, a smart key system, power windows, and various lights such as headlights, backup lights, brake lights, turn signals, and fog lights. In this case, radio waves transmitted from a mobile device that replaces the key or signals from various switches may be input to the body system control unit 7200. The body system control unit 7200 receives these input radio waves or signals and controls the vehicle's door locks, power windows, lights, and other devices.

[0064] The battery control unit 7300 controls the secondary battery 7310, which serves as a power supply source for driving the motor, according to various programs. For example, information regarding battery temperature, battery output voltage, remaining battery charge, and the like is supplied to the battery control unit 7300 from the battery device including the secondary battery 7310. The battery control unit 7300 performs arithmetic processing using these signals and controls the temperature of the secondary battery 7310 or controls a cooling device provided to the battery device.

[0065] The vehicle exterior information detection unit 7400 detects information about the exterior of the vehicle including the vehicle control system 7000. For example, the vehicle exterior information detection unit 7400 is connected to at least one of the imaging unit 7410 and the vehicle exterior information detection unit 7420. The imaging unit 7410 includes at least one of a time-of-flight (ToF) camera, a stereo camera, a monocular camera, an infrared camera, or other cameras. For example, the vehicle exterior information detection unit 7420 includes at least one of an environmental sensor for detecting current atmospheric or weather conditions and a peripheral information detection sensor for detecting another vehicle, obstacles, pedestrians, etc. around the vehicle including the vehicle control system 7000.

[0066] For example, the environmental sensor may be at least one of a rainfall sensor for detecting rain, a fog sensor for detecting fog, a sunshine sensor for detecting the degree of sunshine, and a snow sensor for detecting snowfall. The peripheral information detection sensor may be at least one of an ultrasonic sensor, a radar device, and a LIDAR device (a light detection and ranging device, or a laser imaging detection and ranging device). Each of the imaging unit 7410 and the vehicle exterior information detection unit 7420 may be configured as an independent sensor or device, or may be configured as a device in which multiple sensors or devices are integrated.

[0067] Figure 2 The figure depicts examples of installation locations for the imaging unit 7410 and the vehicle exterior information detection unit 7420. For example, the imaging units 7910, 7912, 7914, 7916, and 7918 are located at least one of the front nose, side mirrors, rear bumper, and rear door of the vehicle 7900, or on the upper portion of the windshield within the vehicle interior. The imaging unit 7910 located on the front nose and the imaging unit 7918 located on the upper portion of the windshield within the vehicle interior primarily capture images in front of the vehicle 7900. The imaging units 7912 and 7914 located on the side mirrors primarily capture images from the sides of the vehicle 7900. The imaging unit 7916 located on the rear bumper or rear door primarily captures images from behind the vehicle 7900. The imaging unit 7918 located on the upper portion of the windshield within the vehicle interior primarily detects vehicles ahead, pedestrians, obstacles, signals, traffic signs, lanes, and the like.

[0068] By the way, Figure 2 The figure shows examples of the imaging ranges of the respective imaging units 7910, 7912, 7914, and 7916. Imaging range a represents the imaging range of imaging unit 7910, which is located at the front nose. Imaging ranges b and c represent the imaging ranges of imaging units 7912 and 7914, respectively, located at the side mirrors. Imaging range d represents the imaging range of imaging unit 7916, which is located at the rear bumper or rear door. For example, by superimposing the image data captured by imaging units 7910, 7912, 7914, and 7916, a bird's-eye view image of vehicle 7900, as viewed from above, can be obtained.

[0069] Vehicle exterior information detection units 7920, 7922, 7924, 7926, 7928, and 7930, located at the front, rear, sides, and corners of vehicle 7900, as well as the upper portion of the windshield within the vehicle interior, may be, for example, ultrasonic sensors or radar devices. For example, vehicle exterior information detection units 7920, 7926, and 7930, located at the front nose, rear bumper, rear door, and upper portion of the windshield within the vehicle interior, may be LIDAR devices. These vehicle exterior information detection units 7920 to 7930 are primarily used to detect preceding vehicles, pedestrians, obstacles, and the like.

[0070] return Figure 1 , will be described further. The vehicle external information detection unit 7400 causes the imaging unit 7410 to image the outside of the vehicle and receive the imaged image data. In addition, the vehicle external information detection unit 7400 receives detection information from the vehicle external information detection unit 7420 connected to the vehicle external information detection unit 7400. In the case where the vehicle external information detection unit 7420 is an ultrasonic sensor, a radar device, or a LIDAR device, the vehicle external information detection unit 7400 transmits ultrasonic waves, electromagnetic waves, etc., and receives information of the received reflected waves. Based on the received information, the vehicle external information detection unit 7400 can perform processing for detecting objects such as people, vehicles, obstacles, signs, characters on the road surface, etc., or processing for detecting the distance thereto. The vehicle external information detection unit 7400 can perform environmental recognition processing for identifying rainfall, fog, road conditions, etc. based on the received information. The vehicle external information detection unit 7400 can calculate the distance to objects outside the vehicle based on the received information.

[0071] In addition, based on the received image data, the vehicle exterior information detection unit 7400 can perform image recognition processing to identify people, vehicles, obstacles, signs, characters on the road, etc., or detect the distance thereto. The vehicle exterior information detection unit 7400 can subject the received image data to processing such as distortion correction and alignment, and combine image data imaged by multiple different imaging units 7410 to generate a bird's-eye view image or a panoramic image. The vehicle exterior information detection unit 7400 can perform viewpoint conversion processing using image data imaged by the imaging unit 7410 including different imaging units.

[0072] The vehicle interior information detection unit 7500 detects information about the interior of the vehicle. For example, the vehicle interior information detection unit 7500 is connected to a driver state detection unit 7510 that detects the driver's state. The driver state detection unit 7510 may include a camera that images the driver, a biosensor that detects the driver's biological information, a microphone that collects sounds inside the vehicle, and the like. For example, the biosensor is set in a seat surface, a steering wheel, etc., and detects the biological information of an occupant sitting in the seat or a driver holding the steering wheel. Based on the detection information input from the driver state detection unit 7510, the vehicle interior information detection unit 7500 can calculate the driver's fatigue level or the driver's concentration level, or can determine whether the driver is dozing off. The vehicle interior information detection unit 7500 can subject the audio signal obtained by collecting the sound to processing such as noise cancellation processing.

[0073] The integrated control unit 7600 controls the overall operation within the vehicle control system 7000 according to various programs. The integrated control unit 7600 is connected to the input unit 7800. The input unit 7800 is implemented by a device capable of inputting operations by the occupant, such as a touch panel, buttons, microphone, switch, lever, etc. Data obtained by voice recognition of voice input through the microphone can be supplied to the integrated control unit 7600. For example, the input unit 7800 can be a remote control device using infrared or other radio waves, or an externally connected device such as a mobile phone or personal digital assistant (PDA) that supports the operation of the vehicle control system 7000. The input unit 7800 can be, for example, a camera. In this case, the passenger can input information through gestures. Alternatively, data obtained by detecting the movement of a wearable device worn by the occupant can be input. In addition, the input unit 7800 can include, for example, an input control circuit that generates an input signal based on information input by the occupant using the input unit 7800 and outputs the generated input signal to the integrated control unit 7600. The occupants and the like input various data or give instructions for processing operations to the vehicle control system 7000 through the operation input unit 7800 .

[0074] The storage unit 7690 may include a read-only memory (ROM) that stores various programs executed by the microcomputer and a random access memory (RAM) that stores various parameters, operation results, sensor values, etc. In addition, the storage unit 7690 may be implemented by a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device.

[0075] The general communication I / F 7620 is a widely used communication I / F that mediates communications with various devices in the external environment 7750. The general communication I / F 7620 can implement cellular communication protocols such as Global System for Mobile Communications (GSM (registered trademark)), Worldwide Interoperability for Microwave Access (WiMAX (registered trademark)), Long Term Evolution (LTE (registered trademark)), and LTE-Advanced (LTE-A), or other wireless communication protocols such as wireless LAN (also known as Wireless Fidelity (Wi-Fi (registered trademark)) and Bluetooth (registered trademark). The general communication I / F 7620 can connect to devices (e.g., application servers or control servers) on an external network (e.g., the internet, a cloud network, or a company's private network) via a base station or access point, for example. Furthermore, for example, the general communication I / F 7620 can connect to a terminal (e.g., a terminal of a driver, a pedestrian, or a store, or a machine type communication (MTC) terminal) located near the vehicle using peer-to-peer (P2P) technology.

[0076] Dedicated communication I / F 7630 is a communication I / F that supports communication protocols developed for use in vehicles. Dedicated communication I / F 7630 can implement standard protocols such as, for example, Wireless Access in Vehicular Environments (WAVE), a combination of Institute of Electrical and Electronics Engineers (IEEE) 802.11p as a lower layer and IEEE 1609 as a higher layer, dedicated short-range communication (DSRC), or cellular communication protocols. Dedicated communication I / F 7630 typically implements V2X communication as a concept that includes one or more of vehicle-to-vehicle communication (V2V), road-to-vehicle communication (V2I), vehicle-to-residence communication (V2R), and pedestrian-to-vehicle communication (V2P).

[0077] The positioning unit 7640 performs positioning by, for example, receiving a global navigation satellite system (GNSS) signal from a GNSS satellite (e.g., receiving a GPS signal from a global positioning system (GPS) satellite) and generates position information including the latitude, longitude, and altitude of the vehicle. Incidentally, the positioning unit 7640 may recognize the current position by exchanging signals with a wireless access point, or may obtain position information from a terminal such as a mobile phone, a personal handyphone system (PHS), or a smartphone having a positioning function.

[0078] The beacon receiving unit 7650 receives, for example, radio waves or electromagnetic waves transmitted from a radio station installed on a road or the like, thereby obtaining information on the current position, congestion, closed roads, necessary time, etc. Incidentally, the function of the beacon receiving unit 7650 may be included in the above-mentioned dedicated communication I / F 7630.

[0079] The vehicle interior device I / F 7660 is a communication interface that mediates the connection between the microcomputer 7610 and various vehicle interior devices 7760 present in the vehicle. The vehicle interior device I / F 7660 can establish a wireless connection using a wireless communication protocol such as wireless LAN, Bluetooth (registered trademark), near field communication (NFC), or wireless universal serial bus (WUSB). In addition, the vehicle interior device I / F 7660 can establish a wired connection via a universal serial bus (USB), a high-definition multimedia interface (HDMI) (registered trademark), a mobile high-definition link (MHL), or the like via a connection terminal (if necessary, and a cable) not depicted in the figure. For example, the vehicle interior device 7760 may include at least one of an information device carried in or attached to the vehicle and a mobile device and a wearable device owned by the occupants. The vehicle interior device 7760 may also include a navigation device that searches for a route to an arbitrary destination. The vehicle interior device I / F 7660 exchanges control signals or data signals with these vehicle interior devices 7760.

[0080] In-vehicle network I / F 7680 is an interface that mediates communication between microcomputer 7610 and communication network 7010. In-vehicle network I / F 7680 transmits and receives signals and the like in accordance with a predetermined protocol supported by communication network 7010.

[0081] The microcomputer 7610 of the integrated control unit 7600 controls the vehicle control system 7000 according to various programs based on information obtained via at least one of the general communication I / F 7620, the dedicated communication I / F 7630, the positioning unit 7640, the beacon receiving unit 7650, the vehicle interior device I / F 7660, and the vehicle network I / F 7680. For example, the microcomputer 7610 can calculate control target values ​​for the driving force generating device, the steering mechanism, or the braking device based on the information obtained about the interior and exterior of the vehicle, and output control commands to the drive system control unit 7100. For example, the microcomputer 7610 can perform cooperative control to implement functions of an advanced driver assistance system (ADAS), including collision avoidance or shock absorption of the vehicle, follow-up driving based on following distance, vehicle speed maintenance driving, vehicle collision warning, vehicle lane departure warning, and the like. In addition, the microcomputer 7610 can perform cooperative control intended for autonomous driving, which controls the driving force generating device, steering mechanism, braking device, etc. based on the information obtained about the vehicle's surroundings, allowing the vehicle to travel autonomously without relying on driver operation, etc.

[0082] Based on information obtained via at least one of general communication I / F 7620, dedicated communication I / F 7630, positioning unit 7640, beacon receiver 7650, in-vehicle device I / F 7660, and in-vehicle network I / F 7680, microcomputer 7610 can generate three-dimensional distance information between the vehicle and objects such as surrounding structures and people, and generate local map information including information about the surroundings of the vehicle's current location. Furthermore, microcomputer 7610 can predict dangers such as vehicle collisions, the approach of pedestrians, or entering closed roads based on the obtained information, and generate warning signals. For example, the warning signal can generate a warning sound or illuminate a warning light.

[0083] The sound / image output unit 7670 transmits an output signal of at least one of sound and image to an output device capable of visually or auditorily notifying the occupants of the vehicle or the outside of the vehicle of information. Figure 1In the example of , an audio speaker 7710, a display unit 7720, and an instrument panel 7730 are illustrated as output devices. The display unit 7720 may, for example, include at least one of an on-board display and a head-up display. The display unit 7720 may have an augmented reality (AR) display function. The output device may be a device other than these devices, and may be another device such as headphones, a wearable device (such as a glasses-type display worn by an occupant), a projector, a lamp, etc. In the case where the output device is a display device, the display device visually displays the results obtained by various processes performed by the microcomputer 7610 or information received from another control unit in various forms such as text, images, tables, graphics, etc. In addition, in the case where the output device is an audio output device, the audio output device converts an audio signal consisting of reproduced audio data or sound data, etc. into an analog signal and audibly outputs the analog signal.

[0084] Incidentally, in Figure 1 In the example depicted in FIG, at least two control units connected to each other via the communication network 7010 can be integrated into one control unit. Alternatively, each individual control unit may include multiple control units. In addition, the vehicle control system 7000 may include another control unit not depicted in the figure. In addition, part or all of the functions performed by one of the control units described above may be assigned to another control unit. That is, as long as information is sent and received via the communication network 7010, predetermined arithmetic processing can be performed by any of the control units. Similarly, a sensor or device connected to one of the control units can be connected to another control unit, and multiple control units can send and receive detection information to each other via the communication network 7010.

[0085] In addition, in the present embodiment, the integrated control unit 7600 is capable of performing semantic segmentation for identifying attributes such as a road surface, a sidewalk, pedestrians, and buildings on each pixel of the image captured by the imaging section 7410 .

[0086] [Configuration of functional blocks of vehicle control system]

[0087] Figure 3 The figure illustrates the configuration of the functional blocks of the computer program implemented in the integrated control unit 7600. The computer program can be provided in the form of a computer-readable recording medium in which the computer program is stored. Examples of recording media include magnetic disks, optical disks, magneto-optical disks, and flash memory. In addition, the computer program can be distributed, for example, via a network without using a recording medium.

[0088] In this embodiment, the integrated control unit 7600 (microcomputer 7610) can perform semantic segmentation for each pixel of the captured image sequentially acquired from the imaging unit 7410, which is applied to identify attributes such as a road surface, a sidewalk, pedestrians, and buildings. Through the performed semantic segmentation, the attributes are identified for each subject area included in the captured image.

[0089] Based on this attribute, the integrated control unit 7600 can set the frequency (update frequency) at which recognition processing is performed and the area to be the target of recognition processing. Note that in the processing, semantic segmentation is performed with respect to the entirety of the first captured image in a series of captured images, and the update frequency is set for each area in subsequent captured images.

[0090] like Figure 3 As shown in the figure, the integrated control unit 7600 includes a relative motion estimator 11, a projection map generator 12, a semantic segmentation projection unit 13, an unobserved area setting unit 14, a region attribute relationship determination unit 15, an update priority map generator 16, a region semantic segmentation unit 17 and a semantic segmentation integration unit 18 as functional blocks.

[0091] Based on the position information about the vehicle's position at time (T-1) and the position information about the vehicle's position at time (T) generated by the positioning unit 7640 (imaging unit 7410), the relative movement estimator 11 generates data (Rt) of the relative movement amount of the vehicle and outputs the generated data to the projection map generator 12.

[0092] Based on the data (z) of the distance between the vehicle and the subject at time (T-1) for each captured image coordinate pair (the distance is detected by the vehicle external information detection unit 7400), and based on the relative movement amount data (Rt) received from the relative movement estimator 11, the projection map generator 12 generates projection map data and outputs the generated data to the semantic segmentation projection unit 13 and the unobserved area setting unit 14.

[0093] Specifically, relative to the distance data (z) of each captured image coordinate pair, the projection map generator 12 transforms the set of all distance data (z) of each captured image coordinate pair (depth image data) into three-dimensional point cloud data, and performs coordinate transformation on the point cloud data using the relative movement amount data (Rt). Then, the projection map generator 12 generates depth image data obtained by projecting the point cloud data obtained after the coordinate transformation onto the captured image plane. Based on the distance data (z) and the image coordinates of the time (T-1) in the depth image data, the projection map generator 12 generates projection map data, which indicates the position of the projection source and is used to project a value indicating the result of image recognition (semantic segmentation) performed on each pixel of the captured image relative to the time (T-1) onto the captured image at time (T).

[0094] Based on the projection map data received from the projection map generator (12) and the semantic segmentation result at time (T-1), the semantic segmentation projection unit 13 generates projected semantic segmentation data obtained by projecting the semantic segmentation result onto the captured image at time (T), and outputs the generated data to the semantic segmentation integration unit 18.

[0095] Based on the projection map data received from the projection map generator 12, the unobserved area setting unit 14 detects an area on which the semantic segmentation result of time (T-1) is not projected in the captured image at time (T) (that is, an unobserved area in which the position of the projection source in the projection map data is not indicated), and outputs data indicating the unobserved area to the update priority map generator 16.

[0096] The region attribute relationship determination unit 15 determines the relationship between the attributes identified by the semantic segmentation performed on the plurality of regions included in the captured image. For example, when the region of a sidewalk or road overlaps with the region of a pedestrian or bicycle, the region attribute relationship determination unit 15 determines that a pedestrian or bicycle is present on the sidewalk or road.

[0097] Based on the relationship between the unobserved area detected by the unobserved area setting unit 14 and the area attributes determined by the area attribute relationship determination unit 15, the update priority map generation unit 16 generates an update priority map in which the update priority (update frequency) of semantic segmentation is set for each area of ​​the captured image.

[0098] For example, the update priority map generator 16 assigns a high update priority to an unobserved area, assigns a low update priority to an area of ​​pedestrians on a sidewalk, and assigns a high update priority to an area of ​​pedestrians on a road surface.

[0099] Based on the generated update priority map, the region semantic segmentation section 17 performs semantic segmentation for each region of the captured image with respect to time (T), and outputs the result of the semantic segmentation to the semantic segmentation integration section 18 .

[0100] The semantic segmentation integration unit 18 integrates the projected semantic segmentation data of time (T) received from the semantic segmentation projection unit 13 and the regional semantic segmentation data of time (T) received from the regional semantic segmentation unit 17, and outputs data of the overall semantic segmentation result of the captured image relative to time (T).

[0101] The semantic segmentation result data can be used to perform (for example) cooperative control aimed at realizing ADAS functions or (for example) cooperative control aimed at realizing (for example) autonomous driving.

[0102] These functional blocks (computer programs) may be implemented in the vehicle exterior information detection unit 7400 instead of the integrated control unit 7600. In this case, the integrated control unit 7600 performs cooperative control for ADAS or autonomous driving based on the semantic segmentation result data output by the vehicle exterior information detection unit.

[0103] [Operation of vehicle control system]

[0104] Next, the operation of the vehicle control system having the above configuration is described. This operation is performed by hardware such as the microcomputer 7610 of the integrated control unit 7600, the vehicle network I / F 7680, and the dedicated communication I / F 7630 and software stored in, for example, the storage unit 7690 ( Figure 3 The corresponding functional blocks shown in the figure) work together to execute.

[0105] Figure 4 is a flowchart illustrating the flow of image recognition processing performed by the vehicle control system.

[0106] As illustrated in the figure, first, the relative movement estimator 11 obtains position information about the position of the vehicle at time (T-1) and position information about the position of the vehicle at time (T) (step 101), and estimates the relative movement distance of the vehicle (imaging part) from time (T-1) to time (T) (step 102).

[0107] Subsequently, the projection map generator 12 acquires data of the distance between the vehicle and the subject in the captured image at time (T-1) (step 103), and generates projection map data based on the distance data and the data of the relative movement distance (step 104).

[0108] Subsequently, based on the projection map data, the unobserved area setting unit 14 calculates the unobserved area, which is included in the captured image at time (T) and obtained by comparing the captured image at time (T) with the captured image at time (T-1), and generates an update priority map in which a high update priority is given to the unobserved area (step 106).

[0109] Subsequently, based on the projected map data, the semantic segmentation projection unit 13 projects the semantic segmentation result at time (T-1) onto the captured image at time (T) (step 107).

[0110] Figure 5 The projection process using the projected map data is illustrated. In (B1) and (B2) of the figure and in subsequent figures, the areas represented by different grayscale shades each indicate the results of the recognition performed by the semantic segmentation performed. In other words, this indicates that the same attribute is recognized relative to the parts represented by the same color.

[0111] Assume that when a vehicle traveling through the point indicated in (A1) of the figure at time T=0 moves to the point indicated in (A2) of the figure at time T=1, relative to all pixels of the input frame (B0) at time T=0, it has been determined from the position information and information about the distance which of the pixels of the input frame (B0) corresponds to which of the pixels of the input frame at time T=1, as illustrated in the figure.

[0112] In this case, the result (B1) of the semantic segmentation of the input frame at time T = 1 is projected onto the entire area of ​​the input frame at time T = 1, as shown in (B2) of the figure. Therefore, the redundant processing of the semantic segmentation performed on the input frame at time T = 1 is reduced, the amount of calculation is reduced, and the recognition accuracy (stability) is improved.

[0113] Figure 6 The process of calculating the unobserved area is illustrated. When the vehicle that travels through the point indicated in (A1) of the figure at T=0 moves to the point indicated in (A2) of the figure at time T=1, an unobserved area R onto which the result (B1) of the semantic segmentation relative to the input frame (B0) at time T=0 is not projected appears in the input frame at time T=1, as illustrated in (B2) of the figure. This is consistent with the above Figure 5 The situation is different.

[0114] As described above, depending on the composition of the image captured by the camera, all semantic segmentation results may be projected onto the next frame, or an unobserved region onto which a portion of the semantic segmentation results is not projected may appear in the next frame.

[0115] Here, the process of projected map generation and the process of unobserved area setting are described in detail.

[0116] Figure 7 Detailed diagram of the projection map generation process, Figure 8 : is a flowchart illustrating the flow of projection map generation processing.

[0117] like Figure 7 As shown in FIG, the projection map generator 12 includes a point cloud conversion unit 121, a coordinate conversion unit 122, a plane projection unit 123, and a map generator 124 as functional blocks.

[0118] First, the point cloud conversion section 121 acquires depth image data D (a captured image including distance information for each pixel) from the vehicle exterior information detection unit 7400. The depth image data has distance data (z) stored therein for each pair of image coordinates (u, v).

[0119] Then, the point cloud conversion unit 121 converts all pixels of the depth image D into three-dimensional point cloud data P ( Figure 7 (A) and Figure 8 Step 201). The point cloud data P stores therein a transformed source image coordinate pair (u, v) for each set of point cloud coordinates (x, y, z).

[0120] Then, with respect to all point clouds included in the point cloud data P, the coordinate transformation section 122 performs coordinate transformation on each point cloud data P based on the relative movement amount data (Rt) acquired from the relative movement estimator 11, the relative movement amount data (Rt) being data of the relative movement amount of the camera ( Figure 7 (B) and Figure 8 The point cloud data P′ obtained by the coordinate transformation stores therein an image coordinate pair (u, v) of the transformation source depth image for each set of point cloud coordinates (x, y, z) obtained by the coordinate transformation.

[0121] Then, with respect to all point clouds included in the point cloud data P' obtained by the coordinate transformation, the plane projection section 123 projects the point cloud data P' onto the image plane ( Figure 7 (C) and Figure 8 Step 203 is repeated to generate depth image data D' after coordinate transformation. For each image coordinate pair (u, v), the depth image data D' after coordinate transformation stores therein the distance data (z) after coordinate transformation and the transformation source image coordinate pair (u, v).

[0122] Next, with respect to all pixels of the depth image D' after coordinate transformation, the map generator 124 associates the coordinate pairs of the pixels in the subsequent (moved) frame of the transformation source frame with the coordinate pairs of the pixels in the (pre-moved) transformation source frame to generate projection map data M ( Figure 7(D) and Figure 8 Step 204).

[0123] The projection map data M stores therein the image coordinate pair (u, v) of the transformed source frame for each image coordinate pair (u, v) of the moved frame. The projection map data M indicates a correspondence relationship indicating which coordinate pair in the frame before the movement will be associated with the coordinate pair in the frame after the movement when the semantic segmentation result of the coordinate pair in the frame before the movement is projected onto the coordinate pair in the frame after the movement.

[0124] Figure 9 Detailed diagram of the unobserved region setup process, Figure 10 : is a flowchart illustrating the flow of the unobserved region setting process.

[0125] like Figure 9 As illustrated in FIG, the unobserved region setting section 14 includes an unassociated pixel extraction section 141 as a functional block.

[0126] With respect to all the coordinate pairs of each pixel in the projection map data M, the unassociated pixel extraction unit 141 performs the following processing: associating the coordinate pairs with the coordinate pairs of the pixels in the next frame (T) to extract the unassociated pixels (or the area including the unassociated pixels) included in the next frame (T) that are not associated with the pixels in the projection map data M as the unobserved area R (step 301).

[0127] Therefore, with respect to the pixels (or the area including the associated pixels) included in the next frame (T-1) and associated with the pixels in the projected map data M due to the association processing, the semantic segmentation projection unit 13 projects the semantic segmentation results relative to the original frame (T-1) onto the associated pixels (or the area including the associated pixels).

[0128] On the other hand, with respect to the unobserved area R included in the next frame (T-1) that is not associated with the pixels in the projection map data M due to the association processing, the processing of generating an updated priority map is performed, and the semantic segmentation processing is newly performed by the regional semantic segmentation unit 17 to identify the attributes of each pixel of the unobserved area R.

[0129] return Figure 4 The region attribute relationship determination unit 15 determines the relationship between the attributes of the plurality of regions in the captured image based on the projected semantic segmentation data based on the projected map data (step 108 ).

[0130] Subsequently, the update priority map generator 16 generates an update priority map based on the determined relationships between the area attributes (step 109 ).

[0131] Figure 11It is a diagram for describing a process of determining an area attribute relationship and a process of generating an update priority map.

[0132] When the semantic segmentation result at time (T-1) illustrated in (A) of the figure is projected as the semantic segmentation result at time (T) illustrated in (B) of the figure, the region attribute relationship determination unit 15 determines that the region of the pedestrian and the region of the sidewalk in the captured image overlap on the left, and also determines that the region of the pedestrian and the road surface overlap on the right in the captured image.

[0133] In this case, the pedestrians and bicycles on the sidewalk are not expected to be in a very dangerous state. Therefore, the update priority map generator 16 gives a low update priority to the area of ​​pedestrians and bicycles on the sidewalk, as illustrated in (C) of the figure.

[0134] On the other hand, pedestrians and bicycles on the road are expected to be in a dangerous state. Therefore, the update priority map generator 16 assigns a high update priority to the areas of pedestrians and bicycles on the road. Note that in the update priority maps shown in (C) of the figure and in subsequent figures, darker gray indicates a higher update priority.

[0135] Furthermore, the update priority map generator 16 may give a high update priority to an area at a boundary between an area of ​​a sidewalk or road surface and an area other than the area thereof, because the boundary area may be a location out of sight and another object may suddenly run outside the boundary area.

[0136] In addition, the update priority map generator 16 is not limited to generating the update priority map based on the relationship between the attributes of two areas, and may generate the update priority map based on the relationship between the attributes of three or more areas.

[0137] For example, the update priority map generator 16 may give high update priority to areas of pedestrians and bicycles on the road surrounding areas of cars, because the car is likely to change its movement to avoid pedestrians and bicycles.

[0138] In addition, the update priority map generator 16 may give high update priority to areas where pedestrians and bicycles are close to each other on the road surface. The reason is that pedestrians and bicycles are likely to change their movements in order to avoid each other.

[0139] return Figure 4 , the update priority map generator 16 integrates the update priority map generated based on the unobserved area in the above step 106 with the update priority map generated based on the relationship between the area attributes in the above step 109 (step 110).

[0140] Figure 12 The figure illustrates how to integrate the update priority map. Assume that based on the semantic segmentation results shown in (A) in the figure, the update priority map shown in (B) in the figure is obtained based on the unobserved area, and the update priority map shown in (C) in the figure is obtained based on the relationship between the area attributes.

[0141] The update priority map generator 16 integrates the two update priority maps to generate an integrated update priority map as shown in (D) of the figure. As a result of the integration, due to the combination of the degrees of priority in the respective update priority maps, a high priority is given to the area where the areas set in the two update priority maps overlap.

[0142] Here, in the update priority map based on the unobserved area, the update priority map generator 16 may set an area slightly larger than the detected unobserved area before integration in order to improve detection accuracy.

[0143] In addition, in the update priority map based on the relationship between area attributes, the update priority map generator 16 may set an area larger than the area in which, for example, a pedestrian is detected before integration in order to cope with the movement of the pedestrian.

[0144] return Figure 4 , then, the region semantic segmentation section 17 performs semantic segmentation processing with respect to each region according to the update priority (update frequency) based on the update priority map obtained by the integration (step 111).

[0145] Figure 13 An example of semantic segmentation processing performed based on the updated priority map obtained through integration is illustrated.

[0146] For example, when the update priority map illustrated in FIG. 1 is obtained, the region semantic segmentation section 17 sets a rectangle circumscribing the high priority region as illustrated in FIG. 1 (B) and performs semantic segmentation with respect to the region of the circumscribed rectangle.

[0147] As illustrated in (C) of the figure, when the region semantic segmentation unit 17 determines that no delay will occur even if processing is performed with respect to all circumscribed rectangles in consideration of computing resources, the region semantic segmentation unit 17 performs semantic segmentation with respect to all of the regions of the set circumscribed rectangles.

[0148] On the other hand, as illustrated in (D) and (E) of the figure, when it is determined that a delay will occur if processing is performed with respect to all bounding rectangles in consideration of computing resources, regions with low update priority may be excluded from the semantic segmentation target.

[0149] return Figure 4Finally, the semantic segmentation integration unit 18 integrates the semantic segmentation result at time T obtained by projection (step 107) with the result of semantic segmentation performed on the region (step 111), and outputs integrated semantic segmentation data. This series of semantic segmentation processing is then terminated (step 112).

[0150] As described above, according to this embodiment, the integrated control unit 7600 of the vehicle control system 7000 does not perform recognition processing equally for each captured image (frame), but rather sets the frequency of performing semantic segmentation processing based on the attributes of the regions in the image. This makes it possible to eliminate redundant processing and reduce the amount of calculation.

[0151] [revised form]

[0152] The present technology is not limited to the above-described embodiments, and various modifications may be made thereto without departing from the scope of the present technology.

[0153] In the above embodiment, the area attribute relationship determination unit 15 and the update priority map generator 16 set update priorities based on the relationship between area attributes. However, update priorities can also be set based on the attributes of each area. For example, a low update priority can be assigned to areas with signals or signs. Taking into account movement speed, a higher update priority can be assigned to areas with bicycles than to areas with pedestrians, and a higher update priority can be assigned to areas with cars than to areas with bicycles.

[0154] In addition, the update priority map generator 16 integrates the update priority map based on the unobserved area and the update priority map based on the relationship between the area attributes to generate an update priority map for performing semantic segmentation. In addition to or instead of the two update priority maps, the update priority map generator 16 can integrate an update priority map generated using another parameter. Figures 14 to 16 is a diagram for describing such an update priority map.

[0155] The update priority map generator 16 may set the update priority according to the location of the area in the captured image.

[0156] For example, Figure 14 As illustrated in the figure, relative to the input frame illustrated in (A) of the figure, the update priority map generator 16 can assign a higher update priority to an area closer to the center part in the image corresponding to the direction of vehicle travel, and can assign a lower update priority to an area closer to the end part in the image not corresponding to the direction of vehicle travel, and can generate the update priority map illustrated in (B) of the figure.

[0157] Furthermore, for example, the update priority map generator 16 may assign a higher update priority to the upper portion of the image compared to the lower portion of the image.

[0158] In addition, the update priority map generator 16 may set the update priority according to the moving (traveling) speed of the vehicle and according to the position of the area in the captured image.

[0159] Discussion in e.g. Figure 15 , in which the input frame illustrated in FIG. (A) is obtained. When the vehicle is moving at high speed (e.g., traveling at a threshold speed of 80 km / h or higher), the update priority map generator 16 assigns a high update priority to the area in the center portion of the image, and assigns a low update priority to the end portion of the image, as illustrated in FIG. (B). This is because, in this case, it is generally more important for the driver to look ahead than to look around.

[0160] On the other hand, when the vehicle is moving at a low speed (e.g., at a threshold speed of 30 km / h or less), the update priority map generator 16 assigns a low update priority to the area in the center portion of the image and assigns a low update priority to the area in the end portion of the image, as illustrated in (C) of the figure. This is because, in this case, it is generally more important for the driver to look around than to look forward.

[0161] In addition, the update priority map generator 16 may set the update priority according to the distance (z) between the subject in the captured image and the vehicle.

[0162] For example, Figure 16 As illustrated in FIG, when the depth image data illustrated in FIG (B) is obtained relative to the input frame illustrated in FIG (A), the update priority map generator 16 can assign a higher update priority to an area of ​​pixels including information about a smaller distance (an area located closer to the subject of the vehicle), and can assign a lower update priority to an area of ​​the subject located farther away from the vehicle, as illustrated in FIG (C).

[0163] when Figures 14 to 16 When at least one of the update priority maps is integrated with an update priority map based on an unobserved area or an update priority map based on a relationship between area attributes, a high update priority is given to areas in which areas of the integrated update priority maps overlap (such as areas in which the unobserved area overlaps with the center area of ​​the image and areas in which the unobserved area and areas including information about small distances overlap).

[0164] In the above embodiment, the regional semantic segmentation unit 17 does not perform semantic segmentation on the entire captured image, but only on the region set by the update priority map generator 16. However, the regional semantic segmentation unit 17 may periodically perform semantic segmentation on all regions of the captured image. This allows for periodic compensation to cover errors caused by the partial recognition processing performed on each region.

[0165] Figure 17 The figure shows an example of performing semantic segmentation for all regions in this case (hereinafter referred to as all-region processing). Figure (A) shows an example of time series processing performed when the periodic all-region processing in the above embodiment is not performed. On the other hand, when the periodic all-region processing is performed, there is a long delay, but accurate recognition results are obtained after performing the all-region processing, as shown in Figure (B).

[0166] In addition, the regional semantic segmentation unit 17 may periodically perform full-region processing, and may allow for delays when performing semantic segmentation on a limited region selected based on the update priority, as shown in (C) of the figure. This causes delays, but when performing semantic segmentation on a limited region, processing can be performed on all regions necessary for recognition without omitting processing due to computational resources.

[0167] Here, various triggering mechanisms for performing full-area processing are contemplated.

[0168] The regional semantic segmentation unit 17 can perform full-region processing when the ratio of the area of ​​one or more unobserved regions (one or more regions on which projection is not performed using the projection map) is equal to or greater than a predetermined ratio. When the area of ​​one or more unobserved regions is large, the difference in the amount of computation between full-region processing and semantic segmentation performed with respect to a limited area is small. Therefore, when the regional semantic segmentation unit 17 performs full-region processing, this makes it possible to improve recognition accuracy while suppressing an increase in the amount of computation.

[0169] The region semantic segmentation unit 17 can perform full-region processing when the vehicle's steering angle detected by the vehicle state detection unit 7110 is equal to or greater than a predetermined angle. It is expected that when a large steering angle is detected, there will be a large change in the image capture target scene and the unobserved area will increase. Therefore, when the region semantic segmentation unit 17 performs full-region processing in this situation, it can eliminate the computational effort required to specifically detect the unobserved area and improve recognition accuracy.

[0170] The region semantic segmentation unit 17 may perform full region processing when the vehicle is moving through a predetermined point. GPS information and map information acquired by the positioning unit 7640 are used as position information.

[0171] For example, the region semantic segmentation unit 17 may perform full-region processing when it detects that the vehicle is traveling uphill or downhill, where the slope of the slope is equal to or greater than a predetermined value. It is expected that on a steeply inclined uphill or downhill slope, the image capture target scene will change significantly and the number of unobserved areas will increase. Therefore, when the region semantic segmentation unit 17 performs full-region processing in this situation, it is possible to eliminate the computational effort required to specifically detect unobserved areas and improve recognition accuracy.

[0172] Furthermore, when the vehicle enters or exits a tunnel, the region semantic segmentation section 17 may perform full-region processing because the image capture target scene will also change greatly in this case.

[0173] In addition, when the ratio of the area of ​​a region or multiple regions in the captured image where the result of attribute recognition performed by the performed semantic segmentation is less reliable is equal to or greater than a predetermined ratio (for example, 50%), or when the ratio of the area of ​​a region or multiple regions in the captured image whose attributes are not recognized by the performed semantic segmentation is equal to or greater than a predetermined ratio (for example, 50%), the regional semantic segmentation unit 17 can perform full-region processing.

[0174] In the above embodiment, the region semantic segmentation unit 17 is configured as follows: Figure 13 The rectangle circumscribing the high-priority area shown in FIG is used, and semantic segmentation is performed with respect to the area of ​​the circumscribing rectangle. However, the method for setting the semantic segmentation target area is not limited to this. For example, the area semantic segmentation unit 17 may set only the area of ​​pixels estimated to be necessary for performing calculations during semantic segmentation, rather than the area cut out along the circumscribing rectangle, as the semantic segmentation target.

[0175] In other words, if Figure 18 As illustrated in (A), when multiple convolution operations are performed on an input image to obtain the final semantic segmentation result (processing performed by following the arrows in the upper part), in order to calculate the area required for the final result, it is sufficient to perform the operation only on the necessary area by following the reverse of the convolution operation (processing performed by following the arrows in the lower part).

[0176] Therefore, when the updated priority map illustrated in (B) of the figure is obtained, the regional semantic segmentation unit 17 can perform reverse calculations to obtain the area required to obtain the high priority area indicated by the updated priority map as the final result, the semantic segmentation target area can be set as illustrated in (C) of the figure, and semantic segmentation can be performed relative to the set area.

[0177] In this case, the region semantic segmentation unit 17 may further exclude low-priority regions from the semantic segmentation targets when determining that a delay will occur in consideration of computational resources.

[0178] In the above embodiment, an example has been described in which a vehicle (car) is a mobile body equipped with the integrated control unit 7600 serving as an information processing device. However, the mobile body equipped with an information processing device capable of performing information processing similar to that performed by the integrated control unit 7600 is not limited to a vehicle. For example, the information processing device can be set as a device installed on any type of mobile body such as a motorcycle, a bicycle, a personal mobile device, an airplane, a drone, a ship, a robot, a construction machine, or an agricultural machine (tractor). In this case, the relationship between the above-mentioned attributes (such as pedestrians, vehicles, road surfaces, and sidewalks) is recognized differently depending on the mobile body.

[0179] Furthermore, the target on which the above-described information processing device is installed is not limited to a mobile object. For example, the present technology is also applicable to images captured by a surveillance camera. In this case, the processing associated with vehicle movement described in the above embodiment is not performed, but the image capture target can be changed in accordance with the panning, tilting, and zooming performed by the surveillance camera. Therefore, the present technology is also applicable when generating an update priority map based on unobserved areas in addition to generating an update priority map based on area attributes.

[0180] [other]

[0181] The present technology may also take the following configurations.

[0182] (1) An information processing device comprising:

[0183] an input device to which a capture image captured by the camera is input, the capture image including distance information for each pixel; and

[0184] A controller that:

[0185] generating a transformed captured image obtained by transforming coordinate pairs of respective pixels of the captured image based on an amount of movement of the camera or a mobile object on which the camera is mounted,

[0186] associating coordinate pairs of pixels of the transformed captured image with coordinate pairs of pixels of a moved captured image captured by the camera at the moved position, and

[0187] Unassociated pixels included in the moved captured image that are not associated with pixels of the transformed captured image are identified.

[0188] (2) The information processing device according to (1), wherein

[0189] The controller

[0190] performing a recognition process of identifying attributes of the unassociated pixels in the captured image after movement, and

[0191] The result of the recognition processing is projected onto the associated pixels in the captured image after movement or onto the area including the associated pixels, and the recognition processing is performed relative to the pixels included in the captured image corresponding to the associated pixels or the area including the associated pixels.

[0192] (3) The information processing device according to (2), wherein

[0193] The controller generates a map obtained by associating coordinate pairs of pixels of the moved captured image with coordinate pairs of pixels of the captured image in order to perform the projection.

[0194] (4) The information processing device according to any one of (1) to (3), wherein

[0195] The controller

[0196] transforming the captured image into three-dimensional point cloud data based on distance information of each pixel,

[0197] generating moved point cloud data obtained by performing a transformation with respect to the three-dimensional point cloud data based on the movement amount, and

[0198] The moving point cloud data is projected onto an image plane to generate the transformed captured image.

[0199] (5) The information processing device according to any one of (2) to (4), wherein

[0200] The controller sets a priority for executing recognition processing according to positions of the unassociated pixels in the moved captured image.

[0201] (6) The information processing device according to (5), wherein

[0202] The controller sets a priority for executing recognition processing for each unassociated pixel according to a position of the unassociated pixel in the captured image after movement and according to a moving speed of the moving object.

[0203] (7) The information processing device according to any one of (2) to (6), wherein

[0204] The controller sets a priority for performing recognition processing for each unassociated pixel according to distance information of the unassociated pixels.

[0205] (8) An information processing method comprising:

[0206] Acquire a captured image captured by a camera, wherein the captured image includes distance information for each pixel;

[0207] generating a transformed captured image obtained by transforming coordinate pairs of respective pixels of the captured image based on an amount of movement of the camera or a mobile object on which the camera is mounted;

[0208] associating coordinate pairs of pixels of the transformed captured image with coordinate pairs of pixels of a moved captured image captured by the camera at the moved position; and

[0209] Unassociated pixels included in the moved captured image that are not associated with pixels of the transformed captured image are identified.

[0210] (9) A program for causing an information processing device to execute a process including the following steps:

[0211] Acquire a captured image captured by a camera, wherein the captured image includes distance information for each pixel;

[0212] generating a transformed captured image obtained by transforming coordinate pairs of respective pixels of the captured image based on an amount of movement of the camera or a mobile object on which the camera is mounted;

[0213] associating coordinate pairs of pixels of the transformed captured image with coordinate pairs of pixels of a moved captured image captured by the camera at the moved position; and

[0214] Unassociated pixels included in the moved captured image that are not associated with pixels of the transformed captured image are identified.

[0215] Reference Symbol List

[0216] 11 Relative Motion Estimator

[0217] 12 Projection Map Generator

[0218] 13 Semantic Segmentation Projection

[0219] 14 Unobserved Area Setting Department

[0220] 15. Regional attribute relationship determination unit

[0221] 16 Update priority map generator

[0222] 17. Region Semantic Segmentation

[0223] 18 Semantic Segmentation Integration Department

[0224] 121 Point Cloud Transformation Department

[0225] 122 Coordinate Transformation Unit

[0226] 123 Plane Projection Department

[0227] 124 Map Generator

[0228] 141 Unrelated pixel extraction unit

[0229] 7000 Vehicle Control Systems

[0230] 7400 Vehicle External Information Detection Unit

[0231] 7600 Integrated Control Unit

[0232] 7610 Microcomputer

[0233] 7680 In-vehicle Network I / F

[0234] 7690 Storage Department

[0235] R Unobserved area

Claims

1. An information processing device, comprising: an input device to which a capture image captured by the camera is input, the capture image including distance information for each pixel; as well as A controller that: generating a transformed captured image obtained by transforming coordinate pairs of respective pixels of the captured image based on an amount of movement of the camera or a mobile object on which the camera is mounted, associating coordinate pairs of pixels of the transformed captured image with coordinate pairs of pixels of a moved captured image captured by the camera at the moved position, and identifying unassociated pixels included in the moved captured image that are not associated with pixels of the transformed captured image, Wherein, the controller performing a recognition process of identifying attributes of the unassociated pixels in the captured image after movement, and The result of the recognition processing is projected onto the associated pixels in the captured image after movement or onto the area including the associated pixels, and the recognition processing is performed relative to the pixels included in the captured image and corresponding to the associated pixels or the area including the associated pixels.

2. The information processing device according to claim 1, wherein The controller generates a map obtained by associating coordinate pairs of pixels of the moved captured image with coordinate pairs of pixels of the captured image in order to perform the projection.

3. The information processing device according to claim 1, wherein The controller transforming the captured image into three-dimensional point cloud data based on distance information of each pixel, generating moved point cloud data obtained by performing a transformation with respect to the three-dimensional point cloud data based on the movement amount, and The moving point cloud data is projected onto an image plane to generate the transformed captured image.

4. The image processing apparatus according to claim 1, wherein: The controller sets a priority for performing recognition processing according to positions of the unassociated pixels in the moved captured image.

5. The information processing apparatus according to claim 4, wherein: The controller sets a priority for executing recognition processing for each unassociated pixel according to a position of the unassociated pixel in the captured image after movement and according to a moving speed of the moving object. The information processing apparatus according to claim 1 , wherein: The controller sets a priority for performing recognition processing for each unassociated pixel according to distance information of the unassociated pixels.

7. An information processing method, comprising: Acquire a captured image captured by a camera, wherein the captured image includes distance information for each pixel; generating a transformed captured image obtained by transforming coordinate pairs of respective pixels of the captured image based on an amount of movement of the camera or a mobile object on which the camera is mounted; associating coordinate pairs of pixels of the transformed captured image with coordinate pairs of pixels of a moved captured image captured by the camera at the moved position; identifying unassociated pixels included in the shifted captured image that are not associated with pixels of the transformed captured image; performing recognition processing to identify attributes of the unassociated pixels in the captured image after movement; as well as The result of the recognition processing is projected onto the associated pixels in the captured image after movement or onto the area including the associated pixels, and the recognition processing is performed relative to the pixels included in the captured image and corresponding to the associated pixels or the area including the associated pixels.

8. A program product for causing an information processing device to execute a process comprising the following steps: Acquire a captured image captured by a camera, wherein the captured image includes distance information for each pixel; generating a transformed captured image obtained by transforming coordinate pairs of respective pixels of the captured image based on an amount of movement of the camera or a mobile object on which the camera is mounted; associating coordinate pairs of pixels of the transformed captured image with coordinate pairs of pixels of a moved captured image captured by the camera at the moved position; identifying unassociated pixels included in the shifted captured image that are not associated with pixels of the transformed captured image; performing recognition processing to identify attributes of the unassociated pixels in the captured image after movement; as well as The result of the recognition processing is projected onto the associated pixels in the captured image after movement or onto the area including the associated pixels, and the recognition processing is performed relative to the pixels included in the captured image and corresponding to the associated pixels or the area including the associated pixels.

Images