Learning model generation method, information processing device, and information processing system

Abstract

The present technology relates to a training model generation method, an information processing device, and an information processing system that enable the construction of a recognizer that makes fewer erroneous detections. A target that has been recognized by a recognition process using a recognizer to which a training model for performing a recognition process on input data has been applied is tracked in a chronologically opposite direction, and the training model is retrained using data generated on the basis of the tracking results. The data is generated by tracking the target in a chronologically opposite direction and applying a label to the tracked target. The present technology is applicable to, for example, an information processing device that retrains a recognizer that recognizes a predetermined object.

Description

[Technical Field]

[0001] The present technology relates to a learning model generation method, an information processing device, and an information processing system, and more particularly to a learning model generation method, an information processing device, and an information processing system that execute processing related to re-learning of a recognizer used in recognition processing, for example. [Background technology]

[0002] Various proposals have been made regarding technologies for recognizing predetermined objects such as people, cars, etc. For example, Patent Document 1 proposes a technology that continues to detect the same person even when the face moves up and down and left and right, or the size of the face changes from frame to frame. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 4389956 Summary of the Invention [Problem to be solved by the invention]

[0004] When recognizing a specific object such as a person or a car using a pre-trained recognizer, there is a possibility that the recognizer will make an incorrect recognition if a similar case occurs in which the recognizer made a previous incorrect recognition. It is desirable to improve the performance of the recognizer so that incorrect recognition is not repeated.

[0005] The present technology has been made in view of such circumstances, and makes it possible to improve the performance of a recognizer. [Means for solving the problem]

[0006] A method for generating a learning model according to one aspect of the present technology includes tracking an object recognized by a recognition process using a recognizer to which a learning model that performs a recognition process on input data is applied in a reverse chronological order, and re-training the learning model using data generated based on the tracking results. Among the recognition results obtained by the recognition processing for the frames captured at the first time, a recognition result that satisfies a predetermined criterion is set as the object to be tracked, and the object captured in a plurality of frames captured at a time before the first time is tracked, and if the object is detected in a frame as a result of the tracking, a label is assigned to the object. .

[0007] According to one aspect of the present technology, there is provided an information processing apparatus including: a re-learning unit that tracks an object recognized by a recognition process using a recognizer in a chronologically reverse direction, and re-learns a learning model of the recognizer based on learning data for re-learning the recognizer, the learning data being generated based on a result of the tracking; a recognition result that satisfies a predetermined criterion among the recognition results obtained by a recognition process on frames captured at a first time is determined to be the object to be tracked, the object captured in a plurality of frames captured at a time before the first time is tracked, and when the object is detected in a frame as a result of the tracking, a label is assigned to the object. .

[0008] An information processing system according to one aspect of the present technology includes a recognition processing unit that performs recognition processing using a recognizer to which a learning model that performs recognition processing on input data is applied; an extraction unit that extracts recognition results that satisfy a predetermined criterion from recognition results recognized by the recognition processing unit; a tracking unit that treats the recognition results extracted by the extraction unit as objects and tracks the objects in a reverse chronological direction; a label assignment unit that assigns labels to the objects tracked by the tracking unit; a re-learning unit that re-learns the learning model using the labels assigned by the label assignment unit; and an update unit that updates the recognizer of the recognition processing unit with the learning model re-learned by the re-learning unit. The extraction unit selects a recognition result that satisfies the predetermined criteria from among the recognition results by the recognition processing unit for frames captured at a first time as the object to be tracked, the tracking unit tracks the object captured in a plurality of frames captured at a time before the first time, and the label assignment unit assigns a label to the object when the object is detected in a frame as a result of the tracking. .

[0009] In a method for generating a learning model according to one aspect of the present technology, an object recognized by a recognition process using a recognizer to which a learning model that performs a recognition process on input data is applied is tracked in a reverse direction in time series, and data generated based on the tracking results is used to re-train the learning model. Among the recognition results obtained by the recognition process on the frames captured at the first time, a recognition result that satisfies a predetermined criterion is determined to be the object to be tracked, and the object captured in a plurality of frames captured at a time before the first time is tracked. If the object is detected in a frame as a result of the tracking, a label is assigned to the object. .

[0010] In an information processing device according to one aspect of the present technology, a re-learning unit is configured to track an object recognized by a recognition process using a recognizer in a reverse direction in time series, and re-learn a learning model of the recognizer based on learning data for re-learning the recognizer, the learning data being generated based on a result of the tracking. Among the recognition results obtained by a recognition process on frames captured at a first time, a recognition result that satisfies a predetermined criterion is set as the object to be tracked, and the object captured in a plurality of frames captured at a time before the first time is tracked, and if the object is detected in a frame as a result of the tracking, a label is assigned to the object. .

[0011] An information processing system according to one aspect of the present technology includes a recognition processing unit that performs recognition processing using a recognizer to which a learning model that performs recognition processing on input data is applied; an extraction unit that extracts recognition results that satisfy a predetermined criterion from recognition results recognized by the recognition processing unit; a tracking unit that treats the recognition results extracted by the extraction unit as objects and tracks the objects in a reverse chronological direction; a label assignment unit that assigns labels to the objects tracked by the tracking unit; a re-learning unit that re-learns the learning model using the labels assigned by the label assignment unit; and an update unit that updates the recognizer of the recognition processing unit with the learning model re-learned by the re-learning unit. The extraction unit selects a recognition result that satisfies the predetermined criteria from among the recognition results by the recognition processing unit for frames captured at a first time as the object to be tracked, the tracking unit tracks the object captured in a plurality of frames captured at a time before the first time, and the label assignment unit assigns a label to the object when the object is detected in a frame as a result of the tracking. .

[0012] The information processing device may be an independent device or an internal block constituting one device. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a block diagram illustrating an example of the configuration of a vehicle control system. [Figure 2] FIG. 2 is a diagram illustrating an example of a sensing region. [Figure 3] 1 is a diagram illustrating an example of the configuration of an embodiment of an information processing device to which the present technology is applied. [Figure 4] FIG. 10 is a diagram for explaining a learning method. [Figure 5] FIG. 10 is a diagram illustrating an example of a recognition result. [Figure 6] FIG. 10 is a diagram illustrating an example of a recognition result. [Figure 7] FIG. 10 is a diagram illustrating an example of a recognition result. [Figure 8] FIG. 10 is a diagram illustrating an example of a recognition result. [Figure 9] FIG. 10 is a diagram illustrating an example of a recognition result. [Figure 10] FIG. 10 is a diagram for explaining detection by tracking. [Figure 11] 10 is a flowchart illustrating an operation of the information processing device. [Figure 12] FIG. 10 is a diagram for explaining update criteria. [Figure 13] FIG. 10 is a diagram illustrating a frame to be tracked. [Figure 14] FIG. 1 is a diagram illustrating a configuration of an information processing system. [Figure 15] 10 is a flowchart illustrating an operation of the information processing device. [Figure 16] 10 is a flowchart illustrating an operation of the server. [Figure 17] FIG. 1 is a diagram illustrating a configuration of an information processing system. [Figure 18] 10 is a flowchart illustrating an operation of the information processing device. [Figure 19] 10 is a flowchart illustrating an operation of the server. [Figure 20] FIG. 1 illustrates an example of the configuration of a personal computer. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, modes for carrying out the present technology (hereinafter referred to as embodiments) will be described.

[0015] <Example of vehicle control system configuration> FIG. 1 is a block diagram showing an example of the configuration of a vehicle control system 11, which is an example of a mobility device control system to which the present technology is applied.

[0016] The vehicle control system 11 is provided in the vehicle 1 and performs processing related to driving assistance and automatic driving of the vehicle 1.

[0017] The vehicle control system 11 includes a processor 21, a communication unit 22, a map information storage unit 23, a GNSS (Global Navigation Satellite System) receiving unit 24, an external recognition sensor 25, an in-vehicle sensor 26, a vehicle sensor 27, a recording unit 28, a driving assistance / autonomous driving control unit 29, a DMS (Driver Monitoring System) 30, an HMI (Human Machine Interface) 31, and a vehicle control unit 32.

[0018] The processor 21, communication unit 22, map information storage unit 23, GNSS receiving unit 24, external recognition sensor 25, in-vehicle sensor 26, vehicle sensor 27, recording unit 28, cruise assist / autonomous driving control unit 29, driver monitoring system (DMS) 30, human-machine interface (HMI) 31, and vehicle control unit 32 are interconnected via a communication network 41. The communication network 41 is configured by an in-vehicle communication network or bus conforming to any standard such as CAN (Controller Area Network), LIN (Local Interconnect Network), LAN (Local Area Network), FlexRay (registered trademark), or Ethernet (registered trademark). Note that the components of the vehicle control system 11 may be directly connected to each other via, for example, near field communication (NFC) or Bluetooth (registered trademark) without using the communication network 41.

[0019] In the following description, when each unit of the vehicle control system 11 communicates via the communication network 41, the description of the communication network 41 will be omitted. For example, when the processor 21 and the communication unit 22 communicate via the communication network 41, it will simply be described that the processor 21 and the communication unit 22 communicate with each other.

[0020] The processor 21 is configured by various processors such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ECU (Electronic Control Unit), etc. The processor 21 controls the vehicle control system 11 as a whole.

[0021] The communication unit 22 communicates with various devices inside and outside the vehicle, other vehicles, servers, base stations, etc., and transmits and receives various types of data. As communication with the outside of the vehicle, for example, the communication unit 22 receives from the outside a program for updating software that controls the operation of the vehicle control system 11, map information, traffic information, information about the surroundings of the vehicle 1, etc. For example, the communication unit 22 transmits information about the vehicle 1 (for example, data indicating the state of the vehicle 1, the recognition result by the recognition unit 73, etc.), information about the surroundings of the vehicle 1, etc., to the outside. For example, the communication unit 22 performs communication compatible with a vehicle emergency notification system such as e-call.

[0022] There is no particular limitation on the communication method of the communication unit 22. A plurality of communication methods may be used.

[0023] For example, the communication unit 22 performs wireless communication with devices inside the vehicle using a communication method such as wireless LAN, Bluetooth (registered trademark), NFC, or WUSB (Wireless USB). For example, the communication unit 22 performs wired communication with devices inside the vehicle using a communication method such as USB (Universal Serial Bus), HDMI (High-Definition Multimedia Interface, registered trademark), or MHL (Mobile High-Definition Link) via a connection terminal (and a cable, if necessary) not shown.

[0024] Here, the in-vehicle device refers to, for example, a device that is in the vehicle and is not connected to the communication network 41. Examples of such devices include mobile devices and wearable devices carried by passengers such as the driver, and information devices that are brought into the vehicle and temporarily installed.

[0025] For example, the communication unit 22 communicates with a server or the like existing on an external network (e.g., the Internet, a cloud network, or an operator-specific network) via a base station or an access point using a wireless communication method such as 4G (fourth generation mobile communication system), 5G (fifth generation mobile communication system), LTE (Long Term Evolution), or DSRC (Dedicated Short Range Communications).

[0026] For example, the communication unit 22 communicates with a terminal present in the vicinity of the vehicle (for example, a terminal of a pedestrian or a store, or an MTC (Machine Type Communication) terminal) using P2P (Peer To Peer) technology. For example, the communication unit 22 performs V2X communication. V2X communication includes, for example, vehicle-to-vehicle communication with another vehicle, vehicle-to-infrastructure communication with a roadside unit or the like, vehicle-to-home communication, and vehicle-to-pedestrian communication with a terminal carried by a pedestrian or the like.

[0027] For example, the communication unit 22 receives electromagnetic waves transmitted by a road traffic information and communication system (VICS (Vehicle Information and Communication System), registered trademark) such as a radio beacon, an optical beacon, or FM multiplex broadcasting.

[0028] The map information storage unit 23 stores maps acquired from an external source and maps created by the vehicle 1. For example, the map information storage unit 23 stores a three-dimensional high-precision map, a global map that is less accurate than a high-precision map and covers a wide area, and the like.

[0029] Examples of high-precision maps include dynamic maps, point cloud maps, and vector maps (also referred to as ADAS (Advanced Driver Assistance System) maps). Dynamic maps are maps consisting of four layers of information, for example, dynamic information, quasi-dynamic information, quasi-static information, and static information, and are provided from an external server or the like. Point cloud maps are maps configured from point clouds (point cloud data). Vector maps are maps in which information such as the positions of lanes and traffic lights is associated with the point cloud map. The point cloud map and the vector map may be provided from an external server or the like, or may be created by the vehicle 1 based on sensing results from the radar 52, the LiDAR 53, and the like as a map for matching with a local map (described later), and stored in the map information storage unit 23. Furthermore, when a high-precision map is provided from an external server or the like, map data of, for example, an area of ​​several hundred square meters related to the planned route along which the vehicle 1 will travel is acquired from the server or the like in order to reduce communication capacity.

[0030] The GNSS receiver 24 receives GNSS signals from GNSS satellites and supplies them to the driving assistance / automated driving control unit 29 .

[0031] The external recognition sensor 25 includes various sensors used to recognize the situation outside the vehicle 1, and supplies sensor data from each sensor to each part of the vehicle control system 11. The type and number of sensors included in the external recognition sensor 25 are arbitrary.

[0032] For example, the external recognition sensor 25 includes a camera 51, a radar 52, a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) 53, and an ultrasonic sensor 54. The number of cameras 51, radars 52, LiDARs 53, and ultrasonic sensors 54 is arbitrary, and an example of the sensing area of ​​each sensor will be described later.

[0033] The camera 51 may be a camera of any imaging method, such as a ToF (Time Of Flight) camera, a stereo camera, a monocular camera, or an infrared camera, as needed.

[0034] Furthermore, for example, the external recognition sensor 25 includes an environmental sensor for detecting the weather, climate, brightness, etc. The environmental sensor includes, for example, a raindrop sensor, a fog sensor, a sunlight sensor, a snow sensor, an illuminance sensor, etc.

[0035] Furthermore, for example, the external recognition sensor 25 includes a microphone used to detect sounds around the vehicle 1 and the positions of sound sources.

[0036] The interior sensor 26 includes various sensors for detecting information inside the vehicle, and supplies sensor data from each sensor to each unit of the vehicle control system 11. The interior sensor 26 may include any type and any number of sensors.

[0037] For example, the in-vehicle sensors 26 include a camera, radar, a seating sensor, a steering wheel sensor, a microphone, a biometric sensor, etc. The camera may be a camera of any imaging method, such as a ToF camera, a stereo camera, a monocular camera, or an infrared camera. The biometric sensor is provided, for example, on a seat or steering wheel, and detects various types of biometric information of a passenger such as a driver.

[0038] The vehicle sensor 27 includes various sensors for detecting the state of the vehicle 1, and supplies sensor data from each sensor to each unit of the vehicle control system 11. The type and number of sensors included in the vehicle sensor 27 are arbitrary.

[0039] For example, the vehicle sensor 27 includes a speed sensor, an acceleration sensor, an angular velocity sensor (gyro sensor), and an inertial measurement unit (IMU). For example, the vehicle sensor 27 includes a steering angle sensor that detects the steering angle of the steering wheel, a yaw rate sensor, an accelerator sensor that detects the amount of accelerator pedal operation, and a brake sensor that detects the amount of brake pedal operation. For example, the vehicle sensor 27 includes a rotation sensor that detects the number of rotations of the engine or motor, an air pressure sensor that detects tire air pressure, a slip ratio sensor that detects tire slip ratio, and a wheel speed sensor that detects the rotation speed of the wheels. For example, the vehicle sensor 27 includes a battery sensor that detects the remaining battery level and temperature, and an impact sensor that detects external impacts.

[0040] The recording unit 28 includes, for example, a magnetic storage device such as a read-only memory (ROM), a random access memory (RAM), or a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The recording unit 28 records various programs and data used by each component of the vehicle control system 11. For example, the recording unit 28 records a rosbag file including messages transmitted and received by a robot operating system (ROS) on which an application program related to autonomous driving runs. For example, the recording unit 28 includes an event data recorder (EDR) or a data storage system for automated driving (DSSAD), and records information about the vehicle 1 before and after an event such as an accident.

[0041] The driving assistance / automatic driving control unit 29 controls driving assistance and automatic driving of the vehicle 1. For example, the driving assistance / automatic driving control unit 29 includes an analysis unit 61, an action planning unit 62, and an operation control unit 63.

[0042] The analysis unit 61 performs an analysis process of the vehicle 1 and the surrounding situation. The analysis unit 61 includes a self-position estimation unit 71, a sensor fusion unit 72, and a recognition unit 73.

[0043] The self-position estimation unit 71 estimates the self-position of the vehicle 1 based on the sensor data from the external recognition sensor 25 and the high-precision map stored in the map information storage unit 23. For example, the self-position estimation unit 71 generates a local map based on the sensor data from the external recognition sensor 25 and estimates the self-position of the vehicle 1 by matching the local map with the high-precision map. The position of the vehicle 1 is based on, for example, the center of the rear wheel pair axle.

[0044] The local map is, for example, a three-dimensional high-precision map or an occupancy grid map created using a technology such as SLAM (Simultaneous Localization and Mapping). The three-dimensional high-precision map is, for example, the point cloud map described above. The occupancy grid map is a map in which the three-dimensional or two-dimensional space around the vehicle 1 is divided into grids of a predetermined size and the occupancy state of an object is indicated on a grid-by-grid basis. The occupancy state of an object is indicated, for example, by the presence or absence of an object and its probability of existence. The local map is also used, for example, in the detection process and recognition process of the situation outside the vehicle 1 by the recognition unit 73.

[0045] The self-position estimation unit 71 may estimate the self-position of the vehicle 1 based on the GNSS signal and sensor data from the vehicle sensor 27.

[0046] The sensor fusion unit 72 performs sensor fusion processing to obtain new information by combining multiple different types of sensor data (for example, image data supplied from the camera 51 and sensor data supplied from the radar 52). Methods for combining different types of sensor data include integration, fusion, and association.

[0047] The recognition unit 73 performs detection processing and recognition processing of the situation outside the vehicle 1.

[0048] For example, the recognition unit 73 performs detection processing and recognition processing of the situation outside the vehicle 1 based on information from the external recognition sensor 25, information from the self-position estimation unit 71, information from the sensor fusion unit 72, etc.

[0049] Specifically, for example, the recognition unit 73 performs detection processing and recognition processing of objects around the vehicle 1. The object detection processing is, for example, processing to detect the presence or absence, size, shape, position, movement, etc. of an object. The object recognition processing is, for example, processing to recognize attributes such as the type of object, or to identify a specific object. However, the detection processing and the recognition processing are not necessarily clearly separated, and may overlap.

[0050] For example, the recognition unit 73 performs clustering to classify a point cloud based on sensor data such as LiDAR or radar into clusters of points, thereby detecting objects around the vehicle 1. This allows the presence, size, shape, and position of objects around the vehicle 1 to be detected.

[0051] For example, the recognition unit 73 performs tracking to follow the movement of clusters of point clouds classified by clustering, thereby detecting the movement of objects around the vehicle 1. As a result, the speed and traveling direction (movement vector) of the objects around the vehicle 1 are detected.

[0052] For example, the recognition unit 73 performs object recognition processing such as semantic segmentation on the image data supplied from the camera 51 to recognize the type of object around the vehicle 1.

[0053] Note that objects to be detected or recognized may include, for example, vehicles, people, bicycles, obstacles, structures, roads, traffic lights, traffic signs, road markings, and the like.

[0054] For example, the recognition unit 73 performs a recognition process of traffic rules around the vehicle 1 based on the map stored in the map information storage unit 23, the estimation result of the vehicle's own position, and the recognition result of objects around the vehicle 1. Through this process, for example, the positions and states of traffic signals, the contents of traffic signs and road markings, the contents of traffic regulations, and lanes on which travel is permitted are recognized.

[0055] For example, the recognition unit 73 performs a process of recognizing the environment around the vehicle 1. The environment to be recognized may include, for example, the weather, temperature, humidity, brightness, and road surface conditions.

[0056] The behavior planning unit 62 creates a behavior plan for the vehicle 1. For example, the behavior planning unit 62 creates the behavior plan by performing route planning and route following processing.

[0057] Global path planning is a process for planning a rough route from the start to the goal. This route planning also includes a process called trajectory planning, which takes into account the motion characteristics of the vehicle 1 on the route planned by the route planning and generates a trajectory (local path planning) that allows the vehicle 1 to proceed safely and smoothly in the vicinity of the vehicle 1.

[0058] Path following is a process of planning an operation for safely and accurately traveling along a route planned by a route planner within a planned time. For example, a target speed and a target angular velocity of the vehicle 1 are calculated.

[0059] The operation control unit 63 controls the operation of the vehicle 1 in order to realize the action plan created by the action planning unit 62.

[0060] For example, the operation control unit 63 controls the steering control unit 81, the brake control unit 82, and the drive control unit 83 to perform acceleration / deceleration control and direction control so that the vehicle 1 travels along the trajectory calculated by the trajectory plan. For example, the operation control unit 63 performs cooperative control aimed at realizing ADAS functions such as collision avoidance or impact mitigation, following driving, vehicle speed maintenance driving, collision warning for the vehicle itself, and lane departure warning for the vehicle itself. For example, the operation control unit 63 performs cooperative control aimed at automatic driving, which allows the vehicle to travel autonomously without relying on driver operation.

[0061] The DMS 30 performs processes such as authenticating the driver and recognizing the driver's state based on the sensor data from the in-vehicle sensors 26 and the input data input to the HMI 31. Examples of the driver's state to be recognized include physical condition, alertness, concentration, fatigue, gaze direction, level of intoxication, driving operation, and posture.

[0062] The DMS 30 may be configured to perform authentication processing for passengers other than the driver and recognition processing for the conditions of the passengers. Furthermore, for example, the DMS 30 may be configured to perform recognition processing for the conditions inside the vehicle based on sensor data from the in-vehicle sensor 26. Possible conditions inside the vehicle to be recognized include, for example, temperature, humidity, brightness, and odor.

[0063] The HMI 31 is used to input various data, instructions, etc., generates input signals based on the input data, instructions, etc., and supplies them to each component of the vehicle control system 11. For example, the HMI 31 includes operation devices such as a touch panel, buttons, a microphone, switches, and levers, as well as operation devices that allow input by voice, gestures, etc., other than manual operation. The HMI 31 may be, for example, a remote control device that uses infrared or other radio waves, or an externally connected device such as a mobile device or wearable device that supports operation of the vehicle control system 11.

[0064] The HMI 31 also performs output control to generate and output visual information, auditory information, and tactile information for the occupant or the outside of the vehicle, as well as to control the output content, output timing, output method, etc. Visual information is information displayed by images or light, such as an operation screen, a status display of the vehicle 1, a warning display, and a monitor image showing the situation around the vehicle 1. Auditory information is information displayed by sound, such as guidance, warning sounds, and warning messages. Tactile information is information imparted to the occupant's sense of touch by force, vibration, movement, etc.

[0065] Possible devices for outputting visual information include, for example, a display device, a projector, a navigation device, an instrument panel, a CMS (Camera Monitoring System), an electronic mirror, a lamp, etc. The display device may be a device having a normal display, or may be a device that displays visual information within the field of view of a passenger, such as a head-up display, a see-through display, or a wearable device with an AR (Augmented Reality) function.

[0066] Possible devices for outputting auditory information include, for example, audio speakers, headphones, and earphones.

[0067] A device that outputs tactile information may be, for example, a haptic element that uses haptic technology, which may be provided on a steering wheel, a seat, or the like.

[0068] The vehicle control unit 32 controls each part of the vehicle 1. The vehicle control unit 32 includes a steering control unit 81, a brake control unit 82, a drive control unit 83, a body system control unit 84, a light control unit 85, and a horn control unit 86.

[0069] The steering control unit 81 detects and controls the state of the steering system of the vehicle 1. The steering system includes, for example, a steering mechanism including a steering wheel, an electric power steering, etc. The steering control unit 81 includes, for example, a control unit such as an ECU that controls the steering system, an actuator that drives the steering system, etc.

[0070] The brake control unit 82 detects and controls the state of the brake system of the vehicle 1. The brake system includes, for example, a brake mechanism including a brake pedal, an ABS (Antilock Brake System), etc. The brake control unit 82 includes, for example, a control unit such as an ECU that controls the brake system, and an actuator that drives the brake system.

[0071] The drive control unit 83 detects and controls the state of the drive system of the vehicle 1. The drive system includes, for example, an accelerator pedal, a drive force generating device for generating drive force such as an internal combustion engine or a drive motor, and a drive force transmission mechanism for transmitting the drive force to the wheels. The drive control unit 83 includes, for example, a control unit such as an ECU for controlling the drive system, and an actuator for driving the drive system.

[0072] The body system control unit 84 detects and controls the states of the body system systems of the vehicle 1. The body system systems include, for example, a keyless entry system, a smart key system, a power window device, a power seat, an air conditioning system, an airbag, a seat belt, a shift lever, etc. The body system control unit 84 includes, for example, a control unit such as an ECU that controls the body system systems, and an actuator that drives the body system systems.

[0073] The light control unit 85 detects and controls the states of various lights of the vehicle 1. Examples of lights to be controlled include headlights, backlights, fog lights, turn signals, brake lights, projections, and bumper displays. The light control unit 85 includes a control unit such as an ECU that controls the lights, an actuator that drives the lights, and the like.

[0074] The horn control unit 86 detects and controls the state of the car horn of the vehicle 1. The horn control unit 86 includes, for example, a control unit such as an ECU that controls the car horn, and an actuator that drives the car horn.

[0075] FIG. 2 is a diagram showing an example of a sensing area by the camera 51, the radar 52, the LiDAR 53, and the ultrasonic sensor 54 of the external recognition sensor 25 in FIG.

[0076] Sensing area 101F and sensing area 101B are examples of sensing areas of the ultrasonic sensor 54. Sensing area 101F covers the periphery of the front end of the vehicle 1. Sensing area 101B covers the periphery of the rear end of the vehicle 1.

[0077] The sensing results in the sensing area 101F and the sensing area 101B are used for, for example, parking assistance for the vehicle 1.

[0078] Sensing area 102F to sensing area 102B show examples of sensing areas of a short-range or medium-range radar 52. Sensing area 102F covers a position farther in front of the vehicle 1 than sensing area 101F. Sensing area 102B covers a position farther behind the vehicle 1 than sensing area 101B. Sensing area 102L covers the surrounding area behind the left side of the vehicle 1. Sensing area 102R covers the surrounding area behind the right side of the vehicle 1.

[0079] The sensing results in sensing area 102F are used, for example, to detect vehicles, pedestrians, etc. that are present in front of the vehicle 1. The sensing results in sensing area 102B are used, for example, for a collision prevention function behind the vehicle 1. The sensing results in sensing area 102L and sensing area 102R are used, for example, to detect objects in blind spots on the sides of the vehicle 1.

[0080] Sensing area 103F to sensing area 103B show examples of sensing areas sensed by camera 51. Sensing area 103F covers a position farther in front of vehicle 1 than sensing area 102F. Sensing area 103B covers a position farther behind vehicle 1 than sensing area 102B. Sensing area 103L covers the periphery of the left side of vehicle 1. Sensing area 103R covers the periphery of the right side of vehicle 1.

[0081] The sensing results in the sensing area 103F are used, for example, for recognizing traffic lights and traffic signs, lane departure prevention support systems, etc. The sensing results in the sensing area 103B are used, for example, for parking assistance and surround view systems, etc. The sensing results in the sensing areas 103L and 103R are used, for example, for surround view systems, etc.

[0082] Sensing area 104 shows an example of the sensing area of ​​LiDAR 53. Sensing area 104 covers a position farther ahead of vehicle 1 than sensing area 103F. On the other hand, sensing area 104 has a narrower range in the left-right direction than sensing area 103F.

[0083] The sensing results in the sensing area 104 are used for, for example, emergency braking, collision avoidance, pedestrian detection, and the like.

[0084] Sensing area 105 shows an example of the sensing area of ​​the long-distance radar 52. Sensing area 105 covers a position further ahead of the vehicle 1 than sensing area 104. On the other hand, sensing area 105 has a narrower range in the left-right direction than sensing area 104.

[0085] The sensing results in the sensing area 105 are used for, for example, adaptive cruise control (ACC).

[0086] The sensing area of ​​each sensor may have various configurations other than that shown in Figure 2. Specifically, the ultrasonic sensor 54 may also sense the sides of the vehicle 1, and the LiDAR 53 may sense the area behind the vehicle 1.

[0087] <Configuration example of information processing device> 3 is a diagram showing the configuration of an embodiment of an information processing device to which the present technology is applied. The information processing device 110 is mounted on, for example, a vehicle 1, and can be used as a device that analyzes captured images and recognizes predetermined objects such as people and cars. When performing recognition processing, the information processing device 110 in this embodiment performs recognition using a recognizer to which a learning model such as machine learning is applied, and has a function of updating the recognizer to reduce false detections.

[0088] The information processing device 110 shown in FIG. 3 includes an image acquisition unit 121, a recognition processing unit 122, an extraction unit 123, a recognition target tracking unit 124, a label assignment unit 125, a re-learning unit 126, and a recognizer update unit 127.

[0089] The image acquisition unit 121 acquires image data of an image captured by an imaging unit (not shown) that captures an image. The image acquisition unit 121 acquires an image captured by, for example, a camera 51 (FIG. 1). The recognition processing unit 122 analyzes the image acquired by the image acquisition unit 121 and recognizes a predetermined object such as a person or a car using a recognizer (learning model). The recognition processing unit 122 executes recognition processing using a recognizer to which a learning model that performs recognition processing on input data is applied.

[0090] When the information processing device 110 is installed in a vehicle, for example, the recognition results obtained by the information processing device 110 can be used for semi-automated driving to assist in steering and braking to avoid the recognized object.

[0091] The recognition results from the recognition processing unit 122 of the information processing device 110 are supplied to the extraction unit 123. The extraction unit 123 extracts recognition results for which the conditions for updating the recognizer, which will be described later, are satisfied. The extraction results from the extraction unit 123 are supplied to the recognition target tracking unit 124. The recognition target tracking unit 124 tracks the extracted recognition results over multiple frames. These multiple frames are frames captured in a reverse direction (past direction) in chronological order, and the recognition target tracking unit 124 executes processing to track the recognition target in a reverse direction in chronological order.

[0092] The tracking results by the recognition target tracking unit 124 are supplied to the label assignment unit 125. The label assignment unit 125 assigns labels to the tracked recognition targets. The labeled recognition targets are supplied to the re-learning unit 126. The re-learning unit 126 re-trains the recognizer using the labeled recognition targets. The new recognizer generated by the re-learning is supplied to the recognizer update unit 127. The recognizer update unit 127 updates the recognizer of the recognition processing unit 122 to the recognizer re-trained by the re-learning unit 126.

[0093] The re-learning unit 126 has a function of executing learning of parameters (parameters that may be referred to as model parameters) of the recognizer of the recognition processing unit 122. For the learning, various machine learning techniques using neural networks such as RNN (Recurrent Neural Network) and CNN (Convolutional Neural Network) can be used.

[0094] The learning process will be further explained with reference to Fig. 4. Labeled images, in which labels for classifying multiple subjects captured in the images have been created in advance, are input to the recognizer. For example, the recognizer performs image recognition on the labeled images, recognizes the multiple subjects captured in the labeled images, and outputs recognition results in which each subject is classified.

[0095] The recognition result output from the recognizer is compared with the correct label for the labeled image, and feedback is provided to the recognizer to bring the recognition result closer to the correct label. In this way, the correct label is used to train the recognizer (the learning model) so that it can perform more accurate recognition. The recognition processing unit 122 can be configured to perform recognition processing using the trained learning model.

[0096] The learning process shown here is just an example, and the present technology can be applied to cases where learning is performed by other learning processes or where a recognizer obtained by other learning processes is used. As a learning process, it is also possible to apply a learning process that does not use labeled images or correct labels to the present technology.

[0097] The re-learning unit 126 re-learns the recognizer (learning model) through the learning process shown in Fig. 4. For example, the recognizer is re-learned using the recognition results from the recognition processing unit 122 as labeled images and the images to which labels have been assigned by the label assignment unit 125 as correct labels.

[0098] Re-learning is performed by using a frame captured at a predetermined time as a reference and using several frames captured before the reference frame. Re-learning is performed to generate a recognizer that reduces false detections. False detections include when a recognition target, such as an object like a person or a car, is not detected even though it appears in the image, or when it is detected as the wrong object, such as when a person is detected as a car.

[0099] Such erroneous detection and re-learning will be explained below with reference to an example of captured image, taking as an example a case where an image captured by an in-vehicle camera is processed.

[0100] 5 to 9 are diagrams showing examples of images (frames) captured at times t1, t2, t3, t4, and t5, respectively. 5 to 9 also show detection frames displayed for objects recognized (detected) by processing frames F1 to F5 by the recognition processing unit 122. The explanation will continue assuming that time passes in the order of time t1, time t2, time t3, time t4, and time t5, in other words, time t1 is the oldest (past) and time t5 is the newest (present).

[0101] Cars C11 and C12 are imaged on the left side of frame F1 shown in Fig. 5, and car C13 is imaged in front. Also, person H11 is imaged on the right side. By processing frame F1 by the recognition processing unit 122 (Fig. 3), cars C11, C12, and C13 are detected. The detected objects are surrounded by rectangular detection frames.

[0102] 5, car C11 is surrounded by a detection frame BC11, car C12 is surrounded by a detection frame BC12, and car C13 is surrounded by a detection frame BC13. In the example shown in Fig. 5, person H11 is imaged but not detected, so no detection frame is displayed.

[0103] As a method for detecting a predetermined object such as a vehicle or a person, semantic segmentation, instance segmentation, panoptic segmentation, etc. can be applied.

[0104] Semantic segmentation is a method of classifying all pixels in an image into classes and labeling each pixel. Instance segmentation is a method of dividing regions into objects and recognizing the type of object. Panoptic segmentation is a method that combines semantic segmentation and instance segmentation, and is a method that can recognize the type of object and label all pixels.

[0105] Here, the description will be continued assuming that panoptic segmentation is applied, but this technology can also be applied to the above-mentioned methods other than panoptic segmentation, or to recognition methods not exemplified here.

[0106] When recognition is performed using panoptic segmentation and the results are displayed as an image such as that shown in Fig. 5, pixels with the same label can be displayed in the same color. For example, different objects can be displayed in different colors, such as displaying pixels labeled as car C11 in red and pixels labeled as car C12 in blue. Although colors are not shown in Figs. 5 to 9, different objects are detected as different objects and are displayed in different colors.

[0107] In the frame F1 shown in FIG. 5, a person H11 is captured in an image, but an erroneous detection occurs in which the person H11 is not detected.

[0108] 6 is a diagram showing an example of frame F2 captured at time t2, which is later than time t1 (a predetermined time has elapsed). Because the car has moved forward, cars C11 and C12, which were captured in frame F1 (FIG. 5), are now outside the captured range and are not captured in frame F2. Car C23 corresponds to car C13 in frame F1, is also detected in frame 2, and is surrounded by detection frame BC23.

[0109] In frame F2, person H21, who corresponds to person H11 (Figure 5), is also imaged but is not detected. In frame F2, people H22 and H23 are newly detected and are surrounded by detection frames BH22 and BH23, respectively.

[0110] Detection frames can be displayed in different colors and line types depending on the label. Figure 6 shows an example in which a solid detection frame is displayed for a recognition result labeled "car," and a dotted detection frame is displayed for a recognition result labeled "person."

[0111] 7 is a diagram showing an example of a frame F3 captured at time t3, which is after time t2. Frame F3 captures a person H31 corresponding to person H11 (FIG. 5) and person H21 (FIG. 6), and a person H32 corresponding to person H22 (FIG. 6). Person H31 and person H32 have each been detected. Person H31 was erroneously detected as a car, so it is labeled as a car, and the detection frame BC31 that is displayed for cars is displayed to surround person H31. Person H32 was correctly detected as a person, so the detection frame BH32 that is displayed for people is displayed to surround person H32.

[0112] 8 is a diagram showing an example of frame F4 captured at time t4, which is after time t3. Frame F4 captures person H41, which corresponds to person H11 (FIG. 5), person H21 (FIG. 6), and person H31 (FIG. 7), and person H44. Since person H41 and person H44 are correctly detected as people, detection frames BH41 and BH44, which are displayed for people, are displayed, respectively.

[0113] 9 is a diagram showing an example of frame F5 captured at time t5, which is after time t4. Frame F5 captures person H51, which corresponds to person H11 (FIG. 5), person H21 (FIG. 6), person H31 (FIG. 7), and person H41 (FIG. 8), and person H54, which corresponds to person H44 (FIG. 9). Person H51 and person H54 have both been correctly detected as people, and therefore detection frames BH51 and BH54, which are displayed for people, are displayed, respectively.

[0114] Consider the case where frames F1 to F5 are captured in this way and the recognition processing results are output. Fig. 10 is a diagram showing frames F1 and F5 side by side. In Fig. 10, attention is focused on people H11 and H51. In frame F1, person H11 is captured but not detected. In frame F5, person H51 is captured and detected.

[0115] The person H11 captured in the frame F1 is not detected at the time of the frame F1. In other words, in the frame F1, an erroneous detection occurs in which the person H11 that should have been detected is not detected.

[0116] Person H11 is detected as person H51 in frame F5. Person H11 is imaged as person H21 (frame F2), person H31 (frame F3), person H41 (frame F4), and person H51 (frame H5). That is, person H11 is imaged consecutively from frames F1 to F5. In this case, if person H51 is tracked in the order of frame F5, frame F4, frame F3, frame F2, and frame F1, it can be detected (tracked) in the order of person H51, person H41, person H31, person H21, and person H11.

[0117] By tracking backward, it is possible to label the person corresponding to person H51 in each frame. For example, in frame F1, person H11 can be labeled. By training using this labeled frame F1, it is possible to generate a recognizer that can detect and label person H11 from images such as frame F1.

[0118] In frame F3 (Figure 7), person H31 is erroneously detected as a car, but by tracking frame F3 from frames F5 and F4, person H51, person H41, and person H31 are tracked, and person H31 is labeled as a person. By performing training using frame F3, in which person H31 is labeled as a person, it is possible to generate a recognizer that can detect person H31 from an image such as frame F3 and correctly label it as a person.

[0119] In frame F5, people H51 and H54 are captured. A person corresponding to person H54 is not detected in frames F3 to F1. If a person corresponding to person H54 is captured in frames F3 to F1 as well, the person corresponding to person H54 can be detected and labeled in frames F3 to F1 by tracking the person corresponding to person H54 in the past. If a label is assigned to the person corresponding to person H54 in frames F3 to F1 as a result of tracking, then a recognizer can be generated that can detect the person corresponding to person H54 even in images such as frames F3 to F1 by performing learning using those frames F3 to F1.

[0120] In this way, by tracing backward in time, it is possible to detect objects that were not detected before, or to correctly detect objects that were incorrectly recognized. By tracing backward in time like this, it is possible to perform learning using newly labeled images. As a result, it is possible to generate a recognizer (learning model) with fewer false positives.

[0121] <About processing by information processing devices> The information processing device 110 executes processing related to such learning (relearning). The processing of the information processing device 110 (FIG. 3) will be described with reference to the flowchart shown in FIG.

[0122] In step S111, the image acquisition unit 121 acquires image data (frame). In step S112, the recognition processing unit 122 executes recognition processing using a recognizer to which a learning model that performs recognition processing is applied, by analyzing an image based on the image data acquired by the image acquisition unit 121. The recognition processing performed by the recognition processing unit 122 is processing using a recognizer that recognizes predetermined objects such as people and cars, and is, for example, processing to detect a car C11 from frame F1 and assign a label of "car" to it, as described with reference to FIG. 5.

[0123] In step S113, the extraction unit 123 extracts recognition results that satisfy an update criterion. The update criterion is a criterion for determining whether or not the data requires updating of the recognizer. The update criterion is a criterion for determining that re-learning should be performed if there is a recognition result that satisfies the criterion described below among the recognition results.

[0124] Here, an object detected by the recognition processing of the recognition processing unit 122 is referred to as a recognition result, and a recognition result extracted by the extraction unit 123 is referred to as a recognition target. As will be described later, a recognition target is a recognition result that is the target of tracking. The update criteria will be described with reference to FIG. 12.

[0125] As shown in A of FIG. 12, a first update criterion is set such that if there is a recognition result whose size Bx is equal to or greater than x% of the area of ​​the image Fx, that recognition result is extracted as a recognition target. The size of the recognition result can be, for example, the area of ​​the region surrounded by the detection frame BC11 of the car C11 in the frame F1 (FIG. 5). Instead of the area, the height or width may be used, and if the height or width is equal to or greater than a predetermined size, the recognition result is extracted as a recognition target. The area of ​​the image Fx is, for example, the image size of the frame F1.

[0126] The first update criterion is a criterion for setting an object detected at a certain size as a tracking target, that is, in this case, as a recognition target. Generally, when a predetermined object is detected as being large, the reliability of the detection result is higher and the possibility of erroneous detection is lower than when the size is small. Therefore, the first update criterion is set so that objects detected with such high accuracy can be re-learned as recognition targets.

[0127] Note that the first update criterion may have a different value of x% depending on the recognition result. For example, if the same x% value is used when the recognition result is a person and when the recognition result is a car, the value for a car is larger than that for a person, so it is likely that the first update criterion will be met when the recognition result is a car, but it is unlikely that the first update criterion will be met when the recognition result is a person. Therefore, the value of x% may be made variable depending on the label of the recognition result, and a different x may be used for each recognition result to determine whether the first update criterion is met.

[0128] As shown in B of Fig. 12, a second update criterion is set such that if there is a recognition result that is y% or more away from a side of image Fy, that recognition result is extracted as a recognition target. Image Fy refers to one frame, and one side of the frame refers to, for example, the left or right side as shown in B of Fig. 12. The distance y% or more from the side refers to, for example, the ratio when the horizontal length of the frame (the distance from the right side to the left side) is 100%.

[0129] For example, referring to frame F1 (FIG. 5), the car C11 is captured in a partially cut-off state. The second update criterion is a criterion for not treating such an object captured in a partially cut-off state as a recognition target.

[0130] Although the horizontal distance (distance from the left and right sides) has been described as an example in FIG. 12B, the vertical distance (distance from the top and bottom sides) may also be used. Criteria may be set for both the horizontal distance and the vertical distance. As with the first update criterion, different y% may be used depending on the label.

[0131] The extraction unit 123 extracts a recognition result that satisfies the first update criterion and / or the second update criterion, and if extracted, sets the recognition result as a recognition target to be tracked. When a recognition target is set, several frames in the past are set as frames to be tracked, based on the frame from which the recognition target was extracted.

[0132] Referring to FIG. 13, for example, if a recognition result that satisfies the first update criterion or / and the second update criterion is extracted at time T0, 0-m The m frames captured up to this point are set as frames to be processed.

[0133] The third update criterion is that if there is a recognition result detected over m consecutive frames, that recognition result is extracted as a recognition target. If an object is detected over m frames, it can be said that the object has been detected with high accuracy. If there is a recognition result detected with such high accuracy, that recognition result is extracted as a recognition target.

[0134] In order to determine whether the third update criterion is satisfied, the extraction unit 123 has a storage unit (not shown) that stores multiple frames. When the extraction unit 123 extracts a recognition result that satisfies the third update criterion, as shown in Fig. 13, n frames captured earlier than m frames are set as frames to be processed. Referring to Fig. 13, for example, when a recognition result that satisfies the third update criterion is extracted at time T0, n frames captured earlier than m frames are set as frames to be processed. 0-m In this case, there are consecutive recognition results detected in m frames captured up to time T 0-m time T before 0-m-n The n frames captured up to are set as frames to be processed.

[0135] Note that m frames (number of frames) in the third update criterion may be a fixed value or a variable value. When the number of frames is a variable value, m may be set based on, for example, the vehicle speed, frame rate, size of the recognition result, etc. From this information, the size of the recognition result may be determined as a height h min , width w min Alternatively, the number of frames required to reach that size may be estimated and set as m.

[0136] For example, if the vehicle speed is high, the distance traveled per unit time will be long and the objects captured will be replaced more frequently, resulting in fewer objects remaining in multiple frames. If the vehicle speed is high, it may be difficult to extract objects to be recognized unless m in m frames is made small. On the other hand, if the vehicle speed is slow, there will be more objects remaining in multiple frames, and if m in m frames is not made large, many objects to be recognized will be extracted, which may result in frequent re-learning.

[0137] Taking this into consideration, m in m frames may be set according to the vehicle speed, frame rate, etc., as described above.

[0138] The size of the recognition result is height h min , width w min is the size when a predetermined recognition result is first captured or detected. It is possible to estimate how many frames ago such a size was reached, in other words, how many frames ago the predetermined recognition result was recognized, and set the estimated number of frames as m. This m can be estimated from information such as the vehicle speed, frame rate, and size of the recognition result.

[0139] The m in the m frames may be set by referring to a correspondence table that assigns m to the size of the recognition target, or may be calculated using a predetermined function.

[0140] The fourth update criterion is a combination of the first to third update criteria described above.

[0141] A fourth update criterion may be set by combining the first and second update criterion, which extracts as recognition targets recognition results whose size is x% or more of the frame size and whose distance from the frame edge is y% or more. In this case, objects that are detected at a certain size and are likely to be captured without being cut off are extracted.

[0142] A fourth update criterion may be set by combining the first and third update criterion, which states that when a recognition result whose size is equal to or greater than x% of the frame size is detected for m consecutive frames, that recognition result is extracted as a recognition target. In this case, an object that is detected at a certain size and is stably detected over several frames is extracted.

[0143] A fourth update criterion may be set by combining the second and third update criterion, such that when a recognition result at a distance of y% or more from a frame edge is detected for m consecutive frames, that recognition result is extracted as a recognition target. In this case, there is a high possibility that the object was captured in an uncut state, and an object that has been stably detected over several frames is extracted.

[0144] A fourth update criterion may be set by combining the first to third update criteria, such that when a recognition result that is x% or more of the frame size and y% or more away from the frame side is detected for m consecutive frames, that recognition result is extracted as a recognition target. In this case, an object that is detected at a certain size, captured without being cut off, and stably detected over several frames is extracted.

[0145] By setting such update criteria, the extraction unit 123 (FIG. 3) extracts recognition results that satisfy the update criteria. In step S113 (FIG. 11), the extraction unit 123 executes a process of extracting recognition results that satisfy the update criteria, and then the process result is used to make a determination in step S114. In step S114, it is determined whether or not there is a recognition result that satisfies the update criteria.

[0146] If it is determined in step S114 that there is no recognition result that satisfies the update criteria, the process returns to step S111, and the subsequent processes are repeated.

[0147] On the other hand, if it is determined in step S114 that a recognition result that satisfies the update criterion is found, the process proceeds to step S115. If a recognition result that satisfies the update criterion is found, the extraction unit 123 outputs information related to the recognition result, i.e., information related to the recognition target, to the recognition target tracking unit 134. The information related to the recognition target is, for example, information such as the coordinates, size, and label of the recognition target.

[0148] In step S115, the recognition target tracking unit 124 selects the oldest frame. The oldest frame depends on which of the first to fourth update criteria is used. When the first or second update criteria, or a combination of the first and second update criteria as the fourth update criteria, is used, the frame that was the subject of extraction processing, in other words, the frame from which the recognition target was extracted, is set as the oldest frame. For example, if it is determined that there is a recognition result that satisfies the update criteria at time T0, the frame containing that recognition result is set as the oldest frame.

[0149] When the third update criterion is used, or when a criterion that combines the first and third update criteria, a criterion that combines the second and third update criteria, or a criterion that combines the first to third update criteria is used as the fourth update criterion, as described with reference to FIG. 13, 0-m The recognition results detected consecutively in m frames between time T0 and time T1 are used as the recognition target. 0-m The frame is captured when

[0150] In step S116, tracking is performed on the past N frames of the selected recognition target. The selected recognition target refers to the recognition target when multiple recognition targets are extracted and one of them is selected as the tracking target. The past N frames refer to (N-1) frames captured before the oldest frame selected in step S115.

[0151] For example, suppose that frame F5 shown in Figure 9 is set as the oldest frame. Also suppose that person H51 and person H54 are extracted as recognition targets from frame F5, and person H51 is selected as the recognition target. In this case, if N of the past N frames is 5, the five frames, including frame F5, frame F4, frame F3, frame F2, and frame F1, are set as the past N frames.

[0152] By tracking person H51 in sequence from frame F5 to frame F1, a person corresponding to person H51 is detected in each of frames F5 to F1 and labeled as a person. That is, in this example, person H51 in frame F5, person H41 in frame F4, person H31 in frame F3, person H21 in frame F2, and person H11 in frame F1 are tracked in this order and each is labeled as a person.

[0153] In step S116, the recognition target tracking unit 124 performs tracking in the reverse direction in time series, and in step S117, a label is assigned to the tracking result by the label assignment unit 125. Such tracking and label assignment are performed for each recognition target.

[0154] In step S118, the re-learning unit 126 re-learns the learning model of the recognizer. The re-learning unit 126 trains the recognizer (learning model) using pairs of images (frames) and labels as training data. As described with reference to FIG. 4, this training can be performed by using frames labeled by the labeling unit 125 as correct labels. Alternatively, training can be performed by another training method using frames labeled by the labeling unit 125 as training data.

[0155] The re-learning unit 126 may perform learning using N frames as a data set, or may perform learning using a data set of frames greater than N that has been accumulated by processing N frames multiple times. The method of learning here does not limit the scope of application of the present technology.

[0156] In the above example, in frame F3 (Figure 7), person H31 is detected as a car, but tracking and labeling can be performed to label person H31 as a person. By training using such accurately labeled frames, it is possible to generate a recognizer that can reduce the chance of incorrectly recognizing person H31 as a car when processing an image such as frame F3.

[0157] Furthermore, although person H21 and person H11 are not detected in frame F2 (Fig. 6) or frame F1 (Fig. 5), tracking and labeling can be performed to label person H21 and person H11 as people. By performing training using frames with such labels, it is possible to generate a recognizer that can reduce the occurrence of not being able to detect person H21 and person H11 when processing images such as frame F2 and frame F1.

[0158] In step S119, the recognizer update unit 127 updates (the learning model of) the recognizer used in the recognition processing unit 122 with the recognizer (learning model) learned by the re-learning unit 126. The update may be performed by replacing the recognizer (learning model), or by replacing some of the parameters of the learning model.

[0159] A mechanism for evaluating the accuracy of the generated recognizer (learning model) may be provided. The accuracy of the generated recognizer may be evaluated, and the recognizer may be updated only when it is determined that the recognition performance has improved.

[0160] In this way, labeling is performed by tracking backward in time from a frame where detection has been performed with high accuracy. The recognizer is updated by performing learning using the labeled frames. This type of learning makes it possible to perform learning using frames in which erroneously detected objects are correctly labeled and frames in which previously undetected objects are detected and labeled, thereby generating a recognizer with improved recognition accuracy.

[0161] <Configuration of information processing system> The processing performed by the information processing device 110 in the above-described embodiment can also be shared and performed by a plurality of devices.

[0162] The information processing device 110 in the above embodiment is configured so that the information processing device 110 itself includes a learning device that performs relearning, but the learning device may also be included in another device.

[0163] Here, the explanation will be continued by taking as an example a case where processing is shared between two devices, an information processing device and a server.

[0164] 15 is a diagram showing the configuration of an embodiment of an information processing system. The information processing system 200 is composed of an information processing device 211 and a server 212. The information processing device 211 is, for example, a device mounted on a vehicle. The server 212 is a device that exchanges data with the information processing device 211 via a predetermined network.

[0165] The information processing device 211 includes an image acquisition unit 221, a recognition processing unit 222, an extraction unit 223, a data transmission unit 224, a recognizer reception unit 225, and a recognizer update unit 226. The server 212 includes a data reception unit 231, a recognition target tracking unit 232, a label assignment unit 233, a relearning unit 234, and a recognizer transmission unit 235.

[0166] The image acquisition unit 221, the recognition processing unit 222, the extraction unit 223, and the recognizer update unit 226 of the information processing device 211 are functions corresponding to the image acquisition unit 121, the recognition processing unit 122, the extraction unit 123, and the recognizer update unit 127 of the information processing device 110 (FIG. 3), respectively. The recognition target tracking unit 232, the label assignment unit 233, and the re-learning unit 234 of the server 212 are functions corresponding to the recognition target tracking unit 124, the label assignment unit 125, and the re-learning unit 126 of the information processing device 110 (FIG. 3), respectively.

[0167] <About information processing systems> The processing of the information processing system 200 shown in Fig. 14 will be described with reference to the flowcharts shown in Fig. 15 and 16. The processing performed by the information processing system 200 is basically the same as the processing performed by the information processing device 110, and since the processing performed by the information processing device 110 has already been described with reference to the flowchart shown in Fig. 11, the description of similar processing will be omitted where appropriate.

[0168] 15 is a flowchart for explaining the processing of the information processing device 211. The processing of steps S211 to S215 is the same as the processing of steps S111 to S115 (FIG. 11), and therefore the description thereof will be omitted.

[0169] In step S216, the image and the recognition target are transmitted to the server 212. The data transmission unit 224 of the information processing device 211 transmits at least the data relating to the recognition target extracted by the extraction unit 223, the oldest frame, and data from the oldest frame to the past N frames. The vehicle speed, frame rate, etc. may also be transmitted as needed.

[0170] The server 212 performs re-learning and transmits the re-learned recognizer to the information processing device 211. In step S217, the recognizer receiving unit 225 of the information processing device 211 receives the recognizer transmitted from the server 212, and the recognizer updating unit 226 updates the recognizer of the recognition processing unit 222 with the received recognizer.

[0171] FIG. 16 is a flowchart for explaining the processing of the server 212.

[0172] In step S231, the data receiving unit 231 of the server 212 receives the image (frame) and the data to be recognized transmitted by the data transmitting unit 224 of the information processing device 211. Steps S232 to S234 are similar to the processes of steps S116 to S118 (FIG. 11), and therefore detailed description thereof will be omitted.

[0173] The server 212 performs tracking by tracing frames back in the past direction, as was done by the information processing device 110, performs labeling, and re-learns the recognizer. The recognizer re-learned in this way is transmitted from the recognizer transmission unit 245 of the server 212 to the information processing device 211 in step S235.

[0174] In this way, the information processing device 211 and the server 212 may share the processing load.

[0175] <Other configurations of information processing systems> 17 is a diagram showing another example of the configuration of an information processing system. The information processing system 300 shown in FIG.

[0176] The information processing device 311 includes an image acquisition unit 321, a recognition processing unit 322, a data transmission unit 323, a recognizer reception unit 324, and a recognizer update unit 325. The server 312 includes a data reception unit 331, an extraction unit 332, a recognition target tracking unit 333, a label assignment unit 334, a relearning unit 335, and a recognizer transmission unit 336.

[0177] The image acquisition unit 321, the recognition processing unit 322, and the recognizer update unit 325 of the information processing device 311 are functions corresponding to the image acquisition unit 121, the recognition processing unit 122, and the recognizer update unit 127 of the information processing device 110 (FIG. 3), respectively. The extraction unit 332, the recognition target tracking unit 333, the label assignment unit 334, and the re-learning unit 335 of the server 312 are functions corresponding to the extraction unit 123, the recognition target tracking unit 124, the label assignment unit 125, and the re-learning unit 126 of the information processing device 110 (FIG. 3), respectively.

[0178] When comparing the information processing system 300 shown in Figure 17 with the information processing system 200 shown in Figure 14, the information processing system 300 has a configuration in which the extraction unit 223 of the information processing device 211 of the information processing system 200 is located on the server 212 side.

[0179] <Other processes in the information processing system> The processing of the information processing system 300 shown in Fig. 17 will be described with reference to the flowcharts shown in Fig. 18 and Fig. 19. The processing performed by the information processing system 300 is basically the same as the processing performed by the information processing device 110, and since the processing performed by the information processing device 110 has already been described with reference to the flowchart shown in Fig. 11, the description of similar processing will be omitted where appropriate.

[0180] 17 is a flowchart for explaining the processing of the information processing device 311. The processing of steps S311 and S312 is similar to the processing of steps S111 and S112 (FIG. 11), and therefore the description thereof will be omitted.

[0181] In step S313, the data transmission unit 323 of the information processing device 311 transmits the image and the recognition result to the server 312. The data transmission unit 323 of the information processing device 311 transmits at least the data and frames related to the recognition result recognized by the recognition processing unit 322. A mechanism may also be adopted in which the vehicle speed, frame rate, etc. are transmitted as necessary.

[0182] The image and the recognition results may be transmitted each time one frame is processed, or may be transmitted for several frames at a time.

[0183] The server 312 side performs re-learning and transmits the re-learned recognizer to the information processing device 311. In step S314, the recognizer receiving unit 324 of the information processing device 311 receives the recognizer transmitted from the server 312, and the recognizer updating unit 325 updates the recognizer of the recognition processing unit 322 with the received recognizer.

[0184] FIG. 19 is a flowchart for explaining the processing of the server 312.

[0185] In step S331, the data receiving unit 331 of the server 312 receives the image (frame) and recognition result data transmitted by the data transmitting unit 323 of the information processing device 311. In step S332, the extraction unit 332 extracts recognition targets that satisfy the update criteria. The processes of steps S332 to S337 are similar to the processes of steps S113 to S118 (FIG. 11), and therefore detailed description thereof will be omitted.

[0186] The server 312 performs the processes of extracting the recognition target, tracking by tracing back frames in the past direction, labeling, and re-learning the recognizer, which were previously performed by the information processing device 110. The recognizer re-learned in this way is transmitted from the recognizer transmission unit 336 of the server 312 to the information processing device 311 in step S338.

[0187] In this way, the information processing device 311 and the server 312 may share the processing load.

[0188] By configuring the information processing system 200 or the information processing system 300 so that the learning process is performed by the server 212 (312), the processing load of the information processing device 211 (311) can be reduced.

[0189] The server 212 (312) may be configured to collect data from a plurality of information processing devices 211 (311) and generate a recognizer (re-train the recognizer) using the data from the plurality of information processing devices 211 (311). By handling a large amount of data and training the recognizer, it is possible to obtain a recognizer with improved accuracy at an earlier stage.

[0190] In the above embodiment, an information processing device that processes images from a camera mounted on a vehicle has been described as an example, but the present invention can also be applied to an information processing device that processes images from a surveillance camera.

[0191] In the above-described embodiment, an example has been described in which an image captured by a camera is processed. However, the image may be a distance measurement image acquired by a ToF (Time-of-Flight) method. A thermal sensor may be used, and data obtained from the thermal sensor may be treated as an image so that a predetermined object, such as a person or a vehicle, can be recognized. This technology can be widely applied to cases in which a predetermined object is recognized using data obtained from a sensor.

[0192] This technology can also be applied when the specifications defined by the NICE (Network of Intelligent Camera Ecosystem) Alliance are applied.

[0193] <Regarding recording media> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs that make up the software are installed on a computer. Here, the term "computer" includes computers built into dedicated hardware, and general-purpose personal computers, for example, that can execute various functions by installing various programs.

[0194] 50 is a block diagram showing an example of the hardware configuration of a computer that executes the above-mentioned series of processes using a program. In the computer, a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, and a RAM (Random Access Memory) 503 are interconnected by a bus 504. An input / output interface 505 is further connected to the bus 504. An input unit 506, an output unit 507, a storage unit 508, a communication unit 509, and a drive 510 are connected to the input / output interface 505.

[0195] The input unit 506 includes a keyboard, a mouse, a microphone, etc. The output unit 507 includes a display, a speaker, etc. The storage unit 508 includes a hard disk, a non-volatile memory, etc. The communication unit 509 includes a network interface, etc. The drive 510 drives removable media 511 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0196] In a computer configured as described above, the CPU 501 performs the above-described series of processes by, for example, loading a program stored in the memory unit 508 into the RAM 503 via the input / output interface 505 and the bus 504 and executing it.

[0197] The program executed by the computer (CPU 501) can be provided by being recorded on removable media 511 such as package media, for example. The program can also be provided via wired or wireless transmission media such as a local area network, the Internet, or digital satellite broadcasting.

[0198] In a computer, a program can be installed in a storage unit 508 via an input / output interface 505 by inserting a removable medium 511 into a drive 510. The program can also be received by a communication unit 509 via a wired or wireless transmission medium and installed in the storage unit 508. Alternatively, the program can be installed in advance in the ROM 502 or the storage unit 508.

[0199] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0200] In this specification, a system refers to an entire device made up of multiple devices.

[0201] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0202] It should be noted that the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the present technology.

[0203] The present technology can also be configured as follows. (1) Tracking the object recognized by the recognition process using a recognizer to which a learning model that performs recognition processing on input data is applied in a chronologically reverse direction; Retraining the learning model using data generated based on the tracking results. How to generate a learning model. (2) The data is generated by tracking the object in a reverse chronological direction and labeling the tracked object. A method for generating a learning model according to (1) above. (3) among the recognition results obtained by the recognition processing on the frame captured at the first time, a recognition result that satisfies a predetermined criterion is determined as the object to be tracked; tracking the object captured in a plurality of frames captured at a time before the first time; If the object is detected in the frame as a result of the tracking, a label is assigned to the object. A method for generating a learning model according to (1) or (2). (4) The recognition result detected when the size of the recognition result relative to the size of the frame is equal to or larger than a predetermined ratio is regarded as the object as a recognition result that satisfies the predetermined criterion. A method for generating a learning model according to (3) above. (5) The ratio varies depending on the label assigned to the recognition result. The method for generating a learning model according to (4) above. (6) The recognition result that is located at a position that is a predetermined distance or more away from the side of the frame is regarded as the object as a recognition result that satisfies the predetermined criterion. A method for generating a learning model according to any one of (3) to (5). (7) The tracking targets include frames captured at the first time and frames captured between the first time and a second time prior to the first time. A method for generating a learning model according to any one of (3) to (6). (8) The recognition result detected over a plurality of frames is regarded as the object as a recognition result that satisfies the predetermined criteria. A method for generating a learning model according to any one of (3) to (6). (9) When the recognition result detected in a frame captured from the first time to a second time before the first time is determined as the target object, frames captured from the second time to a third time before the second time are determined as the tracking target. A method for generating a learning model according to any one of (3) to (6) and (8). (10) The number of frames is set to vary depending on the vehicle speed. The method for generating a learning model according to (8) above. (11) Transmitting the retrained learning model to another device. A method for generating a learning model according to any one of (1) to (10). (12) The learning model is learned by machine learning. A method for generating a learning model according to any one of (1) to (11). (13) a re-learning unit that tracks an object recognized by a recognition process using a recognizer in a reverse direction in time series, and re-learns a learning model of the recognizer based on learning data for re-learning the recognizer, the learning data being generated based on the tracking results; An information processing device comprising: (14) The training data is data generated by assigning labels to the tracked objects. The information processing device according to (13) above. (15) Among the recognition results obtained by the recognition process for the frames captured at a predetermined time, a recognition result that satisfies a predetermined criterion is extracted as the object to be tracked. The information processing device according to (13) or (14). (16) Update the recognizer with the retrained learning model. The information processing device according to any one of (13) to (14). (17) a recognition processing unit that performs recognition processing using a recognizer that applies a learning model that performs recognition processing on input data; an extraction unit that extracts recognition results that satisfy a predetermined criterion from the recognition results recognized by the recognition processing unit; a tracking unit that tracks the object in a reverse direction in time series, using the recognition result extracted by the extraction unit as the object; a labeling unit that assigns a label to the object tracked by the tracking unit; a re-learning unit that re-learns the learning model using the labels assigned by the label assignment unit; an update unit that updates the recognizer of the recognition processing unit using the learning model retrained by the retraining unit; An information processing system comprising: (18) It consists of a first device and a second device, the first device includes the recognition processing unit and the update unit; The second device includes the extractor, the tracker, the labeler, and the retrainer. The information processing system according to (17) above. (19) The second device receives a plurality of pieces of data from the first device and retrains the recognizer using the plurality of pieces of data. The information processing system according to (18) above. [Explanation of symbols]

[0204] 110 Information processing device, 121 Image acquisition unit, 122 Recognition processing unit, 123 Extraction unit, 124 Recognition target tracking unit, 125 Label assignment unit, 126 Re-learning unit, 127 Recognizer update unit, 134 Recognition target tracking unit, 200 Information processing system, 211 Information processing device, 212 Server, 213 Label assignment unit, 221 Image acquisition unit, 222 Recognition processing unit, 223 Extraction unit, 224 Data transmission unit, 225 Recognizer reception unit, 226 Recognizer update unit, 231 Data reception unit, 232 Recognition target tracking unit, 234 Re-learning unit, 245 Recognizer transmission unit, 300 Information processing system, 311 Information processing device, 312 Server, 321 Image acquisition unit, 322 Recognition processing unit, 323 Data transmission unit, 324 Recognizer reception unit, 325 Recognizer update unit, 331 Data reception unit, 332 Extraction unit, 333 Recognition target tracking unit, 334 Label assignment unit, 335 Retraining unit, 336 Recognizer transmission unit

Claims

1. Tracking the object recognized by the recognition process using a recognizer to which a learning model that performs recognition processing on input data is applied in a chronologically reverse direction; retraining the learning model using data generated based on the tracking results; a recognition result that satisfies a predetermined criterion among the recognition results obtained by the recognition process on the frame captured at the first time is determined as the object to be tracked; tracking the object captured in a plurality of frames captured at a time before the first time; If the object is detected in the frame as a result of the tracking, a label is assigned to the object. How to generate a learning model.

2. The recognition result detected when the size of the recognition result relative to the size of the frame is equal to or larger than a predetermined ratio is regarded as the object as a recognition result that satisfies the predetermined criterion. The method for generating a learning model according to claim 1 .

3. The ratio varies depending on the label assigned to the recognition result. The method for generating a learning model according to claim 2 .

4. The recognition result that is located at a position that is a predetermined distance or more away from the side of the frame is regarded as the object as a recognition result that satisfies the predetermined criterion. The method for generating a learning model according to claim 1 .

5. The tracking targets include frames captured at the first time and frames captured between the first time and a second time prior to the first time. The method for generating a learning model according to claim 1 .

6. The recognition result detected over a plurality of frames is regarded as the object as a recognition result that satisfies the predetermined criteria. The method for generating a learning model according to claim 1 .

7. When the recognition result detected in a frame captured from the first time to a second time before the first time is determined as the target object, frames captured from the second time to a third time before the second time are determined as the tracking target. The method for generating a learning model according to claim 1 .

8. The number of frames is set to vary depending on the vehicle speed. The method for generating a learning model according to claim 6 .

9. Transmitting the retrained learning model to another device. The method for generating a learning model according to claim 1 .

10. The learning model is learned by machine learning. The method for generating a learning model according to claim 1 .

11. a re-learning unit that tracks an object recognized by a recognition process using a recognizer in a reverse direction in time series, and re-learns a learning model of the recognizer based on learning data for re-learning the recognizer, the learning data being generated based on the tracking results; a recognition result that satisfies a predetermined criterion among the recognition results obtained by the recognition process on the frame captured at the first time is determined as the object to be tracked; tracking the object captured in a plurality of frames captured at a time before the first time; If the object is detected in the frame as a result of the tracking, a label is assigned to the object. Information processing device.

12. Update the recognizer with the retrained learning model. The information processing device according to claim 11.

13. a recognition processing unit that performs recognition processing using a recognizer that applies a learning model that performs recognition processing on input data; an extraction unit that extracts recognition results that satisfy a predetermined criterion from the recognition results recognized by the recognition processing unit; a tracking unit that tracks the object in a reverse direction in time series, using the recognition result extracted by the extraction unit as the object; a labeling unit that assigns a label to the object tracked by the tracking unit; a re-learning unit that re-learns the learning model using the labels assigned by the label assignment unit; an update unit that updates the recognizer of the recognition processing unit using the learning model retrained by the retraining unit; Equipped with the extraction unit determines, as the object to be tracked, a recognition result that satisfies the predetermined criterion among recognition results by the recognition processing unit for a frame captured at a first time; the tracking unit tracks the object captured in a plurality of frames captured at a time before the first time; When the object is detected in the frame as a result of the tracking, the label assignment unit assigns a label to the object. Information processing system.

14. The system comprises a first device and a second device, the first device includes the recognition processing unit and the update unit; The second device includes the extractor, the tracker, the labeler, and the retrainer. The information processing system according to claim 13.

15. The second device receives a plurality of pieces of data from the first device and retrains the recognizer using the plurality of pieces of data. The information processing system according to claim 14.

Images