Image processing device, image processing method, and program

Abstract

This image processing device comprises a decision unit for deciding, on the basis of a status related to movement of a moving body, the image quality of an image for sensing an object outside the moving body, and an output unit for outputting an image of the image quality decided by the decision unit.

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] 2. Description of the Related Art Conventionally, there is known a technique for detecting an object ahead of a moving object such as a vehicle by using images (frames) obtained at each point in time by a camera mounted on the moving object (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-139631 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the conventional technology, there is room for further improvement in object detection depending on the moving state of the moving body, the surrounding environment of the moving body, etc. In one aspect, an object is to provide a technology that can more appropriately detect objects. [Means for solving the problem]

[0005] In one proposal, the acceleration of a moving object in a given direction is Information that includes at least a determination unit that determines the image quality of an image that allows an object outside the moving body to be detected based on the above-mentioned, and outputs an image of the image quality determined by the determination unit. and causing a control device mounted on the moving body to recognize an object of the moving body using a trained model corresponding to the information including at least the acceleration, based on the information including at least the acceleration and the image of the image quality determined by the determination unit. An image processing apparatus having an output section is provided. Effect of the Invention

[0006] According to one aspect, it is possible to more appropriately detect an object. [Brief description of the drawings]

[0007] [Figure 1] 1A and 1B are diagrams illustrating an example of installation of an imaging device in a moving body according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a configuration of a moving body according to an embodiment. [Diagram 3] FIG. 2 is a diagram illustrating an example of the hardware configuration of an image processing device and a control device according to an embodiment. [Figure 4] FIG. 1 is a diagram illustrating an example of a configuration of an image processing device and a control device according to an embodiment. [Diagram 5] 10 is a flowchart illustrating an example of a process of a server according to the embodiment. [Figure 6] FIG. 4 is a diagram illustrating an example of learning data according to the embodiment. [Figure 7] 5 is a flowchart illustrating an example of processing by an image processing device and a control device according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0009] <Overall composition> Fig. 1 is a diagram illustrating the configuration of a control system 500 according to an embodiment. In the example of Fig. 1, the control system 500 includes a moving object 1 and a server 50. The number of moving objects 1 and servers 50 is not limited to the example of Fig. 1.

[0010] The mobile object 1 and the server 50 communicate via a network such as a mobile phone network such as 5G (5th Generation, 5th generation mobile communication system), 4G, LTE (Long Term Evolution), or 3G, a wireless LAN (Local Area Network), or the Internet.

[0011] The moving body 1 is, for example, a moving machine such as a vehicle that runs on land using wheels, a robot that moves using legs or the like, an aircraft, an unmanned aerial vehicle (drone), etc. Note that the vehicle includes, for example, an automobile, a motorcycle (motorbike), a robot that moves using wheels, a railroad car that runs on a railroad, etc. Note that the automobile also includes automobiles that run on roads, streetcars, construction vehicles used for construction purposes, military vehicles, industrial vehicles for loading and transporting goods, agricultural vehicles, etc.

[0012] The server 50 performs machine learning based on, for example, images captured by the moving body 1, and generates a trained model for recognizing objects. The server 50 also distributes the generated trained model to the moving body 1.

[0013] <Example of imaging device placement> Fig. 1 shows the appearance of a moving object 1, which is an automobile, as viewed from directly above. In the example of Fig. 1, the moving object 1 has an imaging device 12A, an imaging device 12B, an imaging device 12C, and an imaging device 12D (hereinafter, when there is no need to distinguish between them, they will be simply referred to as "imaging devices 12").

[0014] The imaging device 12 is a device that captures an image. The imaging device 12 may be, for example, a camera.

[0015] The imaging device 12A is an imaging device (rear camera, rear camera, back view camera) that captures the rearward (opposite direction from the normal traveling direction) as seen from the moving body 1. The imaging device 12B is an imaging device (left camera) that captures the leftward as seen from the moving body 1. The imaging device 12C is an imaging device (right camera) that captures the rightward as seen from the moving body 1. The imaging device 12D is an imaging device (front camera) that captures the forward (normal traveling direction) as seen from the moving body 1.

[0016] The imaging devices 12A, 12B, 12C, and 12D may be, for example, imaging devices that capture images for an advanced driver-assistance system (ADAS) that assists a driver in driving operations or images for automatic driving. Also, the imaging devices 12A, 12B, 12C, and 12D may be, for example, cameras that capture images for an omnidirectional monitor (around view, panoramic view, multi-view, top view) that generates an image of the moving object 1 as if seen from directly above.

[0017] The imaging device 12A may be, for example, a camera that captures an image to be displayed on a room mirror (back mirror) monitor. Also, the imaging device 12A may be, for example, a camera that captures an image to be displayed on the screen of the navigation device 18 when the moving object 1 moves backward (backing up).

[0018] The image capturing device 12B may be, for example, a camera that captures an image to be displayed on the left side mirror monitor, and the image capturing device 12C may be, for example, a camera that captures an image to be displayed on the right side mirror monitor.

[0019] The imaging device 12D that captures an image of the area ahead (the normal traveling direction) as seen from the moving object 1 may be a stereo camera having a plurality of cameras.

[0020] <Configuration of moving object 1> Fig. 2 is a diagram illustrating an example of the configuration of a moving object 1 according to an embodiment. In the example of Fig. 2, the moving object 1 includes an image processing device 10, a control device 11, an imaging device 12, an ECU 13, a wireless communication device 14, a sensor 15, a drive device 16, a lamp device 17, and a navigation device 18.

[0021] These components are connected to each other via an internal network (eg, an in-vehicle network) such as a Controller Area Network (CAN) or Ethernet (registered trademark).

[0022] The image processing device 10 generates an image that allows the control device 11 to detect objects outside (around) the moving body 1, based on images (still images and moving images) captured by the imaging device 12. Note that the objects may include, for example, other vehicles, pedestrians, bicycles, white lines, side walls of the road, and obstacles.

[0023] The control device 11 is a computer (information processing device) that controls each part of the moving body 1. The control device 11 recognizes objects outside the moving body 1 based on images generated by the image processing device 10. The control device 11 also tracks the recognized objects based on images generated at each point in time by the image processing device 10. The control device 11 controls an ECU (Electronic Control Unit) 13 of the moving body 1 and the like based on detected objects (recognized objects and tracked objects), thereby controlling the movement of the moving body 1 and the like.

[0024] The control device 11 may realize any level of autonomous driving, for example, from level 0, in which a driver (user, driver, passenger) operates the main control system (acceleration, steering, braking, etc.), to level 5, in which the vehicle is unmanned, by controlling the movement of the mobile body 1.

[0025] The ECU 13 is a device that controls each device of the mobile object 1. The ECU 13 may have multiple ECUs. The wireless communication device 14 communicates with devices outside the mobile object 1, such as a server 50 and a server on the Internet, by wireless communication such as a mobile phone network.

[0026] The sensor 15 is a sensor that detects various types of information. The sensor 15 may include, for example, a position sensor that acquires current position information of the mobile object 1. Note that the position sensor may be, for example, a sensor that uses a satellite positioning system such as a Global Positioning System (GPS).

[0027] The sensor 15 may also include a speed sensor that detects the speed of the moving body 1. The speed sensor may be, for example, a sensor that measures the number of rotations of an axle of a wheel. The sensor 15 may also include an acceleration sensor that detects the acceleration of the moving body 1. The sensor 15 may also include a yaw axis angular velocity sensor that detects the yaw axis angular velocity (yaw rate) of the moving body 1.

[0028] The sensor 15 may also include an operation sensor that detects the amount of operation of the mobile object 1 by the driver and the control device 11. The operation sensor may include, for example, an accelerator sensor that detects the amount of depression of an accelerator pedal, a steering sensor that detects the rotation angle of a steering wheel, a brake sensor that detects the amount of depression of a brake pedal, a shift position sensor that detects the gear position, and the like.

[0029] The driving device 16 is various devices for moving the moving body 1. The driving device 16 may include, for example, an engine, a steering device, and a braking device (brake).

[0030] The lamp device 17 is various lighting devices mounted on the mobile body 1. The lamp device 17 may include, for example, a headlight (headlamp, headlight), a turn signal (blinker) lamp for indicating the direction to the surroundings when turning right or left or changing course (lane change), a backlight provided at the rear of the mobile body 1 and turned on when the gear is in the reverse range, a brake lamp, and the like.

[0031] The navigation device 18 is a device (car navigation) that guides the route to a destination by voice and display. Map information may be recorded in the navigation device 18. The navigation device 18 may also transmit information on the current position of the mobile object 1 to an external server that provides a car navigation service, and obtain map information on the surroundings of the mobile object 1 from the external server. Note that the map information may include, for example, information on nodes indicating nodes such as intersections, and links that are road sections between the nodes.

[0032] <Computer hardware configuration> 3 is a diagram illustrating an example of the hardware configuration of the image processing device 10 and the control device 11 according to the embodiment. In the following, the image processing device 10 will be described as an example. The hardware configuration of the control device 11 may be the same as that of the image processing device 10.

[0033] In the example of FIG. 3, the image processing apparatus 10 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, and an interface device 1005, all of which are connected to one another via a bus B.

[0034] An information processing program for realizing processing in the image processing device 10 is provided by a recording medium 1001. When the recording medium 1001 on which the information processing program is recorded is set in the drive device 1000, the information processing program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the information processing program does not necessarily have to be installed from the recording medium 1001, and may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed information processing program as well as necessary files, data, and the like.

[0035] When an instruction to start a program is received, the memory device 1003 reads out the program from the auxiliary storage device 1002 and stores it. The CPU 1004 executes processing in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network.

[0036] An example of the recording medium 1001 is a portable recording medium such as a CD-ROM, a DVD disk, or a USB memory. An example of the auxiliary storage device 1002 is a hard disk drive (HDD) or a flash memory. Both the recording medium 1001 and the auxiliary storage device 1002 correspond to computer-readable recording media.

[0037] The image processing device 10 may be realized by an integrated circuit such as an Application Specific Integrated Circuit (ASIC) or a Field Programmable Gate Array (FPGA).

[0038] <Configuration of image processing device 10 and control device 11> Next, the configurations of the image processing device 10 and the control device 11 will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of the configurations of the image processing device 10 and the control device 11 according to the embodiment.

[0039] <Image processing device 10> The image processing device 10 includes an acquisition unit 101, a judgment unit 102, a decision unit 103, and an output unit 104. Each of these units may be realized by cooperation between one or more programs installed in the image processing device 10 and hardware such as a CPU 1004 of the image processing device 10.

[0040] The acquisition unit 101 acquires data from other devices. For example, the acquisition unit 101 acquires an image captured by the imaging device 12 from the imaging device 12. The acquisition unit 101 also acquires various information from each part of the moving object 1 via, for example, the ECU 13. The acquisition unit 101 also acquires information from a device external to the moving object 1 via, for example, the wireless communication device 14.

[0041] The determination unit 102 determines the situation regarding the movement of the moving object 1 based on the information acquired by the acquisition unit 101 .

[0042] The determination unit 103 determines the image quality of an image that allows an object outside the moving object 1 to be detected, based on the situation regarding the movement of the moving object 1 determined by the determination unit 102 .

[0043] The output unit 104 outputs an image having the image quality determined by the determination unit 103 , and inputs the image to the control device 11 .

[0044] ≪Control device 11≫ The control device 11 includes a storage unit 111, a recognition unit 112, a tracking unit 113, and a control unit 114. Each of these units may be realized by cooperation between one or more programs installed in the control device 11 and hardware such as a CPU of the control device 11.

[0045] The storage unit 111 stores the trained model distributed by the server 50.

[0046] The recognition unit 112 recognizes an object captured in an image based on the trained model stored in the storage unit 111 and the image output by the image processing device 10. The recognition unit 112 may recognize, for example, the type of the object and its relative position (distance) from the moving object 1. Note that the recognition unit 112 may classify the object into types such as vehicles, motorcycles, bicycles, humans, and others.

[0047] The tracking unit 113 tracks the object recognized by the recognition unit 112 based on the image output by the image processing device 10 at each time point, across the time points.

[0048] The control unit 114 controls the moving object 1 based on the distance between the moving object 1 and each object being tracked by the tracking unit 113 .

[0049] <Processing> Learning Phase Next, processing by the server 50 will be described with reference to Fig. 5. Fig. 5 is a flowchart showing an example of processing by the server 50 according to the embodiment. Fig. 6 is a diagram for explaining an example of learning data 501 according to the embodiment.

[0050] In step S1, the server 50 acquires learning data 501 for supervised learning. In the example of FIG. 6, the learning data 501 includes a plurality of sets (data sets) of information on situations (scenes) related to the movement of the moving object 1, images of the imaging device 12, and objects (subjects) in the images. The information on the objects in the images includes information indicating the area of ​​each object in the image and the type (label) of each object. The information indicating the area of ​​the object may be, for example, the upper left coordinates and the lower right coordinates of a rectangular area in which the object is captured in the image. The type of the object may include, for example, a vehicle, a motorcycle, a bicycle, a human, and others.

[0051] The learning data 501 may be created based on, for example, images taken when the data-collecting mobile object 1 is traveling. Information about objects in the images included in the learning data 501 may be set as correct answer data by, for example, a developer of a business entity developing the mobile object 1.

[0052] In addition, the situation regarding the movement of the moving body 1 contained in the learning data 501 may be set as correct answer data, for example, by a developer of a business entity developing the moving body 1, or may be automatically set by the image processing device 10, etc.

[0053] Next, the server 50 performs machine learning based on the learning data 501 to generate a trained model (step S2). Here, the server 50 may perform machine learning using, for example, deep learning or the like. In this case, the server 50 may perform machine learning using, for example, a convolutional neural network (CNN) for each situation related to the movement of the moving object 1. As a result, for example, when the moving object 1 is traveling on a highway, a trained model that classifies the moving object 1 into vehicles, motorcycles, side walls, and others can be generated, and the recognition process can be accelerated. Also, when the moving object 1 is traveling in a shopping district, a trained model that classifies the moving object 1 into vehicles, motorcycles, bicycles, elderly people, adults, children, and others can be generated, and the classification of the recognition target can be subdivided.

[0054] Furthermore, the server 50 may perform machine learning based on the learning data 501 by transfer learning to generate a trained model. In this case, the server 50 may retrain, based on the learning data 501, a convolutional neural network trained for each type of object based on an image other than the image of the imaging device 12 of the moving object 1.

[0055] The server 50 may also improve the recognition accuracy by using another classifier using the situation regarding the movement of the moving object 1. In this case, the server 50 may generate a trained model that classifies the feature amount (CNN feature amount) calculated using a convolutional neural network with another classifier using the situation regarding the movement of the moving object 1. In this case, the server 50 may use, for example, a support vector machine (SVM) or the like as the other classifier. This allows, for example, the likelihood of each type (probability of being each type) according to the situation to be inferred, so that an image of a certain object can be recognized as a bicycle when the moving object 1 is traveling in a shopping district, and as a motorcycle when the moving object 1 is traveling on a highway.

[0056] Next, the server 50 distributes the trained model to the mobile body 1 (step S3). As a result, the trained model is stored in the storage unit 111 of the control device 11 of the mobile body 1. The server 50 may distribute and store the trained model to the mobile body 1 each time depending on the situation around the mobile body 1. The mobile body 1 may store the trained model generated by the server 50 in the storage unit 111 in advance. The mobile body 1 may store multiple trained models generated by the server 50 in the storage unit 111 in advance, and select one of the multiple trained models depending on the situation around the mobile body 1.

[0057] Inference Phase Next, the processing of the image processing device 10 and the control device 11 of the moving object 1 will be described with reference to Fig. 7. Fig. 7 is a flowchart showing an example of the processing of the image processing device 10 and the control device 11 according to the embodiment.

[0058] In step S21, the determination unit 102 of the image processing device 10 determines a situation regarding the movement of the moving object 1. Here, the image processing device 10 may determine a situation regarding the movement of the moving object 1 based on information acquired via the imaging device 12, the ECU 13, the wireless communication device 14, or the like.

[0059] The image processing device 10 may determine, for example, the condition of the road on which the moving body 1 is currently traveling and the condition of objects outside the moving body 1, based on an image captured by the imaging device 12. In this case, the image processing device 10 may determine, for example, the width (roadway width) of the road on which the moving body 1 is currently traveling, the degree of visibility, the presence or absence of a side wall of an expressway or the like, the presence or absence of a vehicle parked on the road shoulder, the traffic congestion state of the road, and the like, based on a still image (one frame) captured by the imaging device 12. In addition, the image processing device 10 may determine, for example, the approaching speed of the moving body 1 and a vehicle following the moving body 1, based on a moving image (multiple frames) captured by the imaging device 12.

[0060] The image processing device 10 may also determine the status of the movement of the mobile object 1 based on information acquired from each part of the mobile object 1 via the ECU 12 or the like. In this case, the image processing device 10 may determine the attributes of the road on which the mobile object 1 is currently traveling and the attributes of the road on which the mobile object 1 is scheduled to travel at each point within a predetermined time (for example, one minute) from the present based on information acquired from, for example, the navigation device 18. Here, the attributes of the road may include information indicating the type of road, such as expressways, general roads (general national roads), major regional roads, general prefectural roads, municipal roads, and private roads. The attributes of the road may also include information such as the number of lanes, road width, and the location of in-link attributes (bridges / overpasses, tunnels, caverns, railroad crossings, pedestrian bridges, toll gates, underpasses, and expected road flooding locations). The image processing device 10 may also determine the congestion status of the road on which the mobile object 1 is currently traveling based on information acquired from, for example, the navigation device 18.

[0061] Furthermore, the image processing device 10 may determine the situation regarding the movement of the moving object 1 based on information such as the current speed, acceleration, steering angle by steering wheel operation, accelerator (accelerator pedal) operation (acceleration operation), brake (brake pedal) operation (deceleration operation), turn signal (blinker) illumination, and headlamp (headlamp, headlight) illumination, etc. In this case, the image processing device 10 may acquire each piece of information by the driver's operation or the control device 11 operation (automatic driving control) from an ECU or the like.

[0062] Furthermore, the image processing device 10 may determine the situation regarding the movement of the moving object 1 based on information acquired from, for example, VICS (registered trademark) (Vehicle Information and Communication System, road traffic information communication system) or a cloud service.

[0063] In this case, the image processing device 10 may determine, for example, whether the road on which the moving body 1 is currently traveling and the road on which the moving body 1 is scheduled to travel at each point within a predetermined time from the present (for example, one minute) are locations where traffic accidents frequently occur, whether they are locations where traffic jams frequently occur, and the weather at the location where the moving body 1 is currently traveling.

[0064] Next, the determination unit 103 of the image processing device 10 determines the image quality of an image for detecting an object outside the moving object 1 (an image for object recognition) based on the situation regarding the movement of the moving object 1 (step S22).

[0065] (Example of low resolution and low frame rate) For example, when the surroundings of the moving object 1 change little over time and there are few objects to be recognized, the image processing device 10 may determine image quality at a low resolution and a low frame rate (for example, 30 fps). Note that the image processing device 10 may determine the low resolution to be a resolution such as QVGA (Quarter Video Graphics Array, 320×240 pixels) or VGA (Video Graphics Array, 640×480 pixels).

[0066] In this case, the image processing device 10 may determine a low resolution and a low frame rate, for example, when the moving object 1 is parked in a parking lot or during parking. For example, the image processing device 10 may determine that the moving object 1 is located in a parking lot when the current location of the moving object 1 acquired from the navigation device 18 is a parking lot and when the location is not a road. Also, for example, when the speed of the moving object 1 is equal to or lower than a threshold (for example, 5 km per hour) and it is detected that the gear is in the reverse range, the image processing device 10 may determine that the moving object 1 is during parking and determine an image quality of low resolution and low frame rate.

[0067] Furthermore, the image processing device 10 may determine a low resolution and a low frame rate, for example, when the moving object 1 is traveling at a low speed in a congested section. The image processing device 10 may determine that the moving object 1 is traveling in a congested section, for example, based on congestion information for the current position of the moving object 1 acquired from the navigation device 18. Furthermore, the image processing device 10 may determine that the moving object 1 is traveling in a congested section, for example, when recognizing that a large number of vehicles are crowded ahead from an image captured by the imaging device 12.

[0068] (Example of low resolution and high frame rate) For example, in a situation where the surroundings of the moving object 1 change significantly over time and there are few objects to be recognized, the image processing device 10 may determine image quality to be low resolution and high frame rate (for example, 60 fps or 120 fps).

[0069] In this case, the image processing device 10 may determine a low resolution and a high frame rate when the moving body 1 is traveling on a highway at a predetermined speed or higher. This is because, on a highway, there are almost no pedestrians or bicycles, which are objects to be recognized in a high-resolution image, and therefore, a low resolution is considered sufficient. Also, since the accuracy of tracking an object is relatively important for predicting the future positional relationship between the moving body 1 and an object around the moving body 1 that cuts into a lane or approaches suddenly from behind, and avoiding a collision, it is considered desirable to perform tracking processing with an image at a high frame rate.

[0070] Note that the image processing device 10 may determine that the moving object 1 is traveling on an expressway, for example, when the current position of the moving object 1 acquired from the navigation device 18 is on an expressway. Also, the image processing device 10 may determine that the moving object 1 is traveling on an expressway, for example, when the side wall of an expressway or the like is recognized from an image captured by the imaging device 12. And, when the speed of the moving object 1 is a predetermined speed (for example, 60 km / h) or higher, the image processing device 10 may determine that the moving object 1 is traveling on an expressway at a predetermined speed or higher.

[0071] Furthermore, the image processing device 10 may determine a low resolution and a high frame rate when the moving object 1 changes course, for example. In this case, the image processing device 10 may detect that the moving object 1 is changing course based on, for example, the operation of a turn signal, the operation of a steering wheel, or the like.

[0072] Furthermore, the image processing device 10 may determine a low resolution and a high frame rate when the speed of the moving body 1 is equal to or higher than a threshold (e.g., 80 km / h). Note that the image processing device 10 may determine a higher frame rate as the speed of the moving body 1 increases. This is to improve the tracking accuracy (trackability) of a recognized object, since the accuracy of the approaching speed is more important than the accuracy of identifying the object approaching the moving body 1.

[0073] Furthermore, the image processing device 10 may determine a low resolution and a high frame rate when the acceleration of the moving object 1 in the traveling direction is equal to or greater than a threshold value, for example, in order to reduce collisions caused by a sudden start of the moving object 1.

[0074] Furthermore, the image processing device 10 may determine a low resolution and a high frame rate when the deceleration of the moving body 1 (acceleration in the opposite direction to the moving direction of the moving body 1) is equal to or greater than a threshold value. This is to reduce rear-end collisions from following vehicles due to sudden stops (sudden braking) of the moving body 1, for example.

[0075] (Example of high resolution and low frame rate) For example, in a situation where the time-dependent change around the moving object 1 is small and there are many objects to be recognized, the image processing device 10 may determine image quality with high resolution and low frame rate. Note that the image processing device 10 may determine resolution such as FHD (Full HD, 1920×1080 pixels) and 4K (4096×2160 pixels) as high resolution.

[0076] In this case, the image processing device 10 may determine a high resolution and a low frame rate, for example, when the moving object 1 is traveling on a road other than an expressway. This is because, when traveling on municipal roads, narrow roads, residential areas, and shopping streets (hereinafter also referred to as "municipal roads, etc." as appropriate), it is considered desirable to perform recognition processing with a high resolution image, since the accuracy of identifying whether an object is a pedestrian, a traveling bicycle, etc. is relatively important in order to predict the future positional relationship between the object and the moving object 1, etc. Also, since the speed of the moving object 1 is low compared to when traveling on an expressway, etc., it is considered that a low frame rate is sufficient.

[0077] (Example of high resolution and high frame rate) The image processing device 10 may determine image quality with high resolution and high frame rate when, for example, the surroundings of the moving object 1 change significantly over time and there are many objects to be recognized. This allows for highly accurate object detection in high-risk situations.

[0078] In this case, the image processing device 10 may determine a high resolution and a high frame rate, for example, when the moving object 1 enters an intersection. For example, when entering an intersection, there are many objects to be recognized, such as oncoming vehicles, pedestrians crossing the road, traffic lights, and following vehicles, and the situation changes rapidly, but by using images with a high resolution and a high frame rate, it is possible to recognize objects to be recognized around the moving object 1 at the intersection at high speed and with high accuracy.

[0079] Furthermore, the image processing device 10 may determine a high resolution and a high frame rate, for example, when the moving object 1 is traveling at high speed on a municipal road, etc. In this case, the image processing device 10 may determine that the moving object 1 is traveling at high speed on a municipal road, etc., when the current position of the moving object 1 acquired from the navigation device 18 is on a municipal road, etc., and the speed of the moving object 1 is equal to or higher than a threshold value (e.g., 80 km / h).

[0080] (Example of determining brightness, contrast, and color) The image processing device 10 may determine image quality such as, for example, brightness, contrast, and color of an image based on the situation regarding the movement of the moving object 1. In this case, the image processing device 10 may increase brightness and contrast when, for example, the moving object is traveling at night or traveling through a tunnel, and may correct discoloration of an object caused by the color of headlights and lighting in a tunnel.

[0081] (Example of determining image quality of images from multiple imaging devices 12) The image processing device 10 may determine the image quality of each of the images obtained from the multiple imaging devices 12 based on the situation regarding the movement of the moving object 1. In this case, for example, when the acceleration of the moving object 1 in a predetermined direction is equal to or greater than a threshold, the image processing device 10 may increase at least one of the resolution and frame rate of an image of a first imaging device that captures an image of the moving object 1 in the predetermined direction. Then, the image processing device 10 may decrease at least one of the resolution and frame rate of an image of a second imaging device that captures an image in a direction different from the predetermined direction.

[0082] In this case, for example, when the deceleration of the moving body 1 is equal to or greater than a threshold, the image processing device 10 may reduce at least one of the resolution and frame rate of the image of the imaging device 12D that captures an image ahead of the moving body, and may increase at least one of the resolution and frame rate of the images of the imaging devices 12A, 12B, and 12C. This can improve the recognition accuracy of vehicles following the moving body 1, for example, when the moving body 1 suddenly stops (suddenly brakes).

[0083] Furthermore, for example, when the acceleration in the traveling direction of the moving body 1 is equal to or greater than a threshold, the image processing device 10 may reduce at least one of the resolution and frame rate of the image of the imaging device 12A capturing an image behind the moving body 1, and increase at least one of the resolution and frame rate of the image of the imaging device 12D, etc. This can improve the recognition accuracy of a vehicle located in front of the moving body 1, for example, when the moving body 1 suddenly accelerates.

[0084] Next, the output unit 104 of the image processing device 10 outputs an image for object recognition with the determined image quality (step S23). This makes it possible to reduce the processing load on the control device 11.

[0085] Here, the image processing device 10 may generate an image for object recognition from the image captured by the imaging device 12.

[0086] Furthermore, the image processing device 10 may cause the imaging device 12 to capture an image of the image quality determined by the determination unit 103. In this case, the image processing device 10 may, for example, transmit a control command for setting the image quality to the imaging device 12. Then, the imaging device 12 may capture an image of the image quality specified by the received control command, and output the captured image to the image processing device 10 or the control device 11.

[0087] The image processing device 10 may also cause the control device 11 to recognize an object outside the moving object 1 based on information indicating the situation related to the movement of the moving object 1 and an image of image quality determined by the determination unit 103. In this case, the image processing device 10 also inputs information on the situation related to the movement of the moving object 1 determined by the determination unit 102 to the control device 11. This allows the control device 11 to perform inference based on the situation related to the movement of the moving object 1, thereby improving the accuracy of object recognition. Note that the image processing device 10 may output an image of the same or different image quality as the image output to the control device 11 to a display device for displaying the image to the driver of the moving object 1. The display device may be, for example, a room mirror monitor or a side mirror monitor, or may be included in the navigation device 18.

[0088] Next, the recognition unit 112 of the control device 11 recognizes objects outside the moving body 1 based on the image for object recognition and the trained model stored in the storage unit 111 (step S24). Note that the control device 11 may recognize white lines on the road by a recognition process that does not use machine learning.

[0089] Here, the control device 11 may infer the area of ​​the object in the image and the type of the object by using a trained model according to the situation regarding the movement of the moving object 1, as described above in the process of step S2 in Fig. 5. The control device 11 may also infer the area of ​​the object in the image and the type of the object by using another classifier using the situation regarding the movement of the moving object 1, as described above in the process of step S2 in Fig. 5.

[0090] Next, the tracking unit 113 of the control device 11 determines (tracks) a change in the positional relationship between the recognized object and the moving body 1 (step S25). This enables the control device 11 to predict the future positional relationship between the recognized object and the moving body 1.

[0091] Here, the control device 11 may track an object by, for example, the following process. First, the control device 11 calculates a predicted position in the current frame of object A that was recognized in the previous frame or is being tracked. Here, the control device 11 may calculate a predicted position of object A in the current frame based on, for example, the speed of the moving body 1, and the speed and traveling direction of the tracked object A relative to the moving body 1. Next, when the type of object A recognized in the previous or previous frame is the same as the type of object B recognized in the current frame, and the difference between the predicted position of object A in the current frame and the position of object B in the current frame is equal to or less than a threshold value, the control device 11 determines that object B is object A, and records the type, position, and traveling direction of object A (object B).

[0092] Next, the control unit 114 of the control device 11 controls each part of the moving body 1 based on the change in the positional relationship between the recognized object and the moving body 1 (step S26). Here, the control device 11 may notify the driver of the presence of an obstacle, a vehicle approaching at high speed from behind, etc., by using, for example, a display or a speaker of the moving body 1. In addition, the control device 11 may perform, for example, automatic driving of the moving body 1.

[0093] <Modification> Each functional unit of the image processing device 10 and the control device 11 may be realized by, for example, cloud computing provided by one or more computers. The image processing device 10 and the control device 11 may be configured as an integrated device. The image processing device 10 and the imaging device 12 may be configured as an integrated device. The machine learning process of the server 50 may be performed by the control device 11. The moving object 1 may have a semiconductor device, and one semiconductor device may include the image processing device 10 and the control device 11. The moving object 1 may have a plurality of semiconductor devices, one of which may include the image processing device 10, and another of which may include the control device 11.

[0094] Although the embodiments of the present invention have been described in detail above, the present invention is not limited to such specific embodiments, and various modifications and variations are possible within the scope of the gist of the present invention described in the claims. [Explanation of symbols]

[0095] 500 Control System 1. Mobile 10 Image processing device 101 Acquisition Department 102 Judgment section 103 Decision Section 104 Output section 11 Control device 111 Storage section 112 Recognition part 113 Tracking Department 114 Control section 12A Imaging device 12B Imaging device 12C Imaging device 12D imaging device 14 Wireless communication devices 15 Sensors 16 Drive unit 17 Lamp unit 18 Navigation devices 50 Servers

Claims

1. A determination unit that determines image quality of an image that allows an object outside the moving body to be detected based on information including at least an acceleration of the moving body in a predetermined direction; an output unit that outputs an image having the image quality determined by the determination unit, and causes a control device mounted on the moving body to recognize an object of the moving body using a trained model corresponding to the information including at least the acceleration, based on the information including at least the acceleration and the image having the image quality determined by the determination unit; An image processing device comprising:

2. A determination unit that determines image quality of an image that allows an object outside the moving body to be detected based on an acceleration of the moving body in a predetermined direction; an output unit that outputs an image having the image quality determined by the determination unit; having the determination unit reduces a resolution of the image for detecting an object outside the moving body and increases a frame rate of the image when an acceleration in a traveling direction of the moving body is equal to or greater than a first threshold and when an acceleration in a direction opposite to the traveling direction of the moving body is equal to or greater than a second threshold. Image processing device.

3. The output unit generates an image having the image quality determined by the determination unit based on an image from an imaging device mounted on the moving object. The image processing device according to claim 1 .

4. The output unit causes an imaging device mounted on the moving object to capture an image having the image quality determined by the determination unit. The image processing device according to claim 1 .

5. The determination unit determines at least one of a resolution and a frame rate of the image for detecting an object outside the moving body, based on information including at least the acceleration. The image processing device according to claim 1 .

6. The determination unit determines at least one of brightness, contrast, and color of the image that allows an object outside the moving body to be detected, based on information including at least the acceleration. The image processing device according to claim 1 .

7. when the acceleration of the moving body in the predetermined direction is equal to or greater than a threshold, the determination unit increases at least one of a resolution and a frame rate of an image of a first imaging device that captures an image of the moving body in the predetermined direction, and decreases at least one of a resolution and a frame rate of an image of a second imaging device that captures an image in a direction different from the predetermined direction. The image processing device according to claim 1 .

8. The image processing device determining an image quality for detecting an object outside the moving body based on information including at least an acceleration of the moving body in a predetermined direction; An image processing method comprising: outputting an image of a determined image quality; and, based on information including at least the acceleration and the image of the determined image quality, executing a process of having a control device mounted on the moving body recognize objects outside the moving body using a trained model corresponding to the information including at least the acceleration.

9. On the computer, determining an image quality for detecting an object outside the moving body based on information including at least an acceleration of the moving body in a predetermined direction; A program that outputs an image of the determined image quality, and causes a control device mounted on the moving body to execute a process of recognizing objects outside the moving body using a trained model corresponding to information including at least the acceleration, based on information including at least the acceleration and the image of the determined image quality.

Images