Information processing device, information processing method, and information processing program

Abstract

This information processing device is provided with: a first processor that outputs, from an image of an object captured by a first camera, point information captured using the captured object as a point; a second processor that outputs, from an image of an object captured by a second camera facing a direction corresponding to the first camera, identification information obtained by identifying the captured object; and a third processor that associates the point information with the identification information, in which a frame rate of the first camera is higher than a frame rate of the second camera.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and an information processing program. Background Art

[0002] A vehicle having an autonomous driving function is described in Patent Document 1.

[0003] Prior Art Documents

[0004] Patent Documents

[0005] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2022-035198. Summary of the Invention

[0006] Problems to be Solved by the Invention

[0007] In addition, in the case of performing autonomous driving of a vehicle as described in Patent Document 1, the autonomous driving is controlled by using a plurality of images around the vehicle captured by a camera. Here, in the case of controlling the autonomous driving by using a plurality of images captured by a plurality of cameras, there is room for improvement in how to set the frame rate of each camera.

[0008] Therefore, in one aspect of the present disclosure, an object is to provide an information processing apparatus, an information processing method, and an information processing program that can capture an object at a frame rate suitable for each camera when the object is captured by a plurality of cameras.

[0009] In addition, in the control of autonomous driving, it is desirable to be able to accurately avoid risks related to the driving of the vehicle by grasping the external environment in which the vehicle is located.

[0010] Therefore, in one aspect of the present disclosure, an object is to provide an information processing apparatus, an information processing method, and an information processing program that can grasp the external environment in which a moving body is located.

[0011] In addition, in the case of performing autonomous driving of a vehicle as described in Patent Document 1, the autonomous driving is controlled by using a plurality of images around the vehicle captured by a camera. Therefore, in conventional autonomous driving, there are problems of an increase in the amount of data acquired by the processor that controls the autonomous driving and an increase in the amount of calculation required to control the autonomous driving.

[0012] Therefore, in one aspect of the present disclosure, an object is to provide an information processing apparatus, an information processing method, and an information processing program that can reduce the amount of data output to a predetermined output destination when outputting the captured information of an object captured by a camera to the predetermined output destination.

[0013] In addition, in recent years, a technology has been studied in which multiple vehicles form a queue and travel on the road through autonomous driving. To achieve the autonomous driving of multiple vehicles forming a queue, an information processing device is used in the same way as for the autonomous driving of a single vehicle. The information processing device acquires information required for autonomous driving from the vehicle external environment and controls the autonomous driving based on the acquired information. For example, the information processing device identifies the situation in front of the queue and the situation behind the queue based on an image obtained from the situation of the vehicle external environment captured by a camera, and controls the autonomous driving based on the identified result. In addition, in order to achieve safe driving through autonomous driving, preferably, in addition to identifying the situation in front of the queue and the situation behind the queue, the information processing device also identifies the situation on the side of the queue and controls the autonomous driving based on the identified result.

[0014] However, on the basis of mounting a side camera for photographing the side situation of the queue on each vehicle among all the vehicles forming the queue, if the information processing device identifies the side situation of the queue based on the entire image obtained by photographing the side situation of the queue by each side camera, a huge processing burden will be imposed on the information processing device. The above also applies to the case where mobile bodies other than vehicles form a queue and travel safely through autonomous driving.

[0015] Therefore, in one aspect of the present disclosure, an object is to provide an information processing device, an information processing method, and an information processing program that can identify the situations in front of, behind, and on the side of the queue without imposing a processing burden as compared with the case where the information processing device identifies the side situation of the queue based on all the side images obtained by photographing the side of the queue by each side camera provided in each of the multiple mobile bodies moving in a queue.

[0016] Solution to the problem

[0017] The information processing device according to one aspect of the present disclosure includes: a first processor that outputs point information capturing the photographed object as a point from an image of the object photographed by a first camera; a second processor that outputs identification information obtained by identifying the photographed object from an image of the object photographed by a second camera facing the direction corresponding to the first camera; and a third processor that correlates the point information output from the first processor with the identification information output from the second processor, wherein the frame rate of the first camera is higher than the frame rate of the second camera.

[0018] In addition, in the information processing apparatus according to an aspect of the present disclosure, the frame rate of the first camera is more than 10 times that of the second camera.

[0019] In addition, in the information processing apparatus according to an aspect of the present disclosure, the frame rate of the first camera is not less than 100 frames per second, and the frame rate of the second camera is 10 frames per second.

[0020] An information processing method according to an aspect of the present disclosure causes a computer to perform the following processing: output point information that captures the object to be photographed as a point from an image of the object photographed by a first camera; output identification information obtained by identifying the object photographed from an image of the object photographed by a second camera having a frame rate lower than that of the first camera and facing a direction corresponding to the first camera; and cause the point information to correspond to the identification information.

[0021] An information processing program according to an aspect of the present disclosure causes a computer to perform the following processing: output point information that captures the object to be photographed as a point from an image of the object photographed by a first camera; output identification information obtained by identifying the object photographed from an image of the object photographed by a second camera having a frame rate lower than that of the first camera and facing a direction corresponding to the first camera; and cause the point information to correspond to the identification information.

[0022] An information processing apparatus according to an aspect of the present disclosure includes: a first processor that outputs point information that captures the photographed object as a point from an image of the object photographed by a first camera; a second processor that outputs identification information obtained by identifying the photographed object from an image of the object photographed by a second camera facing a direction corresponding to the first camera; and a third processor that causes the point information output from the first processor to correspond to the identification information output from the second processor, wherein the first processor calculates a danger level related to the movement of the mobile body as a score related to the external environment for a specified mobile body based on the detection information detected by the detection unit and the point information.

[0023] In the information processing apparatus according to an aspect of the present disclosure, the frame rate of the first camera is variable, and the first processor changes the frame rate of the first camera according to the calculated danger level.

[0024] In the information processing apparatus according to an aspect of the present disclosure, the danger level indicates the degree of how dangerous the place where the mobile body will travel in the future is.

[0025] In an information processing apparatus according to one aspect of the present disclosure, the apparatus includes: a first processor that outputs point information capturing the object to be imaged as points from an image of the object captured by a first camera; a second processor that outputs identification information obtained by identifying the object captured from an image of the object captured by a second camera facing a direction corresponding to the first camera; and a third processor that correlates the point information output from the first processor with the identification information output from the second processor, wherein the third processor calculates a risk related to the movement of a specified moving body as a score related to the external environment for the specified moving body based on detection information detected by a detection unit and the point information.

[0026] In the information processing apparatus according to one aspect of the present disclosure, a frame rate of the first camera is variable, and the third processor outputs an instruction to change the frame rate of the first camera to the first processor according to the calculated risk.

[0027] An information processing method according to one aspect of the present disclosure causes a computer to perform the following processing: outputting point information capturing the object to be imaged as points from an image of the object captured by a first camera; outputting identification information obtained by identifying the object captured from an image of the object captured by a second camera facing a direction corresponding to the first camera; correlating the point information with the identification information; and calculating a risk related to the movement of a specified moving body as a score related to the external environment for the specified moving body based on detection information detected by a detection unit and the point information.

[0028] An information processing program according to one aspect of the present disclosure causes a computer to perform the following processing: outputting point information capturing the object to be imaged as points from an image of the object captured by a first camera; outputting identification information obtained by identifying the object captured from an image of the object captured by a second camera facing a direction corresponding to the first camera; correlating the point information with the identification information; and calculating a risk related to the movement of a specified moving body as a score related to the external environment for the specified moving body based on detection information detected by a detection unit and the point information.

[0029] An information processing apparatus according to an aspect of the present disclosure includes: a first processor that outputs point information capturing the object to be photographed as a point from an image of an object photographed by a first camera capable of changing a frame rate; and a second processor that outputs identification information obtained by identifying the object photographed from an image of the object photographed by a second camera facing a direction corresponding to the first camera, wherein the first processor changes the frame rate of the first camera according to the category of the object based on the identification information.

[0030] In the information processing apparatus according to an aspect of the present disclosure, when the object is an object with agile movements, the first processor may increase the frame rate, and when the object is an object with slow movements or a stationary object, the first processor may decrease the frame rate.

[0031] In the information processing apparatus according to an aspect of the present disclosure, the first processor may also change the frame rate of the first camera according to the number of the objects.

[0032] In the information processing apparatus according to an aspect of the present disclosure, the first processor may make the frame rate higher as the number of the objects is larger and make the frame rate lower as the number of the objects is smaller.

[0033] In the information processing apparatus according to an aspect of the present disclosure, the first processor may calculate a score related to the external environment according to the category of the object, and change the frame rate according to the score related to the external environment.

[0034] In the information processing apparatus according to an aspect of the present disclosure, the first processor may calculate a score related to the external environment according to the category and the number of the objects, and change the frame rate according to the score related to the external environment.

[0035] In the information processing apparatus according to an aspect of the present disclosure, the first processor may extract points indicating the existence positions of the objects from the image photographed by the first camera, and output the points indicating the existence positions of the objects.

[0036] In the information processing apparatus according to an aspect of the present disclosure, a third processor may be provided to make the point information output from the first processor correspond to the identification information output from the second processor.

[0037] An information processing method according to an aspect of the present disclosure: Output point information that captures the object to be photographed as a point from an image of the object photographed by a first camera capable of changing the frame rate; Output identification information obtained by identifying the photographed object from an image of the object photographed by a second camera facing the direction corresponding to the first camera; and Change the frame rate of the first camera according to the category of the object based on the identification information.

[0038] An information processing program according to an aspect of the present disclosure causes a computer to perform the following processing: Output point information that captures the object to be photographed as a point from an image of the object photographed by a first camera capable of changing the frame rate; Output identification information obtained by identifying the photographed object from an image of the object photographed by a second camera facing the direction corresponding to the first camera; and Change the frame rate of the first camera according to the category of the object based on the identification information.

[0039] An information processing apparatus according to an aspect of the present disclosure includes: a first processor that outputs point information that captures the object to be photographed as a point from an image of the object photographed by a first camera capable of changing the frame rate; a second processor that outputs identification information obtained by identifying the photographed object from an image of the object photographed by a second camera facing the direction corresponding to the first camera; and a third processor that correlates the point information output from the first processor with the identification information output from the second processor, wherein the first processor derives, from the image of the object photographed by the first camera, coordinate values in the depth direction of the object in a three-dimensional rectangular coordinate system of a point indicating the existence position of the object as the point information, and changes the frame rate of the first camera according to the coordinate values in the depth direction.

[0040] In the information processing apparatus according to an aspect of the present disclosure, the first processor may derive the coordinate values in the depth direction as the point information from images of the object photographed by a plurality of the first cameras.

[0041] In the information processing apparatus according to an aspect of the present disclosure, the first processor may derive coordinate values in the width direction, height direction, and depth direction of the object as the point information from an image of the object photographed by the first camera and from a radar signal of a reflected wave from the object based on electromagnetic waves irradiated by the radar onto the object.

[0042] In the information processing apparatus according to one aspect of the present disclosure, the first processor may derive, as the point information, coordinate values in the width direction, height direction, and depth direction of the object from an image of the object captured by the first camera and from a result of capturing structured light irradiated onto the object by the irradiation device.

[0043] In the information processing apparatus according to one aspect of the present disclosure, the first processor may derive, as the point information, the coordinate value in the depth direction at a second time point, which is the next time point after the first time point, based on the coordinate values in the width direction, height direction, and depth direction of the object in the three-dimensional rectangular coordinate system at the first time point and the coordinate values in the width direction and height direction at the second time point.

[0044] An information processing method of the present disclosure: output point information that captures the object to be captured as points from an image of the object captured by a first camera capable of changing the frame rate; output recognition information obtained by recognizing the captured object from an image of the object captured by a second camera facing the direction corresponding to the first camera; and associate the point information with the recognition information, wherein the first processor derives, as point information, the coordinate value in the depth direction of the object in the three-dimensional rectangular coordinate system of a point indicating the existence position of the object from the image of the object captured by the first camera, and changes the frame rate of the first camera based on the coordinate value in the depth direction.

[0045] An information processing program of the present disclosure causes a computer to perform the following processing: output point information that captures the object to be captured as points from an image of the object captured by a first camera capable of changing the frame rate; output recognition information obtained by recognizing the captured object from an image of the object captured by a second camera facing the direction corresponding to the first camera; associate the point information with the recognition information; the first processor derives, as point information, the coordinate value in the depth direction of the object in the three-dimensional rectangular coordinate system of a point indicating the existence position of the object from the image of the object captured by the first camera; and change the frame rate of the first camera based on the coordinate value in the depth direction.

[0046] An information processing apparatus according to one aspect of the present disclosure is an information processing apparatus mounted on a vehicle, including a first processor that outputs point information that captures the object to be captured as points from an image of the object captured by a first camera capable of changing the frame rate, and the first processor changes the frame rate of the first camera based on the position of the vehicle.

[0047] In the information processing apparatus according to one aspect of the present disclosure, the first processor may calculate a score related to the external environment based on the position of the vehicle, and change the frame rate based on the score related to the external environment.

[0048] In the information processing apparatus according to one aspect of the present disclosure, the first processor may extract points indicating the presence position of the object from the image of the object captured by the first camera, and output the points indicating the presence position of the object.

[0049] In the information processing apparatus according to one aspect of the present disclosure, the first processor may change the frame rate of the first camera according to the category of the position of the vehicle.

[0050] The information processing apparatus according to one aspect of the present disclosure further includes: a second processor that outputs identification information obtained by identifying the captured object from the image of the object captured by the second camera facing the direction corresponding to the first camera; and a third processor that causes the point information output from the first processor to correspond to the identification information output from the second processor.

[0051] An information processing method according to one aspect of the present disclosure is an information processing method in an information processing apparatus mounted on a vehicle. In the method, point information that captures the captured object as a point is output from an image of the object captured by a first camera capable of changing the frame rate, and the frame rate of the first camera is changed according to the position of the vehicle.

[0052] An information processing program according to one aspect of the present disclosure is an information processing program for causing a computer to execute an information processing method in an information processing apparatus mounted on a vehicle. In the information processing program, point information that captures the captured object as a point is output from an image of the object captured by a first camera capable of changing the frame rate, and the frame rate of the first camera is changed according to the position of the vehicle.

[0053] An information processing apparatus according to an aspect of the present disclosure includes: a first processor that outputs point information capturing the object to be imaged as points from an image of the object captured by a first camera; and a second processor that outputs identification information obtained by identifying the object captured from an image of the object captured by a second camera facing a direction corresponding to the first camera, wherein a frame rate of the first camera is variable, and the first processor changes the frame rate of the first camera based on position information.

[0054] In addition, in the information processing apparatus according to an aspect of the present disclosure, a third processor is provided that causes the point information output from the first processor to correspond to the identification information output from the second processor.

[0055] In addition, in the information processing apparatus according to an aspect of the present disclosure, the first processor generates a heat map based on the frequency of detecting the object at each position around the first camera.

[0056] In addition, in the information processing apparatus according to an aspect of the present disclosure, the first processor changes the frame rate of the first camera based on the position information and the heat map.

[0057] An information processing method according to an aspect of the present disclosure causes a computer to execute the following processes: changing the frame rate of a first camera based on position information; outputting point information capturing the object to be imaged as points from an image of the object captured by the first camera; and outputting identification information obtained by identifying the object captured from an image of the object captured by a second camera facing a direction corresponding to the first camera.

[0058] An information processing program according to an aspect of the present disclosure causes a computer to execute the following processes: changing the frame rate of a first camera based on position information; outputting point information capturing the object to be imaged as points from an image of the object captured by the first camera; and outputting identification information obtained by identifying the object captured from an image of the object captured by a second camera facing a direction corresponding to the first camera.

[0059] An information processing apparatus according to an aspect of the present disclosure includes: a first processor that outputs point information capturing an object to be photographed as a point from an image of the object photographed by a first camera; and a second processor that outputs identification information obtained by identifying the object photographed from an image of the object photographed by a second camera facing a direction corresponding to the first camera, wherein a frame rate of the first camera is variable, and the first processor changes the frame rate of the first camera based on information of the user obtained from the user.

[0060] In addition, in the information processing apparatus according to an aspect of the present disclosure, a third processor is provided that correlates the point information output from the first processor with the identification information output from the second processor.

[0061] In addition, in the information processing apparatus according to an aspect of the present disclosure, the information of the user includes at least one of: voice information from the user; image information of the user being photographed; and heart rate information of the user.

[0062] In addition, in the information processing apparatus according to an aspect of the present disclosure, the user is an occupant of a vehicle in which at least a part of the information processing apparatus is mounted.

[0063] An information processing method according to an aspect of the present disclosure causes a computer to perform the following processing: changing a frame rate of a first camera based on information of the user obtained from the user; outputting point information capturing an object to be photographed as a point from an image of the object photographed by the first camera; and outputting identification information obtained by identifying the object photographed from an image of the object photographed by a second camera facing a direction corresponding to the first camera.

[0064] An information processing program according to an aspect of the present disclosure causes a computer to perform the following processing: changing a frame rate of a first camera based on information of the user obtained from the user; outputting point information capturing an object to be photographed as a point from an image of the object photographed by the first camera; and outputting identification information obtained by identifying the object photographed from an image of the object photographed by a second camera facing a direction corresponding to the first camera.

[0065] In the information processing apparatus according to an aspect of the present disclosure, the side camera photographs the side at a first frame rate, and the first frame rate is higher than frame rates of the front camera and the rear camera.

[0066] In the information processing apparatus according to one aspect of the present disclosure, every time a side image is obtained by photographing the side at the first frame rate, the processor identifies the situation of the side based on the obtained side image.

[0067] In the information processing apparatus according to one aspect of the present disclosure, the plurality of moving bodies are three or more moving bodies,

[0068] The specific moving body is an intermediate moving body located between the leading moving body and the trailing moving body.

[0069] In the information processing apparatus according to one aspect of the present disclosure, each of the plurality of moving bodies is a moving body capable of autonomous driving. In the intermediate moving body, at least one of a leading side camera capable of photographing the leading moving body side and a trailing side camera capable of photographing the trailing moving body side is provided. The processor controls the autonomous driving of the intermediate moving body based on at least one of a leading moving body side image obtained by photographing the leading moving body side by the leading side camera and a trailing moving body side image obtained by photographing the trailing moving body side by the trailing side camera. A second frame rate for the leading side camera and a third frame rate for the trailing side camera are lower than the frame rate of the front camera and the frame rate of the rear camera.

[0070] In the information processing apparatus according to one aspect of the present disclosure, each of the plurality of moving bodies is a moving body capable of autonomous driving. The processor controls the autonomous driving of the intermediate moving body without using at least one of a leading moving body side image obtained by photographing the leading moving body side from the intermediate moving body side and a trailing moving body side image obtained by photographing the trailing moving body side from the intermediate moving body side.

[0071] In the information processing apparatus according to one aspect of the present disclosure, the processor identifies the situation of the front by identifying the type of a front object existing in the front based on the front image, and identifies the situation of the rear by identifying the type of a rear object existing in the rear based on the rear image.

[0072] In the information processing apparatus according to one aspect of the present disclosure, the processor identifies the situation of the side by recognizing a side object existing in the side as a point based on the side image.

[0073] In the information processing apparatus according to one aspect of the present disclosure, each of the plurality of moving bodies is a moving body capable of autonomous driving, and the processor controls the autonomous driving based on the situation in the front, the situation in the rear, and the situation in the side.

[0074] In the information processing apparatus according to one aspect of the present disclosure, each of the plurality of moving bodies is a moving body capable of autonomous driving. The processor obtains front object information capable of determining the type of a front object by identifying the type of the front object existing in the front based on the front image, obtains rear object information capable of determining the type of a rear object by identifying the type of the rear object existing in the rear based on the rear image, and controls the autonomous driving based on front correspondence information and rear correspondence information. The front correspondence information is information that makes front point information representing the front object as a point correspond to the front object information, and is obtained from a first image obtained by photographing the front at a fourth frame rate higher than the frame rate of the front camera. The rear correspondence information is information that makes rear point information representing the rear object as a point correspond to the rear object information, and is obtained from a second image obtained by photographing the rear at a fifth frame rate higher than the frame rate of the rear camera.

[0075] In the information processing apparatus according to one aspect of the present disclosure, the processor obtains side point information representing a side object as a point by identifying the side object existing in the side as a point based on the side image, and controls the autonomous driving based on the front correspondence information, the rear correspondence information, and the side point information.

[0076] In the information processing apparatus according to one aspect of the present disclosure, the processor includes a front recognition processor, a rear recognition processor, and a side recognition processor. The front recognition processor recognizes the situation in the front based on the front image, the rear recognition processor recognizes the situation in the rear based on the rear image, and the side recognition processor recognizes the situation in the side based on the side image.

[0077] In the information processing apparatus according to one aspect of the present disclosure, the side recognition processor recognizes the situation in the side based on the side image by performing processing at a higher speed than the front recognition processor and the rear recognition processor.

[0078] An information processing method according to one aspect of the present disclosure includes: recognizing a situation in front based on a front image obtained by photographing the front with a front camera provided by a leading moving body among a plurality of moving bodies moving in a queue and capable of photographing the front of the queue; recognizing a situation in the rear based on a rear image obtained by photographing the rear with a rear camera provided by a trailing moving body among the plurality of moving bodies and capable of photographing the rear of the queue; and recognizing a situation on the side based on a side image obtained by photographing the side with a side camera provided by a specific moving body among the plurality of moving bodies and capable of photographing the side of the queue, the specific moving body being a moving body fewer in number than the plurality of moving bodies.

[0079] An information processing program according to one aspect of the present disclosure is a program for causing a computer to execute a process including the following steps: recognizing a situation in front based on a front image obtained by photographing the front with a front camera provided by a leading moving body among a plurality of moving bodies moving in a queue and capable of photographing the front of the queue; recognizing a situation in the rear based on a rear image obtained by photographing the rear with a rear camera provided by a trailing moving body among the plurality of moving bodies and capable of photographing the rear of the queue; and recognizing a situation on the side based on a side image obtained by photographing the side with a side camera provided by a specific moving body among the plurality of moving bodies and capable of photographing the side of the queue, the specific moving body being a moving body fewer in number than the plurality of moving bodies.

[0080] An information processing apparatus according to one aspect of the present disclosure includes a first processor. The first processor extracts a point indicating the existence position of an object from an image of the object and outputs motion information at a frame rate of 1000 frames per second or more, the motion information showing the motion of the point indicating the existence position of the object along a prescribed coordinate axis.

[0081] The first processor may output vector information of the motion of the center point or the center of gravity point of the object along a prescribed coordinate axis as the motion information. The first processor may output the vector information for at least two diagonally opposite points among the vertices of a quadrilateral surrounding the contour of the object.

[0082] The image may include an infrared image. The image may include a visible light image and an infrared image that are synchronized with each other.

[0083] It is also possible to use two of the first processors to output vector information of the actions along each of the three coordinate axes in a three-dimensional orthogonal coordinate system of the points indicating the existence position of the object as the action information.

[0084] The first processor may derive the distance to the object based on the reflected wave from the object of the electromagnetic wave irradiated onto the object, and output vector information of the actions along each of the three coordinate axes in a three-dimensional orthogonal coordinate system of the points indicating the existence position of the object as the action information.

[0085] The information processing apparatus may further include: a second processor that outputs an image of the object at a frame rate of less than 1000 frames per second; and a third processor that performs response control of the object based on the action information and the image output from the second processor.

[0086] The information processing apparatus related to the technology of the disclosure includes a first processor. The first processor extracts a point indicating the existence position of the object from the image of the captured object, and outputs the point indicating the existence position of the object.

[0087] The information processing apparatus includes a camera capable of changing the frame rate. The first processor calculates a score related to the external environment, determines the frame rate of the camera according to the score, outputs a control signal indicating image capture at the determined frame rate to the camera, extracts a point indicating the existence position of the object from the image captured by the camera, and outputs the point indicating the existence position of the object.

[0088] The information processing apparatus is mounted on a vehicle. The first processor calculates a risk level related to the running of the vehicle as the score related to the external environment, determines the frame rate of the camera according to the risk level, outputs a control signal indicating image capture at the determined frame rate to the camera, extracts a point indicating the existence position of the object from the image captured by the camera, and outputs the point indicating the existence position of the object.

[0089] The first processor extracts an object from the image. When the existence position of the object is in a predetermined area, the first processor extracts a point indicating the existence position of the object, and outputs the point indicating the existence position of the object.

[0090] The first processor extracts an object from the image, calculates a score for each object, extracts a point indicating the existence position of the object whose score is equal to or greater than a predetermined threshold, and outputs the point indicating the existence position of the object.

[0091] In addition, an information processing apparatus related to the technology of the disclosed content includes: a camera capable of changing the frame rate; and a processor, wherein the processor: detects an object reflected in an image captured by the camera; and controls in such a way as to change the frame rate of the camera according to at least one of the number of the detected objects, the acceleration of the objects, and the size of the objects.

[0092] The processor: when changing the frame rate according to the number of the objects, controls in such a way that the higher the number of the objects, the higher the frame rate, and the lower the number of the objects, the lower the frame rate.

[0093] The processor: when changing the frame rate according to the acceleration of the objects, controls in such a way that the greater the acceleration of the objects, the higher the frame rate, and the smaller the acceleration of the objects, the lower the frame rate.

[0094] The processor: when changing the frame rate according to the size of the objects, controls in such a way that the larger the size of the objects, the higher the frame rate, and the smaller the size of the objects, the lower the frame rate.

[0095] The processor may also calculate a score related to the external environment according to at least one of the number of the objects, the acceleration of the objects, and the size of the objects, and controls in such a way as to change the frame rate according to the score related to the external environment and a preset threshold value.

[0096] The processor may extract points indicating the existence positions of the objects from the image captured by the camera, and output the points indicating the existence positions of the objects.

[0097] The information processing apparatus may use two of the processors, and output vector information of the actions along each of the three coordinate axes in a three-dimensional orthogonal coordinate system indicating the existence positions of the objects as the action information.

[0098] An information processing method related to one aspect of the present disclosure is an information processing method executed by an information processing apparatus, the information processing apparatus including: a camera capable of changing the frame rate; and a processor, wherein the processor: detects an object reflected in an image captured by the camera; and controls in such a way as to change the frame rate of the camera according to at least one of the number of the detected objects, the acceleration of the objects, and the size of the objects.

[0099] An information processing program according to one aspect of the present disclosure is an information processing program for causing a processor of an information processing apparatus to execute. The information processing apparatus includes: a camera capable of changing a frame rate; and a processor. The processor: detects an object reflected in an image captured by the camera; and controls the frame rate of the camera in a manner that changes according to at least one of the number of the detected objects, the acceleration of the objects, and the size of the objects.

[0100] An information processing apparatus according to one aspect of the present disclosure includes: a camera capable of changing a frame rate; and a processor. The processor: detects an object reflected in an image captured by the camera at each moment; and controls the frame rate of the camera in a manner that changes according to at least one of the time series of the number of the detected objects, the time series of the acceleration of the objects, and the time series of the size of the objects.

[0101] When changing the frame rate according to the time series of the number of the objects, the processor may also control in a manner that increases the frame rate when the number of the objects reflected in the image at the current moment is larger than the number of the objects reflected in the image at the previous moment, and controls in a manner that decreases the frame rate when the number of the objects reflected in the image at the current moment is smaller than the number of the objects reflected in the image at the previous moment.

[0102] When changing the frame rate according to the time series of the acceleration of the objects, the processor may also control in a manner that increases the frame rate when the acceleration of the objects reflected in the image at the current moment is larger than the acceleration of the objects reflected in the image at the previous moment, and controls in a manner that decreases the frame rate when the acceleration of the objects reflected in the image at the current moment is smaller than the acceleration of the objects reflected in the image at the previous moment.

[0103] When changing the frame rate according to the time series of the size of the objects, the processor may also control in a manner that increases the frame rate when the size of the objects reflected in the image at the current moment is larger than the size of the objects reflected in the image at the previous moment, and controls in a manner that decreases the frame rate when the size of the objects reflected in the image at the current moment is smaller than the size of the objects reflected in the image at the previous moment.

[0104] The processor may also calculate a score related to the external environment based on at least one of the time series of the number of the objects, the time series of the acceleration of the objects, and the time series of the size of the objects, and perform control in a manner of changing the frame rate based on the score related to the external environment.

[0105] The processor may extract points indicating the presence positions of the objects from the images captured by the camera, and output the points indicating the presence positions of the objects.

[0106] The information processing apparatus may use two of the processors, and output vector information of actions along each of three coordinate axes in a three-dimensional orthogonal coordinate system of the points indicating the presence positions of the objects as the action information.

[0107] An information processing method according to one aspect of the present disclosure is an information processing method executed by an information processing apparatus including: a camera capable of changing a frame rate; and a processor, wherein the processor: detects objects reflected in images captured by the camera at respective times; and performs control in a manner of changing the frame rate of the camera based on at least one of the time series of the number of the detected objects, the time series of the acceleration of the objects, and the time series of the size of the objects.

[0108] An information processing program according to one aspect of the present disclosure is an information processing program for causing the processor of an information processing apparatus to execute, the information processing apparatus including: a camera capable of changing a frame rate; and a processor, wherein the processor: detects objects reflected in images captured by the camera at respective times; and performs control in a manner of changing the frame rate of the camera based on at least one of the time series of the number of the detected objects, the time series of the acceleration of the objects, and the time series of the size of the objects.

[0109] In addition, the above summary of the present disclosure does not list all necessary features of the present disclosure. In addition, sub-combinations of these feature groups may also form the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0110] Figure 1 is a schematic diagram showing an example of a vehicle equipped with a Central Brain.

[0111] Figure 2 is a first block diagram showing an example of the configuration of the information processing apparatus. Figure 3 is a second block diagram showing an example of the configuration of the information processing apparatus.

[0112] Figure 4It is an explanatory diagram showing an example of the point information output by the MoPU.

[0113] Figure 5 It is the third block diagram showing an example of the configuration of the information processing device.

[0114] Figure 6 It is the fourth block diagram showing an example of the configuration of the information processing device.

[0115] Figure 7 It is an explanatory diagram showing an example of the correspondence between the point information and the label information.

[0116] Figure 8 It is an explanatory diagram showing the schematic configuration of the vehicle.

[0117] Figure 9 It is a block diagram showing an example of the functional configuration of the cooling execution device.

[0118] Figure 10 It is the fifth block diagram showing an example of the configuration of the information processing device.

[0119] Figure 11 It is the sixth block diagram showing an example of the configuration of the information processing device.

[0120] Figure 12 It is a diagram schematically showing the detection of the time series coordinates of an object.

[0121] Figure 13 It is the seventh block diagram showing an example of the configuration of the information processing device.

[0122] Figure 14 It is an explanatory diagram for explaining the object image captured by the event camera.

[0123] Figure 15 It is a conceptual diagram showing an example of the manner of multiple vehicles traveling on the road in a queue.

[0124] Figure 16 It is a conceptual diagram showing an example of the configuration of the first information processing device, the second information processing device, and the third information processing device.

[0125] Figure 17 It is a conceptual diagram showing an example of the configuration of the first information processing device, the first aspect sensor, and the second aspect sensor mounted on the leading vehicle.

[0126] Figure 18 It is a conceptual diagram showing an example of the processing contents of the first aspect sensor, the second aspect sensor, the first IPU, the second IPU, the first MoPU, and the second MoPU.

[0127] Figure 19 This is a conceptual diagram showing an example of the structure of each of the second information processing device, third aspect sensor, fourth aspect sensor, fifth aspect sensor, and sixth aspect sensor to be mounted on the intermediate vehicle.

[0128] Figure 20 This is a conceptual diagram showing an example of the processing content of the third low FR camera, fourth low FR camera, fifth low FR camera, sixth low FR camera, third IPU, fourth IPU, fifth IPU, and sixth IPU.

[0129] Figure 21 This is a conceptual diagram showing an example of the processing content of the third high FR camera, third radar, fourth high FR camera, fourth radar, fifth high FR camera, fifth radar, sixth high FR camera, sixth radar, third MoPU, fourth MoPU, fifth MoPU, and sixth MoPU.

[0130] Figure 22 This is a conceptual diagram showing an example of the structure of each of the third information processing device, seventh aspect sensor, and eighth aspect sensor to be mounted on the trailing vehicle.

[0131] Figure 23 This is a conceptual diagram showing an example of the processing content of the seventh aspect sensor, eighth aspect sensor, seventh IPU, eighth IPU, seventh MoPU, and eighth MoPU.

[0132] Figure 24 This is a conceptual diagram showing an example of the processing content of the first central brain for obtaining necessary information to achieve control of platoon autonomous driving.

[0133] Figure 25 This is a conceptual diagram showing an example of the processing content of the second central brain for obtaining necessary information to achieve control of platoon autonomous driving.

[0134] Figure 26 This is a conceptual diagram showing an example of the processing content of the third central brain for obtaining necessary information to achieve control of platoon autonomous driving.

[0135] Figure 27 This is a conceptual diagram showing an example of the processing content of the first central brain for controlling the leading vehicle to achieve control of platoon autonomous driving.

[0136] Figure 28 This is a conceptual diagram showing an example of the processing content of the second central brain for controlling the intermediate vehicle to achieve control of platoon autonomous driving.

[0137] Figure 29It is a conceptual diagram showing an example of the processing content of the third central brain for controlling the trailing vehicle to achieve platoon autonomous driving.

[0138] Figure 30 It is a flowchart showing an example of the process of the leading vehicle IPU processing executed by the leading vehicle processor.

[0139] Figure 31 It is a flowchart showing an example of the process of the leading vehicle MoPU processing executed by the leading vehicle processor.

[0140] Figure 32 It is a flowchart showing an example of the process of the first central brain processing executed by the leading vehicle processor.

[0141] Figure 33 It is a flowchart showing an example of the process of the intermediate vehicle IPU processing executed by the intermediate vehicle processor.

[0142] Figure 34 It is a flowchart showing an example of the process of the intermediate vehicle MoPU processing executed by the intermediate vehicle processor.

[0143] Figure 35 It is a flowchart showing an example of the process of the second central brain processing executed by the intermediate vehicle processor.

[0144] Figure 36 It is a flowchart showing an example of the process of the trailing vehicle IPU processing executed by the trailing vehicle processor.

[0145] Figure 37 It is a flowchart showing an example of the process of the trailing vehicle MoPU processing executed by the trailing vehicle processor.

[0146] Figure 38 It is a flowchart showing an example of the process of the third central brain processing executed by the trailing vehicle processor.

[0147] Figure 39 It is a conceptual diagram showing a first variation of the first central brain deriving the first control variable, the second control variable, and the third control variable.

[0148] Figure 40 It is a conceptual diagram showing a second variation of the first central brain deriving the first control variable, the second control variable, and the third control variable.

[0149] Figure 41 It is a conceptual diagram showing a third variation of the first central brain deriving the first control variable, the second control variable, and the third control variable.

[0150] Figure 42 It is a conceptual diagram illustrating a fourth modification in which a first control variable, a second control variable, and a third control variable are derived by a first central brain.

[0151] Figure 43 It is a diagram illustrating an example of the aspect of data transfer from MoPU to the central brain in the embodiments of the technology of the disclosure.

[0152] Figure 44 It is a diagram illustrating an example of the aspect of data transfer from MoPU to the central brain in the embodiments of the technology of the disclosure.

[0153] Figure 45 It is a diagram illustrating an example of MoPU processing in the embodiments of the technology of the disclosure.

[0154] Figure 46 It is a diagram illustrating an example of MoPU processing in the embodiments of the technology of the disclosure.

[0155] Figure 47 It is a block diagram illustrating an example of the configuration of an information processing apparatus in the embodiments of the technology of the disclosure.

[0156] Figure 48 It is a block diagram illustrating an example of the configuration of an information processing apparatus in the embodiments of the technology of the disclosure.

[0157] Figure 49 It is a diagram for explaining the embodiments of the technology of the disclosure.

[0158] Figure 50 It is an explanatory diagram schematically showing an example of the computer hardware configuration used as an information processing apparatus or a cooling execution apparatus. Detailed Embodiments

[0159] Hereinafter, embodiments of the present disclosure will be described. However, the following embodiments do not limit the invention related to the claims. In addition, the combinations of the features described in the embodiments are not all necessary for the solution of the present disclosure.

[0160] (First Embodiment)

[0161] First, the first embodiment related to this embodiment will be described. As an example, at least a part of the information processing apparatus related to the present disclosure is mounted on the vehicle 100 and performs autonomous driving control of the vehicle 100. In addition, the information processing apparatus can realize autonomous driving in real time based on data obtained through various sensor inputs in Level 6 AI / multivariable analysis / goal seek / strategic planning / optimal probability solution / optimal speed solution / optimal route management / edge, and can provide a driving system adjusted based on the delta optimal solution. The vehicle 100 is an example of an "object object".

[0162] Here, "Level 6" represents the level of autonomous driving, which is equivalent to a level higher than Level 5 representing fully autonomous driving. Level 5 represents fully autonomous driving and is at the same level as human driving. Even so, there is still a probability of accidents and the like. Level 6 represents a level higher than Level 5, which is equivalent to a level with a lower probability of accidents than Level 5.

[0163] The computing power of Level 6 is approximately 1000 times that of Level 5. Therefore, high-performance driving control that cannot be achieved by Level 5 can be realized.

[0164] Figure 1 It is a schematic diagram showing an example of the vehicle 100 equipped with the Central Brain 15. A plurality of gateways are communicably connected to the Central Brain 15. The Central Brain 15 is connected to an external cloud server via the gateway. The Central Brain 15 is configured to be able to access the external cloud server via the gateway. On the other hand, it is configured such that due to the presence of the gateway, the Central Brain 15 cannot be directly accessed from the outside.

[0165] The Central Brain 15 outputs a request signal to the cloud server every time a specified time elapses. Specifically, the Central Brain 15 outputs a request signal representing a query to the cloud server every one billionth of a second. As an example, the Central Brain 15 controls the Level 6 autonomous driving based on a plurality of information obtained via the gateway.

[0166] Figure 2FIG. 1 is a first block diagram showing an example of the configuration of the information processing apparatus 10. The information processing apparatus 10 includes an IPU (Image Processing Unit) 11, a MoPU (Motion Processing Unit) 12, a central brain 15, and a memory 16. The central brain 15 is configured to include a GNPU (Graphics Neural Network Processing Unit) 13 and a CPU (Central Processing Unit) 14.

[0167] The IPU 11 is built into a super high-resolution camera (not shown) provided in the vehicle 100. The IPU 11 performs predetermined image processing such as Bayer conversion, demosaicing, denoising, and sharpening on the object image existing around the vehicle 100 captured by the super high-resolution camera, and outputs the processed object image at a frame rate of, for example, 10 frames per second and a resolution of 12 million pixels. In addition, the IPU 11 outputs recognition information obtained by recognizing the object captured in the image captured by the super high-resolution camera. The recognition information is information required to identify what the captured object is (for example, a person or an obstacle). In the present embodiment, the IPU 11 outputs label information indicating the category of the captured object (for example, information indicating whether the captured object is a dog, a cat, or a bear (information capable of determining the object type)) as the recognition information. Further, the IPU 11 outputs position information indicating the position of the captured object in the camera coordinate system of the super high-resolution camera. The image, label information, and position information output from the IPU 11 are supplied to the central brain 15 and the memory 16. The IPU 11 is an example of a "second processor", and the super high-resolution camera is an example of a "second camera".

[0168] MoPU 12 uses various sensors, including another camera different from the ultra-high-resolution camera (not shown) provided in the vehicle 100, such as internal sensors (e.g., acceleration sensors and / or gyro sensors, etc.) and external sensors (e.g., cameras, radars, and / or optical ranging devices using lasers, etc.). For example, MoPU 12 is a processing device that performs processing such as identifying the position and / or movement, etc. of the external environmental conditions (e.g., objects) of the vehicle 100. For example, MoPU 12 is connected to another camera provided in the vehicle 100 (as an example, MoPU 12 is built into another camera (not shown) different from the ultra-high-resolution camera provided in the vehicle 100, or is connected to another camera). MoPU 12 outputs point information (point information in which the object to be photographed is captured as a point) at a frame rate of 100 frames per second or more from the image of the object photographed by another camera facing the direction corresponding to the ultra-high-resolution camera at a frame rate of 100 frames per second or more. The point information output from MoPU 12 is supplied to the central brain 15 and the memory 16. Thus, the image for MoPU 12 to output point information and the image for IPU11 to output identification information are images photographed by another camera and the ultra-high-resolution camera facing the corresponding direction (images obtained by photographing). Here, the "corresponding direction" means the direction in which the shooting range of another camera overlaps with the shooting range of the ultra-high-resolution camera. In the above case, another camera will photograph the object facing the direction overlapping with the shooting range of the ultra-high-resolution camera. In addition, the ultra-high-resolution camera and another camera photographing the object facing the corresponding direction will be achieved, for example, by previously obtaining the correspondence relationship of the camera coordinate systems between the ultra-high-resolution camera and another camera.

[0169] For example, MoPU 12 outputs the coordinate values of the point indicating the position where the object exists on at least two coordinate axes in the three-dimensional rectangular coordinate system as point information. As an example, the coordinate values indicate the center point (or center of gravity) of the object. In addition, MoPU 12 outputs the coordinate value of the axis (x-axis) along the width direction in the three-dimensional rectangular coordinate system (hereinafter referred to as "x coordinate value") and the coordinate value of the axis (y-axis) along the height direction (hereinafter referred to as "y coordinate value") as the coordinate values on the two coordinate axes. In addition, the x-axis is the axis along the vehicle width direction of the vehicle 100, and the y-axis is the axis along the vehicle height direction of the vehicle 100.

[0170] With the above configuration, the point information output by the MoPU 12 within 1 second includes x - coordinate values and y - coordinate values of more than 100 frames. Therefore, based on this point information, it is possible to grasp the actions (moving direction and moving speed) of an object on the x - axis and y - axis in a three - dimensional rectangular coordinate system. That is, the point information output by the MoPU 12 includes position information indicating the position of the object in the three - dimensional rectangular coordinate system and action information indicating the action of the object.

[0171] As described above, the point information output from the MoPU 12 does not include the information required to identify what the photographed object is (e.g., whether it is a person or an obstacle), but only includes information indicating the actions (moving direction and moving speed) of the center point (or center - of - gravity point) of the object on the x - axis and y - axis. Moreover, since the point information output from the MoPU 12 does not include image information, the amount of data output to the central brain 15 and the memory 16 can be significantly reduced. The MoPU 12 is an example of the "first processor", and the other camera is an example of the "first camera".

[0172] As described above, in this embodiment, the frame rate of the other camera with the MoPU 12 built - in is higher than that of the ultra - high - resolution camera with the IPU11 built - in. Specifically, the frame rate of the other camera is not less than 100 frames per second, and the frame rate of the ultra - high - resolution camera is 10 frames per second. That is to say, the frame rate of the other camera is more than 10 times that of the ultra - high - resolution camera.

[0173] The central brain 15 makes the point information output from the MoPU 12 correspond to the label information output from the IPU 11. For example, there may be a state where due to the frame rate difference between the above - mentioned other camera and the ultra - high - resolution camera, the central brain 15 obtains point information related to an object, but does not obtain the label information. In this state, the central brain 15 identifies the x - coordinate value and y - coordinate value of the object based on the point information, but does not identify what the object is.

[0174] After that, when the label information related to the above - mentioned object is obtained, the central brain 15 derives the category of the label information (e.g., PERSON). Moreover, the central brain 15 makes the label information correspond to the obtained point information. Therefore, the central brain 15 identifies the x - coordinate value and y - coordinate value of the object based on the point information and identifies what the object is. The central brain 15 is an example of the "third processor".

[0175] Here, in the case where there are multiple objects such as object A and object B captured by an ultra-high-resolution camera and another camera, the central brain 15 makes the point information related to each object correspond to the label information as follows. There may be a state where due to the frame rate difference between the above-mentioned another camera and the ultra-high-resolution camera, the central brain 15 obtains the point information related to object A and object B (hereinafter referred to as "point information A" and "point information B"), but does not obtain the label information. In this state, the central brain 15 identifies the x coordinate value and y coordinate value of object A based on point information A and identifies the x coordinate value and y coordinate value of object B based on point information B, but does not identify what these objects are.

[0176] After that, when one label information is obtained, the central brain 15 derives the category of the one label information (for example, PERSON). Moreover, the central brain 15 determines the point information corresponding to the one label information based on the position information output from the IPU 11 together with the one label information and the position information included in the obtained point information A and point information B. For example, the central brain 15 determines the point information including the position information indicating the position closest to the object position indicated by the position information output from the IPU 11, and makes the point information correspond to the one label information. In the case where the determined point information is point information A, the central brain 15 makes the one label information correspond to point information A, identifies the x coordinate value and y coordinate value of object A based on point information A, and identifies what object A is.

[0177] As described above, in the case where there are multiple objects captured by an ultra-high-resolution camera and another camera, the central brain 15 makes the point information correspond to the label information based on the position information output from the IPU 11 and the position information included in the point information output from the MoPU 12.

[0178] In addition, the central brain 15 identifies the objects (people, animals, roads, signals, signs, crosswalks, obstacles, buildings, etc.) existing around the vehicle 100 based on the image and label information output from the IPU 11. In addition, the central brain 15 identifies the positions and actions of the objects existing around the vehicle 100 that have been identified as to what they are based on the point information output from the MoPU 12. The central brain 15 controls the autonomous driving of the vehicle 100 based on the identified information, for example, controls the motor of the driving wheels (speed control), braking control, and steering control. For example, the central brain 15 will control the autonomous driving of the vehicle 100 based on the position information and action information included in the point information output from the MoPU 12 to avoid collisions with objects. In the central brain 15, the GNPU 13 can be responsible for the processing related to image recognition, and the CPU 14 can be responsible for the processing related to vehicle control.

[0179] Generally speaking, ultra-high-resolution cameras are used for image recognition in autonomous driving. Here, from the images captured by the ultra-high-resolution camera (the images obtained by shooting with the ultra-high-resolution camera), it is possible to identify what the objects contained in the images are. However, in the autonomous driving in the Level 6 era, this alone is not enough. In the Level 6 era, it is also necessary to identify the actions of objects with higher precision. By using the MoPU 12 to identify the actions of objects with higher precision, it is possible to achieve, with higher precision, for example, the avoidance action of the vehicle 100 traveling by autonomous driving to avoid obstacles. However, the ultra-high-resolution camera can only acquire approximately 10 frames of images per second, and compared with the camera equipped with the MoPU 12, the precision of analyzing the actions of objects is lower. On the other hand, the camera equipped with the MoPU 12 can output at a high frame rate of, for example, 100 frames per second.

[0180] Therefore, the information processing device 10 according to the first embodiment includes two independent processors, the IPU 11 and the MoPU 12. The information processing device 10 causes the IPU 11 built in the ultra-high-resolution camera to have the function of acquiring the information required to identify what the captured object is, and causes the MoPU 12 built in another camera to have the function of detecting the position and actions of the object. The MoPU 12 captures the captured object as a point and analyzes in which direction and at what speed the coordinates of this point move at least along the x-axis and the y-axis in the above three-dimensional rectangular coordinate system. Since the overall contour of the object and what the object is can be detected from the images from the ultra-high-resolution camera, through the MoPU 12, for example, as long as it is known how the center point of the object moves, the overall behavior of the object can be known.

[0181] According to the means of only analyzing the movement and speed of the center point of the object, compared with judging how the entire image of the object moves, the amount of data output to the central brain 15 can be significantly reduced, and the amount of calculation in the central brain 15 can be significantly reduced. For example, in the case of outputting an image of 1000 pixels × 1000 pixels to the central brain 15 at a frame rate of 1000 frames per second, if color information is included, 4 billion bits per second of data will be output to the central brain 15. The MoPU 12 can compress the amount of data output to the central brain 15 to 20,000 bits per second by only outputting the point information indicating the action of the center point of the object. That is, the amount of data output to the central brain 15 is compressed to one two-hundred-thousandth.

[0182] In this way, by using in combination the low-frame-rate and high-resolution images and label information output from the IPU 11, and the high-frame-rate and lightweight point information output from the MoPU 12, object recognition including object actions can be achieved with a small amount of data.

[0183] In addition, in the information processing device 10, the central brain 15 makes the point information output from the MoPU 12 correspond to the tag information output from the IPU 11, and it is possible to grasp information related to what kind of object is performing what kind of action.

[0184] (Second Embodiment)

[0185] Next, for the second embodiment related to the present embodiment, it will be described while omitting or simplifying the parts overlapping with the above-described embodiment.

[0186] Figure 3 FIG. 1 is a first block diagram showing an example of the configuration of the information processing device 10. As Figure 3 shown, the information processing device 10 mounted on the vehicle 100 includes a MoPU 12L corresponding to the left eye, a MoPU 12R corresponding to the right eye, an IPU 11, and a central brain 15.

[0187] The MoPU 12L includes a camera 30L, a radar 32L, an infrared camera 34L, and a core 17L. For example, the MoPU 12L has a core 17L, and the camera 30L, the radar 32L, and the infrared camera 34L are connected to the core 17L. In addition, the MoPU 12R includes a camera 30R, a radar 32R, an infrared camera 34R, and a core 17R, and it is configured in the same manner as the MoPU 12L. Furthermore, hereinafter, when not distinguishing between the MoPU 12L and the MoPU 12R, it will be described as "MoPU 12", when not distinguishing between the camera 30L and the camera 30R, it will be described as "camera 30", when not distinguishing between the radar 32L and the radar 32R, it will be described as "radar 32", when not distinguishing between the infrared camera 34L and the infrared camera 34R, it will be described as "infrared camera 34", and when not distinguishing between the core 17L and the core 17R, it will be described as "core 17".

[0188] The camera 30 included in the MoPU 12 captures an object at a frame rate larger than that of the ultra-high-resolution camera (e.g., 10 frames per second) included in the IPU 11 (120, 240, 480, 960, or 1920 frames per second). The frame rate of the camera 30 is variable. The camera 30 is an example of the "first camera".

[0189] The radar 32 included in the MoPU 12 will acquire a radar signal, which is a signal of the reflected wave from the object based on the electromagnetic wave irradiated on the object. The infrared camera 34 included in the MoPU 12 is a camera that captures an infrared image (a camera that acquires an infrared image showing the object by capturing infrared rays from the object).

[0190] The core 17 included in the MoPU 12 (for example, composed of one or more CPUs) will extract feature points for each frame image captured by the camera 30 (a frame image obtained by capturing with the camera 30), and output the x coordinate value and y coordinate value of the object in the three-dimensional rectangular coordinate system as point information. For example, the core 17 takes the center point (center of gravity) of the object extracted from the image as a feature point. In addition, similar to the above-described embodiment, the point information output by the core 17 includes position information and motion information.

[0191] The IPU 11 includes an ultra-high-resolution camera (not shown), and outputs an image of the object captured by the ultra-high-resolution camera (an image obtained by capturing the object with the ultra-high-resolution camera), label information indicating the category of the object, and position information indicating the position of the object in the camera coordinate system of the ultra-high-resolution camera.

[0192] The central brain 15 will acquire the point information output from the MoPU 12 and the image, label information, and position information output from the IPU 11. Moreover, the central brain 15 makes the label information related to the object existing at the position corresponding to the position information included in the point information output from the MoPU 12 and the position information output from the IPU 11 correspond to the point information. Thus, the information processing device 10 can make the information on what the object indicated by the label information is correspond to the object position and motion indicated by the point information.

[0193] Here, the MoPU 12 changes the frame rate of the camera 30 according to a prescribed factor. In the present embodiment, as an example of the prescribed factor, the MoPU 12 changes the frame rate of the camera 30 according to a score related to the external environment. In this case, the MoPU 12 calculates a score related to the external environment of the vehicle 100, and changes the frame rate of the camera 30 according to the calculated score. Moreover, the MoPU 12 outputs a control signal for the camera 30 to capture an image at the changed frame rate. Thus, the camera 30 captures an image at the frame rate indicated by the control signal (the camera 30 obtains an image by capturing at the frame rate indicated by the control signal). According to this configuration, the information processing device 10 can capture an object image at a frame rate suitable for the external environment (can capture an object at a frame rate suitable for the external environment).

[0194] In addition, the information processing device 10 mounted on the vehicle 100 is equipped with a variety of sensors (not shown). The MoPU 12 calculates the risk associated with the movement of the vehicle 100 as a score related to the external environment of the vehicle 100 based on the sensor information obtained from the various sensors (e.g., movement of the center of gravity of the weight, detection of road material, detection of external air temperature, detection of external air humidity, detection of the inclination angles of the ramp in the up, down, left, right, and diagonal directions, road icing state, detection of moisture content, material, wear condition, and air pressure of each tire, road width, whether overtaking is prohibited, type information of oncoming vehicles and vehicles in front and behind, cruising states of these vehicles, or surrounding conditions (birds, animals, footballs, accident vehicles, earthquakes, fires, winds, typhoons, heavy rains, light rains, snowstorms, fog, etc.)) and the point information. The risk indicates the degree to which the vehicle 100 will travel in a dangerous place in the future. In this case, the MoPU 12 changes the frame rate of the camera 30 according to the calculated risk. The vehicle 100 is an example of a "moving body". With this configuration, the information processing device 10 can change the frame rate of the camera 30 according to the risk associated with the movement of the vehicle 100. The sensor is an example of a "detection unit", and the sensor information is an example of "detection information".

[0195] For example, the MoPU 12 makes the frame rate of the camera 30 higher as the calculated risk is higher. When the calculated risk is less than the first threshold, the MoPU 12 changes the frame rate of the camera 30 to 120 frames per second. In addition, when the calculated risk is equal to or greater than the first threshold and less than the second threshold, the MoPU 12 changes the frame rate of the camera 30 to any one of 240, 480, and 960 frames per second. In addition, when the calculated risk is equal to or greater than the second threshold, the MoPU 12 changes the frame rate of the camera 30 to 1920 frames per second. In addition, in the case where the risk is any of the above, in addition to causing the camera 30 to capture an image at the selected frame rate (causing the camera 30 to perform shooting at the selected frame rate), the MoPU 12 can also output a control signal to the radar 32 and the infrared camera 34 so that a radar signal is obtained and an infrared image is captured at a value corresponding to the frame rate.

[0196] For example, the lower the calculated risk level, the lower the frame rate of the camera 30 by the MoPU 12. When the frame rate of the camera 30 is set to 1920 frames per second, if the calculated risk level is equal to or greater than the first threshold and less than the second threshold, the MoPU 12 changes the frame rate of the camera 30 to any one of 240, 480, and 960 frames per second. Additionally, when the frame rate of the camera 30 is set to 1920 frames per second, if the calculated risk level is less than the first threshold, the MoPU 12 changes the frame rate of the camera 30 to 120 frames per second. Furthermore, when the frame rate of the camera 30 is set to any one of 240, 480, and 960 frames per second, if the calculated risk level is less than the first threshold, the MoPU 12 changes the frame rate of the camera 30 to 120 frames per second. Moreover, in this case, similar to the above situation, in order to obtain radar signals and capture infrared images with values corresponding to the changed frame rate of the camera 30, a control signal can be output to the radar 32 and the infrared camera 34 (obtaining radar signals with values corresponding to the changed frame rate of the camera 30 and performing shooting for obtaining infrared images).

[0197] In addition, the MoPU 12 can use big data on driving that is known before the vehicle 100 travels, such as long tail incident AI (Artificial Intelligence) data (e.g., travel data of a vehicle implementing a level 5 autonomous driving control mode) or map information, as information for predicting the risk level to calculate the risk level.

[0198] For example, in the information processing device 10, a sensor for detecting the vehicle position through GPS (Global Positioning System) can also be provided, and while referring to the map information, the risk level can be calculated based on the vehicle position. In this case, in a storage device (not shown) provided in the information processing device 10, a table or the like that correlates the vehicle position with the risk level is prepared in advance. In the table, for example, near intersections corresponds to a relatively high risk level, highways correspond to a relatively low risk level, and residential roads correspond to a relatively high risk level.

[0199] When the vehicle is traveling near an intersection or on a residential road, since the risk level obtained by referring to the table is relatively high, the MoPU 12 changes the frame rate to, for example, 1920 frames per second. Additionally, when the vehicle is traveling on a highway, since the risk level obtained by referring to the table is relatively low, the MoPU 12 changes the frame rate to, for example, 120 frames per second.

[0200] Although in the above description, the risk level is calculated as a score related to the external environment, the indicators used as scores related to the external environment are not limited to the risk level. For example, the MoPU 12 can calculate a score related to the external environment different from the risk level based on the moving direction, speed, etc. of the object captured by the camera 30, and change the frame rate of the camera 30 according to this score. Hereinafter, a case where the MoPU 12 calculates a speed score and changes the frame rate of the camera 30 according to the speed score will be described. The speed score is a score related to the speed of the object captured by the camera 30. As an example, the speed score is set to be higher when the object speed is faster and lower when the object speed is slower. Moreover, the MoPU 12 makes the frame rate of the camera 30 higher when the calculated speed score is higher, and the frame rate of the camera 30 lower when the calculated speed score is lower. Therefore, when the calculated speed score is equal to or greater than the threshold due to the fast object speed, the MoPU 12 changes the frame rate of the camera 30 to 1920 frames per second. In addition, when the calculated speed score is less than the threshold due to the slow object speed, the MoPU 12 changes the frame rate of the camera 30 to 120 frames per second. In addition, in this case, similar to the above case, in order to acquire radar signals and capture infrared images with values corresponding to the changed frame rate of the camera 30, control signals can be output to the radar 32 and the infrared camera 34 (acquire radar signals with values corresponding to the changed frame rate of the camera 30 and perform shooting for obtaining infrared images).

[0201] Hereinafter, a case where the MoPU 12 calculates a direction score and changes the frame rate of the camera 30 according to the speed score will be described. The direction score is a score related to the moving direction of the object captured by the camera 30. As an example, the direction score is set to be higher when the moving direction of the object is the direction approaching the road and lower when the moving direction of the object is the direction away from the road. Moreover, the MoPU 12 makes the frame rate of the camera 30 higher when the calculated direction score is higher, and the frame rate of the camera 30 lower when the calculated direction score is lower. Specifically, the MoPU 12 determines the moving direction of the object by using AI or the like, and calculates a direction score based on the determined moving direction. Moreover, when the calculated direction score is equal to or greater than the threshold due to the moving direction of the object being the direction approaching the road, the MoPU 12 changes the frame rate of the camera 30 to 1920 frames per second. In addition, when the calculated direction score is less than the threshold due to the moving direction of the object being the direction away from the road, the MoPU 12 changes the frame rate of the camera 30 to 120 frames per second. In addition, in this case, similar to the above case, in order to acquire radar signals and capture infrared images with values corresponding to the changed frame rate of the camera 30, control signals can be output to the radar 32 and the infrared camera 34.

[0202] In addition, the MoPU 12 may output point information only for an object whose calculated score related to the external environment is equal to or greater than a predetermined threshold. In this case, for example, the MoPU 12 may determine whether to output point information related to the object according to the moving direction of the object captured by the camera 30. For example, the MoPU 12 may not output point information related to an object that has a relatively small impact on the driving of the vehicle 100. Specifically, the MoPU 12 calculates the moving direction of the object captured by the camera 30 and does not output point information related to an object such as a pedestrian away from the road. On the other hand, the MoPU 12 outputs point information related to an object approaching the road (for example, an object such as a pedestrian who may suddenly rush into the road). With this configuration, the information processing device 10 may not output point information related to an object that has a relatively small impact on the driving of the vehicle 100.

[0203] In addition, the MoPU 12 may calculate a score related to the external environment based on the object category, and change the frame rate of the camera 30 according to the score, where the object category is based on the recognition information output by the IPU 11. For example, the degree of danger may be calculated as the score, and the frame rate of the camera 30 may be changed according to the degree of danger. Hereinafter, the case of changing the frame rate of the camera 30 according to the score based on the object category will be described. For example, when the object is an animal with agile movements such as a person, a dog, and a deer, the degree of danger related to the movement of the vehicle 100 is relatively high. Therefore, the MoPU 12 will calculate a relatively high score and increase the frame rate of the camera 30 according to the calculated score. Specifically, the frame rate is changed to 1920 frames per second. On the other hand, when the object is a vehicle with a relatively small change in moving speed or a stationary object, the degree of danger related to the movement of the vehicle 100 is relatively low. Therefore, the MoPU 12 calculates a relatively low score and decreases the frame rate of the camera 30 according to the calculated score. Specifically, the frame rate of the camera 30 is changed to any one of 240, 480, and 960 frames per second.

[0204] In addition, the object category can be classified more finely, and by stepwise changing the degree of danger according to the classified object category, the frame rate is stepwise changed according to the stepwise changed degree of danger. For example, animals other than humans such as dogs and deer are more agile than humans. Therefore, in the case where the object category is an animal other than a human, the MoPU 12 may increase the frame rate compared to a human. Specifically, in the case where the object category is an animal other than a human, the frame rate of the camera 30 can be changed to 1920 frames per second, and in the case where the object category is a human, the frame rate of the camera 30 can be changed to 960 frames per second.

[0205] In addition, in the IPU 11, the number of recognized objects is detected. In the MoPU 12, based on the object category and the number of objects, a score related to the external environment can be calculated, and according to the calculated score, the frame rate of the camera 30 is changed. The object category and the number of objects are based on the recognition information output by the IPU 11. Hereinafter, the case of changing the frame rate of the camera 30 according to the score based on the object category and the number of objects will be described. For example, the MoPU 12 makes the frame rate of the camera 30 higher as the number of objects increases. Here, the MoPU 12 will derive a score S1 of the risk level corresponding to the object category and a score S2 of the risk level corresponding to the number of objects. The larger the number of objects, the larger the score S2 of the risk level corresponding to the number of objects. For example, the score of the risk level is set to a value greater than or equal to 0 and less than or equal to 1, and the first to fourth thresholds are set according to the number of objects. For example, 0.2, 0.4, 0.6, and 0.8 can be used as the first to fourth thresholds.

[0206] Moreover, the MoPU 12 multiplies the score S1 corresponding to the object category by the score S2 corresponding to the number of objects, thereby calculating scores S1 and S2 related to the external environment based on the object category and the number of objects. Then, the MoPU 12 changes the frame rate of the camera 30 based on the calculated scores S1 and S2. Here, when the object is a person, when the number of detected people is greater than or equal to a predetermined number, the risk level related to the movement of the vehicle 100 is relatively high. Therefore, the MoPU 12 will increase the frame rate of the camera 30. Specifically, the frame rate is changed to 1920 frames per second. On the other hand, when the number of people is less than the predetermined number, the risk level related to the movement of the vehicle 100 is slightly reduced, so the MoPU 12 changes the frame rate of the camera 30 to, for example, 960 frames per second. In addition, when the object is a stationary object, when the number of detected stationary objects is greater than or equal to a predetermined number, the risk level related to the movement of the vehicle 100 is lower than that when the object is a person. Therefore, the MoPU 12 will make the frame rate of the camera 30 lower than the case when the object is a person. Specifically, the frame rate is changed to 240 frames per second. On the other hand, when the number of stationary objects is less than the predetermined number, the risk level related to the movement of the vehicle 100 is smaller than the case when the number of objects is equal to or greater than the predetermined number, so the MoPU 12 changes the frame rate of the camera 30 to, for example, 120 frames per second.

[0207] In addition, although the case where the MoPU 12 calculates the risk level is illustrated in the above description, the technology of the disclosed content is not limited to this aspect. For example, the central brain 15 may calculate the risk level instead of the MoPU 12. In this case, the central brain 15 calculates the risk level related to the movement of the vehicle 100 as a score related to the external environment of the vehicle 100 based on the sensor information acquired from various sensors and the point information output from the MoPU 12. Moreover, the central brain 15 outputs an instruction to change the frame rate of the camera 30 to the MoPU 12 according to the calculated risk level.

[0208] In addition, although the case where the MoPU 12 outputs point information based on the image captured by the camera 30 (the image obtained by capturing through the camera 30) is illustrated in the above description, the technology of the disclosed content is not limited to this aspect. For example, instead of the image captured by the camera 30 (the image obtained by capturing through the camera 30), the MoPU 12 may output point information based on radar signals and infrared images. The MoPU 12 can derive the x - coordinate value and y - coordinate value of an object from the infrared image of the object captured by the infrared camera 34 in the same way as the image captured by the camera 30 (the image obtained by capturing through the camera 30). The radar 32 can acquire the three - dimensional point cloud data of the object based on the radar signal. That is, the radar 32 can detect the coordinate of the z - axis in the three - dimensional rectangular coordinate system. Here, the z - axis is the axis along the depth direction of the object and the traveling direction of the vehicle 100. In the following text, the coordinate value of the z - axis is described as the "z - coordinate value". In this case, the MoPU 12 uses the principle of a stereo camera to combine the x - coordinate value and y - coordinate value of the object captured by the infrared camera 34 at the same timing as the timing when the radar 32 acquires the three - dimensional point cloud data of the object with the z - coordinate value of the object indicated by the three - dimensional point cloud data, and derives the coordinate values of the three axes (x - axis, y - axis, and z - axis) of the object as point information. Then, the MoPU 12 outputs the derived point information to the central brain 15.

[0209] In addition, when the MoPU 12 derives the z coordinate value of an object in this way, the frame rate of the camera 30 can be changed according to the z coordinate value. The z coordinate value is an example of the depth direction coordinate value of the present disclosure. As described above, the z coordinate value is the coordinate value in the depth direction of the object and the traveling direction of the vehicle 100. Therefore, if the z coordinate value is large, the object exists at a position far from the vehicle 100. Thus, for example, a threshold value is set for the z coordinate value. When the z coordinate value is equal to or greater than the threshold value, the object exists at a position far from the vehicle 100 and the risk level is relatively low, so the frame rate is decreased (for example, 120 frames per second). On the other hand, when the z coordinate value is less than the threshold value, the object exists at a position close to the vehicle 100, so the risk level is relatively high, and thus the frame rate is increased (for example, 1920 frames per second).

[0210] In addition, the frame rate can be finely changed by finely setting the threshold value of the z coordinate value step by step. For example, three threshold values that increase in the order of the first threshold value, the second threshold value, and the third threshold value are set. When the z coordinate value is less than the first threshold value, the frame rate can be set to, for example, 1920 frames per second. When the z coordinate value is greater than or equal to the first threshold value and less than the second threshold value, the frame rate can be set to, for example, 960 frames per second. When the z coordinate value is greater than or equal to the second threshold value and less than the third threshold value, the frame rate can be set to 480 frames per second. When the z coordinate value is greater than or equal to the third threshold value, the frame rate can be set to 120 frames per second.

[0211] In addition, although in the above description, the case where the MoPU 12 derives the point information is exemplified, the technology of the present disclosure is not limited to this aspect. For example, the central brain 15 can derive the point information instead of the MoPU 12. The central brain 15 derives the point information, for example, by combining the information detected by the camera 30L, the camera 30R, the radar 32, and the infrared camera 34. As a specific example, the central brain 15 performs three-point measurement based on the x coordinate value and the y coordinate value of the object captured by the camera 30L and the x coordinate value and the y coordinate value of the object captured by the camera 30R, thereby deriving the coordinate values of the three coordinate axes (x-axis, y-axis, and z-axis) of the object as the point information.

[0212] In addition, although in the above description, a case where the central brain 15 controls the autonomous driving of the vehicle 100 based on the images and label information output from the IPU 11 and the point information output from the MoPU 12 is illustrated, the technology of the disclosed content is not limited to this aspect. For example, the central brain 15 can perform motion control of a robot based on the above information output from the IPU 11 and the MoPU 12. The robot can be a humanoid intelligent robot that performs operations as a substitute for a human. In this case, the central brain 15 performs motion control of the robot's arms, palms, fingers, feet, etc. based on the above information output from the IPU 11 and the MoPU 12, enabling it to perform actions such as grasping, holding, hugging, carrying on the back, moving, transporting, throwing, kicking, and avoiding objects. When the central brain 15 performs motion control of the robot, the IPU 11 and the MoPU 12 can be mounted at the positions of the right and left eyes of the robot. That is, the IPU 11 and the MoPU 12 for the right eye can be mounted on the right eye, and the IPU 11 and the MoPU 12 for the left eye can be mounted on the left eye.

[0213] In addition, in the second aspect of the present embodiment, as an example of a specified factor, the MoPU 12 changes the frame rate of the camera 30 according to the position information. The position information mentioned here can be the position of the camera 30, the information processing device 10 including the camera 30, or the position where the information processing device 10 is mounted. Specifically, it can be the position of the vehicle 100 or the robot. In the following description, a case where the position information is the position information of the vehicle 100 is illustrated. In this case, the MoPU 12 can change the frame rate of the camera 30 considering the likelihood of detecting an object at the position where the vehicle 100 is traveling.

[0214] When the position where the vehicle 100 is traveling is a position where the likelihood of the camera 30 detecting an object is high, such as a road with a large flow of people, the MoPU 12 will increase the frame rate of the camera 30. On the contrary, when the position where the vehicle 100 is traveling is a position where the likelihood of the camera 30 detecting an object is low, such as a road with a small flow of people, the MoPU 12 will decrease the frame rate of the camera 30. In this way, if the frame rate of the camera 30 is changed considering the likelihood of detecting an object, the amount of data output from the MoPU 12 to the central brain 15 can be further compressed.

[0215] At a position where the possibility of the object being detected by the above-described camera 30 is relatively high, change the frame rate of the camera 30 to, for example, 1920 frames per second. On the other hand, at a position where the possibility of the object being detected by the camera 30 is relatively low, change the frame rate of the camera 30 to, for example, 120 frames per second. In addition, the switching of the above frame rate is an example, and for example, it can also be stepped to any one of optional multiple frame rates such as 120, 240, 480, 960, and 1920 frames per second in accordance with the possibility of the object being detected by the camera 30.

[0216] In order to estimate the possibility of the object being detected at the position where the vehicle 100 travels, the MoPU 12 can collect the frequency of the object being previously detected at each position around the camera 30 and generate corresponding collected data or a heat map. For example, the frequency of the object being previously detected can be collected from the historical information of the object being previously detected by the camera 30, and / or by obtaining the frequency of the object being previously detected from a server (not shown) that records the historical information collected at the plurality of information processing devices 10 in the central brain 15 via a network. The MoPU 12 generates a heat map reflecting the previously detected frequency of the object based on the collected information, and changes the frame rate of the camera 30 based on the position information of the vehicle 100 and the above heat map. Thus, if the frame rate is changed using the heat map, the possibility of the object being detected at the position where the vehicle 100 travels can be estimated immediately and with high accuracy.

[0217] The first aspect and the second aspect of the above-described embodiment can be combined. That is, the MoPU 12 can also change the frame rate of the camera 30 according to the position information of the camera 30 and a score related to the external environment.

[0218] In addition, in the third aspect of the present embodiment, the MoPU 12 changes the frame rate of the camera 30 according to user information obtained from the user as an example of a prescribed factor. In this case, the MoPU 12 can determine the optimal frame rate of the camera 30 based on information that can be obtained from the user, for example, an occupant of the vehicle 100 equipped with at least a part of the information processing device 10.

[0219] As the user information obtained from the user, various types can be imagined. For example, it can include at least one of voice information from the user, image information of the user, and heart rate information of the user. In addition, the user information is not limited to the above information, and can also include information input by the user via an input device such as a button.

[0220] In the case where voice information from the user is used as user information, the information processing device 10 can be connected to a microphone (not shown) disposed at an appropriate position in the vehicle 100, for example, to acquire the voice information. For example, when the voice information acquired via the microphone includes voices such as "poor visibility" or "many people" spoken by the user, the MoPU 12 determines that the possibility of the camera 30 detecting an object is relatively high, and increases the frame rate so as to immediately detect an object. On the contrary, when the voice information acquired via the microphone includes voices such as "good visibility" and "no one" issued by the user, for example, the MoPU 12 determines that the possibility of the camera 30 detecting an object is relatively low, and decreases the frame rate.

[0221] In the case where image information of the user is used as user information, the information processing device 10 can be connected to an in-vehicle camera (not shown) disposed at an appropriate position in the vehicle 100, for example, to acquire the image information. The in-vehicle camera can be set to a direction capable of photographing the user's expression in the vehicle interior. When it is detected from the image information acquired via the in-vehicle camera that the user's expression is tense or the user is carefully observing the surrounding environment, the MoPU 12 determines that the possibility of the camera 30 detecting an object is relatively high, and increases the frame rate so as to immediately detect an object. On the contrary, when the user's expression is relaxed, the MoPU 12 determines that the possibility of the camera 30 detecting an object is relatively low, and decreases the frame rate.

[0222] In the case where the user's heart rate information is used as user information, the information processing device 10 can be connected to a sensor (not shown) disposed on a seat in the vehicle interior or the like, for example, to acquire the heart rate information. When the heart rate per unit time included in the heart rate information acquired via the sensor is higher than the normal heart rate of the user, it is determined that the user is tense and the possibility of the camera 30 detecting an object is relatively high, and the frame rate is increased so as to immediately detect an object. On the contrary, when the heart rate per unit time included in the heart rate information acquired via the sensor is lower than the normal heart rate of the user, the MoPU 12 determines that the possibility of the camera 30 detecting an object is relatively low, and decreases the frame rate.

[0223] At a position where the possibility of the camera 30 detecting an object is relatively high as described above, the frame rate of the camera 30 is changed to, for example, 1920 frames per second. On the other hand, at a position where the possibility of the camera 30 detecting an object is relatively low, the frame rate of the camera 30 is changed to, for example, 120 frames per second. In addition, the switching of the above frame rate is an example, and for example, it can also be stepped to any one of optional multiple frame rates such as 120, 240, 480, 960, and 1920 frames per second in accordance with the possibility of the camera 30 detecting an object.

[0224] As described above, if the frame rate of the camera 30 is changed by estimating the possibility of detecting an object from the user information, the amount of data output from the MoPU 12 to the central brain 15 can be further compressed. In addition, the first aspect and the second aspect of the above-described embodiment can be combined. That is, the MoPU 12 can also change the frame rate of the camera 30 according to the user information obtained from the user and the score related to the external environment.

[0225] (Third Embodiment)

[0226] Next, for the third embodiment related to this embodiment, the description will be given while omitting or simplifying the parts repeated with the above embodiments.

[0227] As an example, the information processing device 10 related to the third embodiment has the same configuration as shown in the Figure 2 first embodiment.

[0228] The MoPU 12 related to the third embodiment outputs, as point information, the coordinate values of at least two diagonally opposite points among the vertices of the polygon that encloses the object contour identified from the image captured by another camera (the image obtained by capturing with another camera). Similar to the first embodiment, the coordinate values are the x coordinate value and the y coordinate value of the object in the above three-dimensional rectangular coordinate system.

[0229] Figure 4 is an explanatory diagram showing an example of the point information output by the MoPU 12. In Figure 4 the figure, the MoPU 12 shows bounding boxes 21, 22, 23, and 24, which are formed by enclosing the contour of each of the four objects included in the image captured by another camera (the image obtained by capturing with another camera) with a quadrilateral. Moreover, Figure 4 an aspect is illustrated in which the MoPU 12 outputs the coordinate values of two diagonally opposite points among the vertices of the quadrilateral bounding boxes 21, 22, 23, and 24 that enclose the object contour as point information. In this way, the MoPU 12 can capture the object as an object with a certain size rather than a point.

[0230] In addition, when capturing the object as an object with a certain size, the MoPU 12 can output, as point information, the coordinate values of multiple vertices of the polygon that encloses the object contour, rather than the coordinate values of two diagonally opposite points among the vertices of the polygon that encloses the object contour identified from the image captured by another camera (the image obtained by capturing with another camera). For example, Figure 4For example, the MoPU 12 can output the coordinate values of all four vertices of the bounding boxes 21, 22, 23, and 24 that enclose the outline of the object with a quadrilateral as point information.

[0231] (Fourth Embodiment)

[0232] Next, for the fourth embodiment related to this embodiment, it will be described while omitting or simplifying the parts that overlap with the above embodiments.

[0233] As an example, the information processing device 10 related to the fourth embodiment has the same configuration as shown in the Figure 2 above.

[0234] The vehicle 100 equipped with the information processing device 10 related to the fourth embodiment includes sensors composed of at least one of a radar, a lidar, a high-pixel / long-focus / ultra-wide-angle / 360-degree / high-performance camera, a vision sensor, a sound sensor, an ultrasonic sensor, a vibration sensor, an infrared sensor, a ultraviolet sensor, a radio wave sensor, a temperature sensor, and a humidity sensor. As the sensor information obtained by the information processing device 10 from the sensors, examples include the movement of the center of gravity of the body weight, the detection of the road material, the detection of the external air temperature, the detection of the external air humidity, the detection of the inclination angles of the ramp in the up / down / left / right / diagonal directions, the road icing state, the detection of the moisture content, the material, wear condition, and air pressure of each tire, the road width, whether overtaking is prohibited, the vehicle types of oncoming vehicles and front / rear vehicles, the cruising states of these vehicles, the surrounding conditions (birds, animals, footballs, accident vehicles, earthquakes, fires, winds, typhoons, heavy rains, light rains, snowstorms, fogs, etc.). The sensor is an example of the "detection unit", and the sensor information is an example of the "detection information".

[0235] The central brain 15 related to the fourth embodiment calculates control variables for controlling the autonomous driving of the vehicle 100 based on the sensor information detected by the sensors. The central brain 15 acquires the sensor information once every billionth of a second. Specifically, the central brain 15 calculates the control variables for the wheel speeds, inclinations, and suspensions that support the wheels of each of the four wheels of the vehicle 100. In addition, the inclination of the wheel includes the inclination of the wheel with respect to the axis horizontal to the road and the inclination of the wheel with respect to the axis perpendicular to the road. In this case, the central brain 15 calculates the control variables for the wheel speeds of each of the four wheels, the inclinations of each of the four wheels with respect to the axis horizontal to the road, the inclinations of each of the four wheels with respect to the axis perpendicular to the road, and the suspensions that support the four wheels respectively, for a total of 16 control variables.

[0236] Moreover, the central brain 15 controls the autonomous driving of the vehicle 100 based on the control variables calculated above, the point information output from the MoPU 12, and the label information output from the IPU 11. Specifically, the central brain 15 controls the in-wheel motors mounted on the four wheels respectively based on the above 16 control variables, thereby controlling the wheel speed, inclination, and suspension that supports each of the four wheels of the vehicle 100 to perform autonomous driving. In addition, the central brain 15 identifies the positions and movements of the objects recognized as existing around the vehicle 100 and what they are based on the point information and label information, and controls the autonomous driving of the vehicle 100 based on the identified information to avoid collisions with objects, for example. By controlling the autonomous driving of the vehicle 100 in this way by the central brain 15, for example, when the vehicle 100 is driving on a mountain road, it can perform the optimal steering suitable for the mountain road, and when the vehicle 100 is parked in a parking lot, it can drive at the optimal angle suitable for the parking lot.

[0237] Here, the central brain 15 can use machine learning, more specifically, deep learning, to infer the control variables based on the above sensor information and the information that can be obtained via a network from a server (not shown), etc. In other words, the central brain 15 can be composed of AI.

[0238] The central brain 15 can perform multivariate analysis (for example, refer to formula (2)) based on the above sensor information per billionth of a second, the computing power of the long-tail event AI data required to achieve Level 6 (hereinafter also referred to as "Level 6 computing power"), and the integration method shown in the following formula (1) to obtain the control variables. More specifically, while obtaining the integral value of the delta values of various ultra-high resolutions through the Level 6 computing power, it can obtain each control variable at the edge level and in real time, and obtain the results generated in the next billionth of a second (i.e., each control variable) with the highest probability value. To achieve this, for example, the integral value obtained by performing time integration on the delta values (for example, the change values in a short time) of the functions of each variable (for example, the above sensor information and the information that can be obtained via the network) that can determine air resistance, road resistance, road factors (for example, garbage), and slip coefficient, etc. (in other words, the functions indicating the behavior of each variable) is input into the deep learning model of the central brain 15 (for example, a pre-trained model obtained by performing deep learning on a neural network). The deep learning model of the central brain 15 outputs the control variable corresponding to the input integral value (for example, the control variable with the highest confidence level (i.e., evaluation value)). The output of the control variable is in units of billionths of a second.

[0239] [Mathematical formula 1]

[0240]

[0241] [Mathematical formula 2]

[0242] V n = DL(f(A, B, C, D, …, N)(dA n / dt)) (2)

[0243] In addition, as an example, in formula (1), "f(A)" is a formula expressed by simplifying a function that indicates the behaviors of various variables such as air resistance, road resistance, road factors (such as garbage), and slip coefficient. Additionally, as an example, formula (1) is a formula that represents the time integral v of "f(A)" from time a to time b. DL in formula (2) represents deep learning (for example, a deep learning model optimized by performing deep learning on a neural network), and dA n / dt represents the incremental value of f(A, B, C, D, …, N), A, B, C, D, …, N represent variables such as air resistance, road resistance, road factors (such as garbage), and slip coefficient, f(A, B, C, D, …, N) represents a function that indicates the behaviors of A, B, C, D, …, N, and V n represents the value (control variable) output from the deep learning model optimized by performing deep learning on a neural network.

[0244] Furthermore, although an example is listed here in which the integral value obtained by performing a time integral on the incremental value of a function is input into the deep learning model of the central brain 15, this is merely an example. For example, the integral value obtained by performing a time integral on the incremental value of a function that indicates the behaviors of various variables such as air resistance, road resistance, road factors, and slip coefficient (for example, the result generated in the next billionth of a second) can be inferred by the deep learning model of the central brain 15, and as an inference result, the integral value with the highest confidence level (i.e., evaluation value) can be obtained by the central brain 15 every billionth of a second.

[0245] In addition, although an example is listed here in which the integral value is input into the deep learning model or the integral value is output from the deep learning model, this is merely an example, and the technology of the present disclosure can be implemented even without using the integral value. For example, at least one control variable can be inferred by a deep learning model that is optimized by performing deep learning on a neural network using teacher data, where the teacher data uses values equivalent to A, B, C, D, …, N as example data and values equivalent to at least one control variable (for example, the result generated in the next billionth of a second) as correct answer data.

[0246] The control variables obtained in the central brain 15 can be further refined by increasing the number of deep learning. For example, more accurate control variables can be calculated by using a large amount of data such as tires, motor rotation, steering angle, road material, weather, garbage, impact during quadratic curve deceleration, slipping, loss of balance, or steering and speed control methods for restoring balance, and long-tail event AI data.

[0247] (Fifth Embodiment)

[0248] Next, a fifth embodiment according to the present embodiment will be described while omitting or simplifying portions overlapping with the above-described embodiments.

[0249] Figure 5 3 is a block diagram showing an example of the structure of the information processing device 10. Figure 5 Only a part of the configuration of the information processing device 10 is shown.

[0250] like Figure 5 As shown, in the MoPU 12, the visible light image and the infrared image of the object captured by the camera 30 are input to the core 17 at a frame rate of 100 frames / second or more. The camera 30 is configured to include a visible light camera 30A capable of capturing a visible light image of the object and an infrared camera 30B capable of capturing an infrared image of the object. In addition, the core 17 outputs point information to the central brain 15 based on at least one of the input visible light image and the infrared image.

[0251] Here, when the core 17 is able to identify an object from the visible light image of the object captured by the visible light camera 30A, the core 17 outputs point information based on the visible light image. On the other hand, when the core 17 cannot capture the object from the visible light image due to specified factors, the core 17 outputs point information based on the infrared image of the object captured by the infrared camera 30B. For example, imagine a situation where the core 17 cannot capture the object from the visible light image due to the influence of the dark environment as a specified factor. In this case, the core 17 uses the infrared camera 30B to detect the heat of the object, and outputs the point information of the object based on the infrared image as the detection result. In addition, without limitation to this, the core 17 can output point information based on the visible light image and the infrared image.

[0252] In addition, MoPU 12 synchronizes the timing of taking a visible light image by the visible light camera 30A (performing shooting for obtaining a visible light image through the visible light camera 30A) with the timing of taking an infrared image by the infrared camera 30B (performing shooting for obtaining an infrared image through the infrared camera 30B). Specifically, MoPU 12 outputs a control signal to the camera 30 to take a visible light image and an infrared image at the same timing (MoPU 12 performs shooting of visible light and infrared rays). Thereby, the number of images per second taken by the visible light camera 30A (the number of images per second obtained by performing shooting through the visible light camera 30A) is synchronized with the number of images per second taken by the infrared camera 30B (the number of images per second obtained by performing shooting through the infrared camera 30B) (for example, 1920 frames / second).

[0253] (Sixth Embodiment)

[0254] Next, regarding the sixth embodiment related to this embodiment, it will be described while omitting or simplifying parts that overlap with the above embodiments.

[0255] Figure 6 It is the fourth block diagram showing an example of the configuration of the information processing device 10. In addition, Figure 6 only a part of the configuration of the information processing device 10 is shown.

[0256] As Figure 6 shown, in MoPU 12, the image of the object captured by the camera 30 and the radar signal based on the reflected wave from the object of the electromagnetic wave irradiated onto the object by the radar 32 are respectively input to the core 17 at a frame rate of 100 frames / second or more. Then, based on the input object image and radar signal, the core 17 outputs point information to the central brain 15. The core 17 can derive the x coordinate value and y coordinate value of the object from the input object image. As described above, the radar 32 can acquire the three-dimensional point cloud data of the object based on the radar signal and detect the coordinate of the z axis in the three-dimensional rectangular coordinate system. In this case, the core 17 uses the principle of a stereo camera to combine the x coordinate value and y coordinate value of the object captured by the camera 30 at the same timing as the timing when the radar 32 acquires the three-dimensional point cloud data of the object with the z coordinate value of the object indicated by the three-dimensional point cloud data, and derives the coordinate values of the three coordinate axes (x axis, y axis, and z axis) of the object as point information. In addition, the object image input to the core 17 above may include at least one of a visible light image and an infrared image.

[0257] In addition, the MoPU 12 synchronizes the timing of capturing an image by the camera 30 (the camera 30 captures an image) with the timing of the radar 32 acquiring three-dimensional point cloud data of an object based on a radar signal. Specifically, the MoPU 12 captures an image at the same timing and outputs a control signal to the camera 30 and the radar 32 to acquire three-dimensional point cloud data of the object. Thereby, the number of images per second captured by the camera 30 (the number of images per second obtained by capturing an image with the camera 30) is synchronized with the number of three-dimensional point cloud data per second acquired by the radar 32 (for example, 1920 frames / second). Thereby, the number of images per second captured by the camera 30 and the number of three-dimensional point cloud data per second acquired by the radar 32 are more than the frame rate of the ultra-high-resolution camera provided in the IPU 11, that is, more than the number of images per second captured by the ultra-high-resolution camera (the number of images per second obtained by capturing an image with the ultra-high-resolution camera).

[0258] (Seventh Embodiment)

[0259] Next, regarding the seventh embodiment related to the present embodiment, it will be described while omitting or simplifying parts that overlap with the above-described embodiments.

[0260] As an example, the information processing apparatus 10 related to the seventh embodiment has the same configuration shown in Figure 2 the

[0261] According to the central brain 15 related to the seventh embodiment, at the same timing as the timing when the IPU 11 outputs label information, the point information output from the MoPU 12 is made to correspond to the label information. In addition, when new point information is output from the MoPU 12 after the point information is made to correspond to the label information, the central brain 15 also makes the new point information correspond to the label information. The new point information is the point information of the same object as the object indicated by the point information corresponding to the label information, and is one or more pieces of point information within the period until the next label information is output after the correspondence is made. In the seventh embodiment, similar to the above-described embodiment, the frame rate of another camera incorporating the MoPU 12 is 100 frames / second or more (for example, 1920 frames / second), and the frame rate of the ultra-high-resolution camera incorporating the IPU 11 is 10 frames / second.

[0262] Figure 7 is an explanatory diagram showing an example of making point information correspond to label information. In the following description, the number of pieces of point information output per second from the MoPU 12 is referred to as the "output rate of point information", and the number of pieces of label information output per second from the IPU 11 is referred to as the "output rate of label information".

[0263] In Figure 7 it, a time series of the output rate of the point information P4 of the object B14 is shown. The output rate of the point information P4 related to the object B14 is 1920 frames per second. In addition, the point information P4 is moving from right to left in the figure. The output rate of the label information related to the object B14 is 10 frames per second, which is lower than the output rate of the point information P4.

[0264] First, at the time point t0, the label information related to the object B14 is not output from the IPU 11. Therefore, at the time point t0, although the central brain 15 identifies the coordinate value (position information) of the object B14 based on the point information P4, it does not identify what the object B14 is.

[0265] Next, at the time point t1, the label information related to the object B14 is output from the IPU 11. Therefore, based on this label information, the central brain 15 derives the label information "PERSON" for the object B14. Moreover, the central brain 15 makes the label information "PERSON" derived at the time t1 correspond to the coordinate value (position information) of the point information P4 output from the MoPU 12 at the time t1. Thus, at the time point t1, the central brain 15 identifies the coordinate value (position information) of the object B14 based on the point information P4 and identifies what the object B14 is.

[0266] In Figure 7 it, the timing at which the next label information related to the object B14 is output from the IPU 11 is set to the time point t2. Therefore, at the time point t2, based on the label information output from the IPU 11, the central brain 15 derives the label information "PERSON" for the object B14. Moreover, the central brain 15 makes the label information "PERSON" derived at the time t2 correspond to the coordinate value (position information) of the point information P4 output from the MoPU 12 at the time t2.

[0267] Here, due to the frame rate difference between another camera with the MoPU 12 and the ultra-high resolution camera with the IPU 11, during the period from the time point t1 to the time point t2, the central brain 15 obtains the point information P4 related to the object B14, but does not obtain the label information. In this case, for the point information P4 obtained by the central brain 15 during the period from the time point t1 to the time point t2, it is made to correspond to the label information "PERSON" corresponding to the point information P4 at the previous time point t1. Here, the point information P4 obtained by the central brain 15 during the period from the time point t1 to the time point t2 is an example of "new point information". In Figure 7In the example shown, since a plurality of point information P4 was output from the MoPU 12 during the period from time t1 to time t2, the central brain 15 acquired the plurality of point information P4. Therefore, in Figure 7 In the example shown, the central brain 15 makes any one of the plurality of point information P4 acquired during the period from time t1 to time t2 correspond to the label information "PERSON" corresponding to the previous time t1. In addition, different from Figure 7 the example shown, when one point information P4 is output from the MoPU 12 during the period from time t1 to time t2, the central brain 15 makes the one point information P4 correspond to the label information "PERSON" corresponding to the previous time t1.

[0268] Here, even when there is a period in which the category of the object with the tracking action is uncertain, the central brain 15 continuously outputs the point information of the object at a high frame rate, so the risk of losing the coordinate value (position information) of the object is low. Therefore, when the point information is made to correspond to the label information once, the central brain 15 can speculatively assign the previous label information to the point information acquired during the period before the next label information is acquired.

[0269] (Eighth Embodiment)

[0270] Next, regarding the eighth embodiment according to the present embodiment, it will be described while omitting or simplifying the parts repeated with the above embodiments.

[0271] When the information processing device 10 that controls the autonomous driving of the vehicle 100 performs high-level arithmetic processing, heat generation will become a problem. Therefore, the eighth embodiment provides a vehicle 100 having a cooling function for the information processing device 10.

[0272] Figure 8 is an explanatory diagram showing the schematic configuration of the vehicle 100. As Figure 8 shown, on the vehicle 100, an information processing device 10, a cooling execution device 110, and a cooling unit 120 are mounted.

[0273] The information processing device 10 according to the eighth embodiment is a device that controls the autonomous driving of the vehicle 100, and as an example, has the same in the first embodiment Figure 2The configuration shown in the figure. The cooling execution device 110 obtains the detection result of the object by the information processing device 10, and based on this detection result, causes the cooling unit 120 to cool the information processing device 10. The cooling unit 120 uses at least one of a forced air cooling device, a water cooling device, a liquid nitrogen cooling device, etc. to cool the information processing device 10. Hereinafter, although the object to be cooled in the information processing device 10 is the central brain 15 that controls the autonomous driving of the vehicle 100 (specifically, the CPU 14 that constitutes the central brain 15), it is not limited thereto.

[0274] The information processing device 10 and the cooling execution device 110 are communicably connected via a network (not shown). This network can be any one of a vehicle network, the Internet, a LAN (Local Area Network), and a mobile communication network. The mobile communication network can conform to any one of the 5G (5th Generation) communication scheme, the LTE (Long Term Evolution) communication method, the 3G (3rd Generation) communication method, and the 6G (6th Generation) communication method and subsequent communication methods.

[0275] Figure 9 is a block diagram showing an example of the functional configuration of the cooling execution device 110. As Figure 9 shown, as a functional configuration, the cooling execution device 110 has an acquisition unit 112, an execution unit 114, and a prediction unit 116.

[0276] The acquisition unit 112 acquires the detection result of the object by the information processing device 10. For example, as this detection result, the acquisition unit 112 acquires the point information of the object output from the MoPU 12.

[0277] The execution unit 114 cools the central brain 15 based on the detection result of the object acquired by the acquisition unit 112. For example, when the execution unit 114 identifies that the object is moving based on the point information of the object output from the MoPU 12, it causes the cooling unit 120 to start cooling the central brain 15.

[0278] In addition, the execution unit 114 is not limited to cooling the central brain 15 based on the detection result of the object, and can also cool the central brain 15 based on the prediction result of the operating condition of the information processing device 10.

[0279] Here, the prediction unit 116 predicts the operating condition of the information processing device 10 (more specifically, the central brain 15) based on the detection result of the object obtained by the acquisition unit 112. For example, the prediction unit 116 acquires a learning model stored in a specified storage area. Then, the prediction unit 116 inputs the point information of the object output from the MoPU 12 obtained by the acquisition unit 112 into the learning model to predict the operating condition of the central brain 15. Here, the learning model outputs the computing power condition and the change amount of the central brain 15 as the operating condition. In addition, the prediction unit 116 can predict and output the operating condition and the temperature change of the information processing device 10 (more specifically, the central brain 15). For example, the prediction unit 116 predicts the temperature change of the central brain 15 based on the number of point information of the object output from the MoPU 12 obtained by the acquisition unit 112. In this case, the prediction unit 116 predicts that the greater the number of point information, the greater the temperature change, and the smaller the number of point information, the smaller the temperature change.

[0280] In the above case, the execution unit 114 causes the cooling unit 120 to start cooling the central brain 15 based on the prediction result of the operating condition of the central brain 15 by the prediction unit 116. For example, when the computing power condition and the change amount of the central brain 15 predicted as the operating condition exceed a predetermined threshold, the execution unit 114 causes the cooling unit 120 to start cooling. In addition, when the temperature obtained based on the temperature change of the central brain 15 predicted as the operating condition exceeds a predetermined threshold, the execution unit 114 causes the cooling unit 120 to start cooling.

[0281] In addition, the execution unit 114 can use the cooling device according to the prediction result of the temperature change of the central brain 15 by the prediction unit 116 to cool the central brain 15. For example, the higher the predicted temperature of the central brain 15, the more cooling devices the execution unit 114 can cause the cooling unit 120 to use to perform cooling. As a specific example, when it is predicted that the temperature of the central brain 15 exceeds the first threshold, the execution unit 114 causes the cooling unit 120 to use one cooling device to perform cooling. On the other hand, when it is predicted that the temperature of the central brain 15 exceeds the second threshold higher than the first threshold, the execution unit 114 causes the cooling unit 120 to use multiple cooling devices to perform cooling.

[0282] In addition, the higher the predicted temperature of the central brain 15, the more powerful the cooling device that the execution unit 114 can use to cool the central brain 15. For example, when it is predicted that the temperature of the central brain 15 exceeds a first threshold, the execution unit 114 causes the cooling unit 120 to use an air cooling device to perform cooling. In addition, when it is predicted that the temperature of the central brain 15 exceeds a second threshold that is higher than the first threshold, the execution unit 114 causes the cooling unit 120 to use a water cooling device to perform cooling. Furthermore, when it is predicted that the temperature of the central brain 15 exceeds a third threshold that is higher than the second threshold, the execution unit 114 causes the cooling unit 120 to use a liquid nitrogen cooling device to perform cooling.

[0283] In addition, the execution unit 114 can determine the cooling device for cooling based on the number of point information of the objects output from the MoPU 12 obtained by the acquisition unit 112. In this case, the more the number of point information, the more powerful the cooling device that the execution unit 114 can use to cool the central brain 15. For example, when the number of point information exceeds a first threshold, the execution unit 114 causes the cooling unit 120 to use an air cooling device to perform cooling. In addition, when the number of point information exceeds a second threshold that is higher than the first threshold, the execution unit 114 causes the cooling unit 120 to use a water cooling device to perform cooling. Furthermore, when the number of point information exceeds a third threshold that is higher than the second threshold, the execution unit 114 causes the cooling unit 120 to use a liquid nitrogen cooling device to perform cooling.

[0284] In addition, when a moving object present on the lane is detected, it may trigger the central brain 15 to operate. For example, in the case where a moving object present on the road is detected while the vehicle 100 is performing autonomous driving, the central brain 15 can perform arithmetic processing for controlling the vehicle 100 on the object. As described above, the heat generation during the advanced arithmetic processing of the central brain 15 that controls the autonomous driving of the vehicle 100 will become a problem. Therefore, the cooling execution device 110 according to the eighth embodiment predicts the heat dissipation of the central brain 15 based on the detection result of the object by the information processing device 10, and cools the central brain 15 before or at the same time as the start of heat dissipation. Thus, the situation where the central brain 15 becomes high temperature during the autonomous driving of the vehicle 100 is suppressed, and advanced arithmetic operations can be performed during the autonomous driving.

[0285] (Ninth Embodiment)

[0286] Next, for the ninth embodiment according to the present embodiment, it will be described while omitting or simplifying the parts that overlap with the above embodiments.

[0287] The MoPU 12 included in the information processing apparatus 10 according to the ninth embodiment derives the z coordinate value of an object as point information from an image of the object captured by the camera 30. Hereinafter, each aspect of the information processing apparatus 10 according to the ninth embodiment will be described in sequence.

[0288] The information processing apparatus 10 according to the first aspect has the same configuration as that shown in Figure 3 in the second embodiment.

[0289] In the first aspect described above, the MoPU 12 derives the z coordinate value of an object as point information from images of the object captured by a plurality of cameras 30 (more specifically, the camera 30L and the camera 30R). As described above, when one MoPU 12 is used, the x coordinate value and the y coordinate value of an object can be derived as point information. Here, when two MoPUs 12 are used, based on the principle of a stereo camera, the z coordinate value of an object can be derived as point information based on the images of the object captured by the two cameras 30. Therefore, in this first aspect, based on the principle of a stereo camera, the z coordinate value of an object is derived as point information from the images of the object captured by the camera 30L of the MoPU 12L and the camera 30R of the MoPU 12R, respectively.

[0290] The information processing apparatus 10 according to the second aspect has the same configuration as that shown in Figure 3 in the second embodiment.

[0291] In the second aspect described above, the MoPU 12 derives the x coordinate value, the y coordinate value, and the z coordinate value of an object as point information from an image of the object captured by the camera 30 and from a radar signal of a reflected wave from the object based on electromagnetic waves irradiated onto the object by the radar 32. As described above, the radar 32 can acquire three-dimensional point cloud data of an object based on the radar signal. That is, the radar 32 can detect the coordinate of the z axis in a three-dimensional rectangular coordinate system. In this case, the MoPU 12 combines the x coordinate value and the y coordinate value of the object captured by the camera 30 at the same timing as the timing when the radar 32 acquires the three-dimensional point cloud data of the object, with the z coordinate value of the object indicated by the three-dimensional point cloud data, and derives the coordinate values of the object on the three coordinate axes as point information.

[0292] The information processing apparatus 10 according to the third aspect has the configuration shown in Figure 10 in. Figure 10 is the fifth block diagram showing an example of the configuration of the information processing apparatus 10. In addition, Figure 10 only shows a part of the configuration of the information processing apparatus 10.

[0293] In the third aspect described above, the MoPU 12 derives the z coordinate value of the object as point information from the image of the object captured by the camera 30 and from the result of capturing the structured light irradiated onto the object by the irradiation device 130 .

[0294] like Figure 10 As shown, in the MoPU 12, an image representing an object captured by the camera 30 and distortion information indicating distortion of a structured light pattern are input to the core 17 at a frame rate of 100 frames / second or more. The structured light pattern is a result of the camera 140 capturing structured light irradiated onto the object by the irradiation device 130. Furthermore, the core 17 outputs point information to the central brain 15 based on the input object image and distortion information.

[0295] Here, as one method of recognizing the three-dimensional position or shape of an object, there is a structured light method, for example. The structured light method is a method of irradiating a structured light patterned in a dotted shape onto an object and obtaining depth information from the distortion of the pattern. The structured light method is described in reference 1.

[0296] (http: / / ex-press.jp / wp-content / uploads / 2018 / 10 / 018_teledyne_3rd.pdf).

[0297] Figure 10 The illumination device 130 shown irradiates the structured light onto the object. In addition, the camera 140 captures the structured light irradiated onto the object by the illumination device 130. Then, the camera 140 outputs distortion information based on the distortion of the captured structured light pattern to the core 17.

[0298] Here, the MoPU 12 synchronizes the timing of capturing an image by the camera 30 (camera 30 captures images) with the timing of capturing structured light by the camera 140. Specifically, the MoPU 12 outputs control signals to the camera 30 and the camera 140 to capture images at the same timing (capture objects at the same timing). As a result, the number of images captured per second by the camera 30 (the number of images per second obtained by capturing images by the camera 30) is synchronized with the number of images captured per second by the camera 140 (the number of images per second obtained by capturing images by the camera 140) (for example, 1920 frames / second). In this way, the number of images per second captured by the camera 30 (the number of images per second obtained by capturing images by the camera 30) and the number of images per second captured by the camera 140 (the number of images per second obtained by capturing images by the camera 140) are greater than the frame rate of the ultra-high resolution camera possessed by the IPU 11, that is, greater than the number of images per second captured by the ultra-high resolution camera.

[0299] Furthermore, the core 17 combines the x-coordinate value and the y-coordinate value of the object captured by the camera 30 at the same timing as the timing of capturing the structured light by the camera 140 with the distortion information based on the distortion of the structured light pattern, and derives the z-coordinate value of the object as point information.

[0300] The information processing device 10 according to the fourth aspect includes: Figure 11 The structure shown in . Figure 11 6 is a block diagram showing an example of the structure of the information processing device 10. Figure 11 Only a part of the configuration of the information processing device 10 is shown.

[0301] Figure 11 The block diagram shown is in Figure 2 A diagram of a Lidar sensor 18 is added to the structure of the block diagram shown. The Lidar sensor 18 is a sensor that acquires point cloud data including objects existing in a three-dimensional space and the road surface on which the vehicle 100 is traveling. The information processing device 10 is able to derive position information of the object in the depth direction, that is, the z coordinate value of the object, by using the point cloud data acquired by the Lidar sensor 18. In addition, it is assumed that the point cloud data acquired by the Lidar sensor 18 is acquired at an interval longer than the interval at which the x coordinate value and the y coordinate value of the object are output from the MoPU 12. In addition, as in the above-mentioned aspects of the ninth embodiment, the MoPU 12 is equipped with a camera 30.

[0302] In the fourth aspect, MoPU 12 utilizes the principle of a stereo camera to combine the x-coordinate value and the y-coordinate value of the object captured by the camera 30 at the same timing as the timing at which the lidar sensor 18 acquires the point cloud data of the object with the z-coordinate value of the object indicated by the point cloud data, and derives the coordinate values ​​of the object on the three coordinate axes as point information.

[0303] Here, in the fourth aspect, the MoPU 12 derives the z coordinate value of the object at time t+1 as point information based on the x coordinate value, y coordinate value, and z coordinate value of the object at time t, and the x coordinate value and y coordinate value of the object at the next time point after time t (for example, time t+1). Time t is an example of a "first time point", and time t+1 is an example of a "second time point". In the fourth aspect, the z coordinate value of the object at time t+1 is derived by using shape information, that is, geometry. This is described in detail below.

[0304] Figure 12 FIG. 1 is a diagram schematically showing time series coordinate detection of an object. Figure 12 In , J indicates the position of an object represented by a rectangle, and the position of the object moves from J1 to J2 in a time series. Figure 12 In the figure, the object coordinate value at the time t when the object is located at J1 is (x1, y1, z1), and the object coordinate value at the time t+1 when the object is located at J2 is (x2, y2, z2).

[0305] First, time t will be described.

[0306] The MoPU 12 derives the x-coordinate value and the y-coordinate value of the object from the image of the object captured by the camera 30. Then, the MoPU 12 integrates the z-coordinate value of the object indicated by the point cloud data obtained from the lidar sensor 18 with the x-coordinate value and the y-coordinate value to derive the three-dimensional coordinate value (x1, y1, z1) of the object at time t.

[0307] Next, time t+1 will be described.

[0308] The MoPU 12 derives the z coordinate value of the object at time t+1 based on the spatial geometric structure and the changes in the x coordinate value and y coordinate value of the object from time t to time t+1. The spatial geometric structure includes the image captured by the ultra-high resolution camera of the IPU 11 (the image obtained by capturing by the ultra-high resolution camera), the road shape obtained from the point cloud data of the laser radar sensor 18, and the shape of the vehicle 100.

[0309] The geometric structure indicating the shape of the road surface is generated in advance at time t. The MoPU 12 uses the geometric structure indicating the shape of the vehicle 100 together with the geometric structure indicating the shape of the road surface, thereby simulating the vehicle 100 traveling on the road surface and estimating the movement amount of each axis of the x-axis, y-axis, and z-axis.

[0310] Therefore, MoPU 12 derives the x-coordinate value and y-coordinate value of the object at time t+1 from the image of the object captured by camera 30. MoPU 12 simulates and calculates the z-axis movement of the object when it changes from the x-coordinate value and y-coordinate value (x1, y1) at time t to the x-coordinate value and y-coordinate value (x2, y2) at time t+1, thereby being able to derive the z-coordinate value of the object at time t+1. Moreover, MoPU 12 integrates the above x-coordinate value and y-coordinate value with the z-coordinate value to derive the three-dimensional coordinate value (x2, y2, z2) of the object at time t+1.

[0311] like Figure 12 As shown, the object moves in the depth direction while moving on the plane coordinates (i.e., the x-axis and the y-axis), so in order to control the automatic driving of the vehicle 100 with high precision, it is also necessary to detect the movement in the z-axis direction. Here, the MoPU 12 is sometimes unable to obtain the z-coordinate value of the object that can be derived from the point cloud data of the lidar sensor 18 as quickly as the x-coordinate value and the y-coordinate value of the object. Here, in the fourth aspect above, the MoPU 12 derives the z-coordinate value of the object at time t+1 based on the x-coordinate value, y-coordinate value, and z-coordinate value of the object at time t, and the x-coordinate value and y-coordinate value of the object at the next time point after time t (for example, time t+1). Therefore, according to the information processing device 10 involved in the fourth aspect, the MoPU 12 can realize two-dimensional motion detection and three-dimensional motion detection based on high-speed frame shooting with high performance and low data volume.

[0312] In addition, although in the above description, the case where the MoPU 12 derives the object z coordinate value as point information from the image of the object captured by the camera 30 is illustrated, the technology of the disclosed content is not limited to this aspect. For example, the central brain 15 can derive the object z coordinate value as point information instead of the MoPU 12. In this case, the central brain 15 derives the object z coordinate value as point information by performing the processing performed by the MoPU 12 in the above description on the image of the object captured by the camera 30. As an example, the central brain 15 derives the object z coordinate value as point information from the image of the object captured by multiple cameras 30 (specifically, the camera 30L and the camera 30R). In this case, the central brain 15 uses the principle of stereo cameras to derive the object z coordinate value as point information based on the images of the object captured by the camera 30L of the MoPU 12L and the camera 30R of the MoPU 12R.

[0313] (Tenth Embodiment)

[0314] Next, a tenth embodiment according to the present embodiment will be described while omitting or simplifying portions overlapping with the above-described embodiments.

[0315] Figure 13 7 is a seventh block diagram showing an example of the structure of the information processing device 10. Figure 13 Only a part of the configuration of the information processing device 10 is shown.

[0316] like Figure 13 As shown, in the MoPU 12, an image of an object captured by the event camera 30C (hereinafter, sometimes referred to as an "event image") is input to the core 17. Then, the core 17 outputs point information to the central brain 15 based on the input event image. Incidentally, the event camera is disclosed in, for example, the reference document (https: / / dendenblog.xyz / event-based-camera / ).

[0317] Figure 14 It is an explanatory diagram for explaining an image of an object captured by the event camera 30C (event image). Figure 14 (A) is a diagram showing an object to be photographed by the event camera 30C. Figure 14 (B) is a diagram showing an example of an event image. Figure 14(C) shows an example in which the centroid of the difference between the image captured at the current moment (the image obtained by capturing at the current moment) and the image captured at the previous moment (the image obtained by capturing at the previous moment) represented by the event image is calculated as point information. With respect to the event image, the difference between the image captured at the current moment (the image obtained by capturing at the current moment) and the image captured at the previous moment (the image obtained by capturing at the previous moment) is extracted as a point. Therefore, when the event camera 30C is used, for example, Figure 14 As shown in (B), Figure 14 Points at various locations in the human figure region shown in (A) that are moving are extracted.

[0318] In contrast, Figure 14 As shown in (C), after extracting the person as the object, the core 17 extracts the coordinates of the feature point representing the person area (for example, only one point). As a result, the amount of data transmitted to the central brain 15 and the memory 16 can be reduced. With regard to the event image, the person as the object can be extracted at any frame rate. Therefore, in the case of the event camera 30C, it is also possible to extract at a frame rate not lower than the maximum frame rate of the camera 30 mounted on the MoPU 12 in the above-mentioned embodiment (for example, 1920 frames / second), thereby being able to capture the point information of the object with high precision.

[0319] In addition, similar to the above-mentioned embodiment, the MoPU 12 of the information processing device 10 according to the tenth embodiment may include a visible light camera 30A in addition to the event camera 30C. In this case, in the MoPU 12, the visible light image of the object captured by the visible light camera 30A and the event image are respectively input to the core 17. And the core 17 outputs point information to the central brain 15 based on at least one of the input visible light image and event image.

[0320] For example, when the core 17 can identify an object from a visible light image of the object captured by the visible light camera 30A, the core 17 outputs point information based on the visible light image. On the other hand, when the core 17 cannot capture the object from the visible light image due to a specified factor, the core 17 outputs point information based on the event image. The specified factor includes at least one of the following cases: a case where the moving speed of the object is equal to or greater than a specified value and a case where the change in the amount of ambient light per unit time is equal to or greater than a specified value. For example, when the movement of the object cannot be captured from the visible light image due to high speed, the core 17 identifies the object based on the event image and outputs the x-coordinate value and y-coordinate value of the object as point information. In addition, when the object cannot be captured from the visible light image due to a sharp change in the amount of ambient light such as backlight, the core 17 identifies the object based on the event image and outputs the x-coordinate value and y-coordinate value of the object as point information. Through this structure, the information processing device 10 can distinguish the camera 30 used to capture the object according to the specified factor.

[0321] (Eleventh Embodiment)

[0322] Next, the eleventh embodiment will be described while omitting or simplifying the parts that overlap with the above-mentioned embodiments. In addition, in this eleventh embodiment, as in the above-mentioned embodiments, the description is based on the premise that the MoPU performs processing at a higher speed than the IPU.

[0323] Figure 15 2 is a schematic top view showing an example of a plurality of vehicles 204 traveling on a road 202 in a state where a platoon 200 is formed. Figure 15 In the illustrated example, three vehicles 204 are illustrated as an example of a plurality of vehicles 204, wherein the three vehicles 204 are a leading vehicle 204A located at the head of the queue 200, a middle vehicle 204B located in the middle of the queue 200, and a last vehicle 204C located at the end of the queue 200. The leading vehicle 204A, the middle vehicle 204B, and the last vehicle 204C constitute the queue 200 and travel on the road 202 by automatic driving. The middle vehicle 204B follows the leading vehicle 204A, and the last vehicle 204C follows the middle vehicle 204B. For example, the leading vehicle 204A, the middle vehicle 204B, and the last vehicle 204C travel while maintaining a certain distance from each other.

[0324] Here, the queue 200 is an example of a "queue" in the present disclosure. In addition, the plurality of vehicles 204 is an example of a "plurality of moving bodies moving in a queue" in the present disclosure. In addition, the leading vehicle 204A is an example of a "leading moving body" in the present disclosure. In addition, the middle vehicle 204B is an example of a "specific moving body" and an "intermediate moving body" in the present disclosure. In addition, the last vehicle 204C is an example of a "last moving body" in the present disclosure.

[0325] In the platoon 200, an information processing device 206 is used. The automated driving of the platoon 200 (i.e., the automated driving of the leading vehicle 204A, the middle vehicle 204B, and the last vehicle 204C) is realized by using the information processing device 206. The information processing device 206 is an example of an “information processing device” and a “computer” in the present disclosure.

[0326] The information processing device 206 includes a first information processing device 206A, a second information processing device 206B, and a third information processing device 206C. The first information processing device 206A is used for the leading vehicle 204A, the second information processing device 206B is used for the middle vehicle 204B, and the third information processing device 206C is used for the last vehicle 204C. Figure 15 In the example shown, the first information processing device 206A is mounted on the leading vehicle 204A, the second information processing device 206B is mounted on the middle vehicle 204B, and the third information processing device 206C is mounted on the last vehicle 204C.

[0327] The leading vehicle 204A is equipped with a first sensor 208 and a second sensor 210. The first sensor 208 and the second sensor 210 are connected to the first information processing device 206A. The first sensor 208 is a sensor that obtains information related to the situation in front of the leading vehicle 204A. The second sensor 210 is a sensor that obtains information related to the situation behind the leading vehicle 204A. As an example of the situation in front of the leading vehicle 204A, a first front object can be listed. The first front object refers to at least one object that exists in front of the leading vehicle 204A. As an example of the situation behind the leading vehicle 204A, a first rear object can be listed. The first rear object refers to at least one object that exists behind the leading vehicle 204A (in other words, from the perspective of the leading vehicle 204A, it refers to at least one object that exists on the side of the middle vehicle 204B).

[0328] The intermediate vehicle 204B is equipped with a third sensor 212, a fourth sensor 214, a fifth sensor 216, and a sixth sensor 218. The third sensor 212, the fourth sensor 214, the fifth sensor 216, and the sixth sensor 218 are connected to the second information processing device 206B.

[0329] The third aspect sensor 212 is a sensor for acquiring information related to the situation in front of the middle vehicle 204B. The fourth aspect sensor 214 is a sensor for acquiring information related to the situation behind the middle vehicle 204B. The fifth aspect sensor 216 is a sensor for acquiring information related to the situation to the left of the middle vehicle 204B. The sixth aspect sensor 218 is a sensor for acquiring information related to the situation to the right of the middle vehicle 204B. The left side of the middle vehicle 204B and the right side of the middle vehicle 204B are examples of the "side" involved in the technology of the present disclosure.

[0330] As an example of the situation in front of the middle vehicle 204B, the second front object can be listed. The second front object refers to at least one object existing in front of the middle vehicle 204B (in other words, it refers to at least one object existing on the side of the leading vehicle 204A from the perspective of the middle vehicle 204B). As an example of the situation behind the middle vehicle 204B, the second rear object can be listed. The second rear object refers to at least one object existing behind the middle vehicle 204B (in other words, it refers to at least one object existing on the side of the last vehicle 204C from the perspective of the middle vehicle 204B).

[0331] As an example of the situation on the left side of the middle vehicle 204B, a left side object can be listed. The left side object refers to at least one object existing on the left side of the middle vehicle 204B. As an example of the situation on the right side of the middle vehicle 204B, a right side object can be listed. The right side object refers to at least one object existing on the right side of the middle vehicle 204B.

[0332] The last vehicle 204C is equipped with a seventh sensor 220 and an eighth sensor 222. The seventh sensor 220 and the eighth sensor 222 are connected to the third information processing device 206C. The seventh sensor 220 is a sensor that obtains information related to the situation in front of the last vehicle 204C. The eighth sensor 222 is a sensor that obtains information related to the situation behind the last vehicle 204C. As an example of the situation in front of the last vehicle 204C, a third front object can be listed. The third front object refers to at least one object that exists in front of the last vehicle 204C (in other words, from the perspective of the last vehicle 204C, it refers to at least one object that exists on the side of the middle vehicle 204B). As an example of the situation behind the last vehicle 204C, a third rear object can be listed. The third rear object refers to at least one object that exists behind the last vehicle 204C.

[0333] Furthermore, in the present eleventh embodiment, the front of the leading vehicle 204A is, in other words, the front of the platoon 200. Furthermore, in the present eleventh embodiment, the left side of the middle vehicle 204B is, in other words, the left side of the platoon 200. Furthermore, in the present eleventh embodiment, the right side of the middle vehicle 204B is, in other words, the right side of the platoon 200. Furthermore, in the present eleventh embodiment, the rear side of the last vehicle 204C is, in other words, the rear side of the platoon 200.

[0334] Figure 16 This is a conceptual diagram showing an example of the structure of the first information processing device 206A, the second information processing device 206B, and the third information processing device 206C.

[0335] The first information processing device 206A includes a lead vehicle processor 224, a lead vehicle memory 226, and a lead vehicle communication I / F 228. Here, I / F is an abbreviation of "Interface".

[0336] The lead vehicle memory 226 is a computer-readable non-transitory storage medium (for example, a non-volatile memory such as a flash memory). The lead vehicle memory 226 stores a lead vehicle program 230. The lead vehicle program 230 is an example of an "information processing program" of the present disclosure.

[0337] The lead vehicle processor 224 performs the lead vehicle control process. The lead vehicle control process is implemented by reading out and executing the lead vehicle program 230 from the lead vehicle memory 226 by the lead vehicle processor 224. As will be described in detail later, the lead vehicle control process includes the lead vehicle IPU process, the lead vehicle MoPU process, and the first central brain process.

[0338] The lead vehicle communication I / F 228 is an interface for communication including a communication processor and an antenna, etc., and is included in Figure 1 The leading vehicle communication I / F 228 communicates with different vehicles 204 (see Figure 1 ) are managed. As an example of a communication standard applicable to the lead vehicle communication I / F 228, wireless communication standards such as Wi-Fi (registered trademark) and 5G (5th Generation Mobile Communication System) can be cited.

[0339] The second information processing device 206 b includes an intermediate vehicle processor 232 , an intermediate vehicle memory 234 , and an intermediate vehicle communication I / F 236 .

[0340] The intermediate vehicle memory 234 is a computer-readable non-transitory storage medium (for example, a non-volatile memory such as a flash memory). The intermediate vehicle memory 234 stores an intermediate vehicle program 238. The intermediate vehicle program 238 is an example of an "information processing program" of the present disclosure.

[0341] The intermediate vehicle processor 232 performs intermediate vehicle control processing. The intermediate vehicle control processing is implemented by reading and executing the intermediate vehicle program 238 from the intermediate vehicle memory 234 by the intermediate vehicle processor 232. As will be described in detail later, the intermediate vehicle control processing includes intermediate vehicle IPU processing, intermediate vehicle MoPU processing, and second central brain processing.

[0342] The intermediate vehicle communication I / F 236 has the same structure as the lead vehicle communication I / F 228, and is used for different vehicles 204 (see Figure 1 ) to manage communications between them.

[0343] The third information processing device 206C includes a last vehicle processor 240 , a last vehicle memory 242 , and a last vehicle communication I / F 244 .

[0344] The last vehicle memory 242 is a non-transitory storage medium readable by a computer (for example, a non-volatile memory such as a flash memory). The last vehicle memory 242 stores a last vehicle program 246. The last vehicle program 246 is an example of an "information processing program" of the present disclosure.

[0345] The last vehicle processor 240 performs the last vehicle control process. The last vehicle control process is realized by reading out and executing the last vehicle program 246 from the last vehicle memory 242 by the last vehicle processor 240. As will be described in detail later, the last vehicle control process includes the last vehicle IPU process, the last vehicle MoPU process and the third central brain process.

[0346] The trailing vehicle communication I / F 244 has the same structure as the leading vehicle communication I / F 228, and is used for different vehicles 204 (see Figure 1 ) to manage communications between them.

[0347] In addition, the above-mentioned leading vehicle processor 224, the middle vehicle processor 232, and the last vehicle processor 240 are examples of the "processor" of the present disclosure.

[0348] Figure 17 This is a conceptual diagram showing an example of the structure of each of the first information processing device 206A, the first aspect sensor 208, and the second aspect sensor 210 mounted on the lead vehicle 204A.

[0349] In the first information processing device 206A, the lead vehicle processor 224 includes a first central brain 224A, a first IPU 224B, a second IPU 224C, a first MoPU 224D, and a second MoPU 224E. The first central brain 224A is a processing device corresponding to the central brain 15 described in the above embodiment. Each of the first IPU 224B and the second IPU 224C is a processing device corresponding to the IPU 11 described in the above embodiment. Each of the first MoPU 224D and the second MoPU 224E is a processing device corresponding to the MoPU 12 described in the above embodiment. Figure 17 In the example shown, the first IPU 224B is an example of a "front identification processor" of the present disclosure.

[0350] The first aspect sensor 208 includes a first low FR camera 208A, a first high FR camera 208B, and a first radar 208C. The first low FR camera 208A is an example of a “front camera” in the present disclosure.

[0351] Here, FR is an abbreviation for "frame rate". The first low FR camera 208A is for example Figure 2 The first high FR camera 208B is for Figure 2The first low FR camera 208A captures the front of the leading vehicle 204A at the frame rate of the first low FR camera 208A, i.e., the first low frame rate. The first high FR camera 208B captures the front of the leading vehicle 204A at the frame rate of the first high FR camera 208B, i.e., the first high frame rate. Between the first low frame rate and the first high frame rate, the relationship of "first low frame rate < first high frame rate" holds. The first low frame rate is, for example, a frame rate of 10 frames / second or more, and the first high frame rate is, for example, a frame rate of 100 frames / second or more.

[0352] The photographing direction and photographing range of the first low FR camera 208A are consistent with the photographing direction and photographing range of the first high FR camera 208B. The first low FR camera 208A and the first high FR camera 208B photograph the front of the leading vehicle 204A at a viewing angle θ1.

[0353] The first radar 208C is a radar corresponding to the radar 32 described in the second embodiment, and irradiates electromagnetic waves in front of the leading vehicle 204A and receives a first front object reflected wave obtained by reflecting the irradiated electromagnetic waves at the first front object.

[0354] The second aspect sensor 210 includes a second low FR camera 210A, a second high FR camera 210B, and a second radar 210C. The second low FR camera 210A is a camera having the same specifications as the first low FR camera 208A, and the second high FR camera 210B is a camera having the same specifications as the first high FR camera 208B. The second low FR camera 210A photographs the rear of the leading vehicle 204A (i.e., the side of the middle vehicle 204B from the perspective of the leading vehicle 204A) at a second low frame rate. The second high FR camera 210B photographs the rear of the leading vehicle 204A at a second high frame rate. The second low frame rate is the same as the first low frame rate, and the second high frame rate is the same as the first high frame rate.

[0355] The second low FR camera 210A has the same photographing direction and photographing range as the second high FR camera 210B. The second low FR camera 210A and the second high FR camera 210B photograph the rear of the leading vehicle 204A at a viewing angle θ2. An example of the viewing angle θ2 is the same as the viewing angle θ1.

[0356] The second radar 210C is a radar having the same specifications as the first radar 208C, irradiates electromagnetic waves toward the rear of the leading vehicle 204A and receives a first rear object reflected wave obtained by reflecting the irradiated electromagnetic waves at the first rear object.

[0357] Figure 18 This is a conceptual diagram showing an example of processing contents of the first aspect sensor 208 , the second aspect sensor 210 , the first IPU 224B, the second IPU 224C, the first MoPU 224D, and the second MoPU 224E.

[0358] The first low FR camera 208A captures the front of the leading vehicle 204A at a first low frame rate to generate an image showing the situation in front of the leading vehicle 204A, namely, a first low FR camera image 208A1. The first low FR camera image 208A1 is an example of a “front image” in the present disclosure.

[0359] The first IPU 224B acquires the first low FR camera image 208A1 from the first low FR camera 208A at a time interval specified by the first low frame rate. Moreover, each time the first IPU 224B acquires the first low FR camera image 208A1, it recognizes the situation in front of the leading vehicle 204A based on the first low FR camera image 208A1, and generates the first label information 248 indicating the recognized result. The first label information 248 is information of the same concept as the label information described in the first embodiment, etc. As an example of the first label information 248, information that is labeled so as to be able to identify the type of the first front object can be listed.

[0360] The second low FR camera 210A captures the rear of the leading vehicle 204A at a second low frame rate to generate an image showing the rear of the leading vehicle 204A, ie, a second low FR camera image 210A1 .

[0361] The second IPU 224C acquires the second low FR camera image 210A1 from the second low FR camera 210A at a time interval specified by the second low frame rate. Moreover, the second IPU 224C recognizes the situation behind the lead vehicle 204A based on the second low FR camera image 210A1 each time it acquires the second low FR camera image 210A1, and generates second label information 250 indicating the recognized result. The second label information 250 is information equivalent to the label information described in the first embodiment and the like. As an example of the second label information 250, information that is labeled so as to be able to identify the type of the first rear object can be listed.

[0362] The first high FR camera 208B captures the front of the leading vehicle 204A at a first high frame rate to generate an image showing the front situation, ie, a first high FR camera image 208B1 .

[0363] The first radar 208C receives the first front object reflected wave at time intervals specified by the first high frame rate. Moreover, each time the first radar 208C receives the first front object reflected wave, it generates a first radar signal 208C1 capable of determining the location of the first front object based on the received first front object reflected wave.

[0364] The first MoPU 224D acquires the first high FR camera image 208B1 from the first high FR camera 208B at a time interval specified by the first high frame rate, and acquires the first radar signal 208C1 from the first radar 208C. Moreover, the first MoPU 224D recognizes the first front object based on the first high FR camera image 208B1 and the first radar signal 208C1, and generates first point information 252 indicating the recognized result. For example, here, the first front object is recognized as a point. The first point information 252 is information of the same concept as the point information described in the above-mentioned first embodiment, etc. That is, the first point information 252 is point information (for example, three-dimensional coordinates) captured with the first front object as a point.

[0365] The second high FR camera 210B captures the rear of the leading vehicle 204A at the second highest frame rate to generate an image showing the rear of the leading vehicle 204A, ie, a second high FR camera image 210B1 .

[0366] The second radar 210C receives the first rear object reflected wave at a time interval specified by the second high frame rate. Moreover, each time the second radar 210C receives the first rear object reflected wave, it generates a second radar signal 210C1 capable of determining the location of the first rear object based on the received first rear object reflected wave.

[0367] The second MoPU 224E acquires a second high FR camera image 210B1 from the second high FR camera 210B at a time interval specified by the second high frame rate, and acquires a second radar signal 210C1 from the second radar 210C. Moreover, the second MoPU 224E recognizes the first rear object based on the second high FR camera image 210B1 and the second radar signal 210C1, and generates second point information 254 indicating the recognized result. For example, here, the first rear object is recognized as a point. The second point information 254 is information of the same concept as the point information described in the above-mentioned first embodiment, etc. That is, the second point information 254 is point information (for example, three-dimensional coordinates) captured with the first rear object as a point.

[0368] exist Figure 18In the example shown, the first high FR camera image 208B1 is an example of the "first image" of the present disclosure. In addition, the first high frame rate is an example of the "fourth frame rate" of the present disclosure.

[0369] Figure 19 This is a conceptual diagram showing an example of the structure of each of the second information processing device 206B, the third sensor 212, the fourth sensor 214, the fifth sensor 216, and the sixth sensor 218 mounted on the intermediate vehicle 204B.

[0370] In the second information processing device 206B, the intermediate vehicle processor 232 includes a second central brain 232A, a third IPU 232B, a fourth IPU 232C, a fifth IPU 232D, a sixth IPU 232E, a third MoPU 232F, a fourth MoPU 232G, a fifth MoPU 232H and a sixth MoPU 232I.

[0371] The second central brain 232A is a processing device corresponding to the central brain 15 described in the above embodiment. Each of the third IPU 232B, the fourth IPU 232C, the fifth IPU 232D, and the sixth IPU 232E is a processing device corresponding to the IPU 11 described in the above embodiment. Each of the third MoPU 232F, the fourth MoPU 232G, the fifth MoPU 232H, and the sixth MoPU 232I is a processing device corresponding to the MoPU 12 described in the above embodiment. Figure 19 In the example shown, the fifth MoPU 232H and the sixth MoPU 232I are examples of the “side recognition processor” of the present disclosure.

[0372] The third side sensor 212 includes a third low FR camera 212A, a third high FR camera 212B, and a third radar 212C. The third low FR camera 212A is an example of a "leading side camera" of the present disclosure.

[0373] The third low FR camera 212A is a camera having the same specifications as the first low FR camera 208A, and the third high FR camera 210B is a camera having the same specifications as the first high FR camera 208B.

[0374] The third low FR camera 212A captures the side of the leading vehicle 204A as seen from the middle vehicle 204B (i.e., the front of the middle vehicle 204B) at a third low frame rate. The third high FR camera 212B captures the side of the leading vehicle 204A as seen from the middle vehicle 204B at a third high frame rate. The third low frame rate is the same as the first low frame rate, and the third high frame rate is the same as the first high frame rate.

[0375] The photographing direction and photographing range of the third low FR camera 212A are consistent with the photographing direction and photographing range of the third high FR camera 212B. The third low FR camera 212A and the third high FR camera 212B photograph the leading vehicle 204A side at a viewing angle θ1.

[0376] The third radar 212C is a radar having the same specifications as the first radar 208C, and irradiates electromagnetic waves toward the leading vehicle 204A side and receives second front object reflected waves obtained by reflecting the irradiated electromagnetic waves at the second front object.

[0377] The fourth side sensor 214 includes a fourth low FR camera 214A, a fourth high FR camera 214B, and a fourth radar 214C. The fourth low FR camera 214A is an example of a "rear side camera" of the present disclosure.

[0378] The fourth low FR camera 214A is a camera having the same specifications as the first low FR camera 208A, and the fourth high FR camera 214B is a camera having the same specifications as the first high FR camera 208B.

[0379] The fourth low FR camera 214A captures the rear vehicle 204C side (i.e., the rear of the middle vehicle 204B) as viewed from the middle vehicle 204B at a fourth low frame rate. The fourth high FR camera 214B captures the rear vehicle 204C side as viewed from the middle vehicle 204B at a fourth high frame rate. The fourth low frame rate is the same as the first low frame rate, and the fourth high frame rate is the same as the first high frame rate.

[0380] The fourth low FR camera 214A has the same photographing direction and photographing range as the fourth high FR camera 214B. The fourth low FR camera 214A and the fourth high FR camera 214B photograph the rear vehicle 204C side at a viewing angle θ2.

[0381] The fourth radar 214C is a radar having the same specifications as the first radar 208C, irradiates electromagnetic waves toward the rear vehicle 204C side and receives second rear object reflected waves obtained by reflecting the irradiated electromagnetic waves at the second rear object.

[0382] The fifth side sensor 216 includes a fifth low FR camera 216A, a fifth high FR camera 216B, and a fifth radar 216C. The fifth high FR camera 216B is an example of a "side camera" of the present disclosure.

[0383] The fifth low FR camera 216A is a camera having the same specifications as the first low FR camera 208A, and the fifth high FR camera 216B is a camera having the same specifications as the first high FR camera 208B.

[0384] The fifth low FR camera 216A captures the left side of the middle vehicle 204B (in other words, the left side of the queue 200) at a fifth low frame rate. The fifth high FR camera 216B captures the left side of the middle vehicle 204B at a fifth high frame rate. The fifth low frame rate is the same as the first low frame rate, and the fifth high frame rate is the same as the first high frame rate.

[0385] The photographing direction and photographing range of the fifth low FR camera 216A are consistent with the photographing direction and photographing range of the fifth high FR camera 216B. The fifth low FR camera 216A and the fifth high FR camera 216B photograph the left side of the middle vehicle 204B at a viewing angle θ3. The viewing angle θ3 is a wider viewing angle than the viewing angles θ1 and θ2. In the viewing angle θ3, the entire left side of the queue 200 is included as a subject.

[0386] The fifth radar 216C is a radar having the same specifications as the first radar 208C, irradiating electromagnetic waves toward the left side of the middle vehicle 204B and receiving reflected waves from the left side object, which are reflected waves obtained by the irradiated electromagnetic waves being reflected from the left side object.

[0387] The sixth side sensor 218 includes a sixth low FR camera 218A, a sixth high FR camera 218B, and a sixth radar 218C. The sixth high FR camera 218B is an example of a "side camera" of the present disclosure.

[0388] The sixth low FR camera 218A is a camera having the same specifications as the first low FR camera 208A, and the sixth high FR camera 218B is a camera having the same specifications as the first high FR camera 208B.

[0389] The sixth low FR camera 218A captures the right side of the middle vehicle 204B (in other words, the right side of the queue 200) at a sixth low frame rate. The sixth high FR camera 218B captures the right side of the middle vehicle 204B at a sixth high frame rate. The sixth low frame rate is the same as the first low frame rate, and the sixth high frame rate is the same as the first high frame rate.

[0390] The photographing direction and photographing range of the sixth low FR camera 218A are consistent with the photographing direction and photographing range of the sixth high FR camera 218B. The sixth low FR camera 218A and the sixth high FR camera 218B photograph the right side of the middle vehicle 204B at a viewing angle θ4. The viewing angle θ4 is a wider viewing angle than the viewing angles θ1 and θ2. In the viewing angle θ4, the entire right side of the queue 200 is included as a subject.

[0391] The sixth radar 218C is a radar having the same specifications as the first radar 208C, irradiating electromagnetic waves toward the right side of the middle vehicle 204B and receiving reflected waves from an object on the right side, the reflected waves from the object on the right side being the reflected waves from the irradiated electromagnetic waves.

[0392] Figure 20 1 is a conceptual diagram showing an example of processing contents of the third low FR camera 212A, the fourth low FR camera 214A, the fifth low FR camera 216A, and the sixth low FR camera 218A, the third IPU 232B, the fourth IPU 232C, the fifth IPU 232D, and the sixth IPU 232E.

[0393] The third low FR camera 212A generates a third low FR camera image 212A1 showing aspects of the leading vehicle 204A side by capturing an image of the leading vehicle 204A side (hereinafter referred to as “the leading vehicle 204A side”) viewed from the middle vehicle 204B at a third low frame rate.

[0394] The third IPU 232B acquires the third low FR camera image 212A1 from the third low FR camera 212A at a time interval specified by the third low frame rate. Moreover, the third IPU 224B recognizes the situation on the leading vehicle 204A side based on the third low FR camera image 212A1 each time it acquires the third low FR camera image 212A1, and generates third label information 256 indicating the recognized result. The third label information 256 is information equivalent to the label information described in the first embodiment and the like. As an example of the third label information 256, information that is labeled so as to be able to identify the type of the second front object can be listed.

[0395] The fourth low FR camera 214A generates an image showing the situation on the rear vehicle 204C side, i.e., the fourth low FR camera image 214A1, by photographing the rear vehicle 204C side (hereinafter referred to as "rear vehicle 204C side") as viewed from the middle vehicle 204B at a fourth low frame rate.

[0396] The fourth IPU 232C obtains the fourth low FR camera image 214A1 from the fourth low FR camera 214A at time intervals specified by the fourth low frame rate. Moreover, each time the fourth IPU 232C obtains the fourth low FR camera image 214A1, it identifies the situation on the trailing vehicle 204C side based on the fourth low FR camera image 214A1 and generates fourth tag information 258 indicating the recognized result. The fourth tag information 258 is information equivalent to the tag information described in the first embodiment and the like. As an example of the fourth tag information 258, information that is tagged so as to be able to determine the type of the second rear object can be cited.

[0397] The fifth low FR camera 216A generates an image showing the situation on the left side of the middle vehicle 204B, that is, the fifth low FR camera image 216A1, by photographing the left side of the middle vehicle 204B at the fifth low frame rate.

[0398] The fifth IPU 232D obtains the fifth low FR camera image 216A1 from the fifth low FR camera 216A at time intervals specified by the fifth low frame rate. Moreover, each time the fifth IPU 232D obtains the fifth low FR camera image 216A1, it identifies the situation on the left side of the middle vehicle 204B based on the fifth low FR camera image 216A1 and generates fifth tag information 260 indicating the recognized result. The fifth tag information 260 is information equivalent to the tag information described in the first embodiment and the like. As an example of the fifth tag information 260, information that is tagged so as to be able to determine the type of the object on the left side can be cited.

[0399] The sixth low FR camera 218A generates an image showing the situation on the right side of the middle vehicle 204B, that is, the sixth low FR camera image 218A1, by photographing the right side of the middle vehicle 204B at the sixth low frame rate.

[0400] The sixth IPU 232E obtains the sixth low FR camera image 218A1 from the sixth low FR camera 218A at time intervals specified by the sixth low frame rate. Moreover, each time the sixth IPU 224E obtains the sixth low FR camera image 218A1, it identifies the situation on the right side of the middle vehicle 204B based on the sixth low FR camera image 218A1 and generates sixth tag information 262 indicating the recognized result. The sixth tag information 262 is information equivalent to the tag information described in the first embodiment and the like. As an example of the sixth tag information 262, information that is tagged so as to be able to determine the type of the object on the right side can be cited.

[0401] At Figure 20In the example shown, the fifth lowest FR camera image 216A1 is an example of the "leading moving object side image" of the present disclosure. In addition, the sixth lowest FR camera image 218A1 is an example of the "trailing moving object side image" of the present disclosure. In addition, the fifth lowest frame rate is an example of the "second frame rate" of the present disclosure. In addition, the sixth lowest frame rate is an example of the "third frame rate" of the present disclosure.

[0402] Figure 21 It is a conceptual diagram showing an example of the processing contents of the third highest FR camera 212B, the third radar 212C, the fourth highest FR camera 214A, the fourth radar 214C, the fifth highest FR camera 216A, the fifth radar 216C, the sixth highest FR camera 218B, the sixth radar 218C, the third MoPU 232F, the fourth MoPU 232G, the fifth MoPU 232H, and the sixth MoPU 232I.

[0403] The third highest FR camera 212B generates an image showing the situation on the side of the leading vehicle 204A, that is, the third highest FR camera image 212B1, by photographing the side of the leading vehicle 204A at the third highest frame rate.

[0404] The third radar 212C receives the reflected wave of the second front object at time intervals specified by the third highest frame rate. Moreover, each time the third radar 212C receives the reflected wave of the second front object, it generates a third radar signal 212C1 capable of determining the existence position of the second front object based on the received reflected wave of the second front object.

[0405] The third MoPU 232F acquires the third highest FR camera image 212B1 from the third highest FR camera 212B and the third radar signal 212C1 from the third radar 212C at time intervals specified by the third highest frame rate. Moreover, the third MoPU 232F identifies the second front object based on the third highest FR camera image 212B1 and the third radar signal 212C1, and generates third point information 264 indicating the identified result. For example, here, the second front object is identified as a point. The third point information 264 is information of the same concept as the point information described in the above first embodiment and the like. That is, the third point information 264 is point information (for example, three-dimensional coordinates) that captures the second front object as a point.

[0406] The fourth highest FR camera 214B generates an image showing the situation on the side of the trailing vehicle 204C, that is, the fourth highest FR camera image 214B1, by photographing the side of the trailing vehicle 204C at the fourth highest frame rate.

[0407] The fourth radar 214C receives the reflected waves of the second rear object at time intervals specified by the fourth highest frame rate. Moreover, each time the fourth radar 214C receives the reflected waves of the second rear object, it generates a fourth radar signal 214C1 capable of determining the existence position of the second rear object based on the received reflected waves of the second rear object.

[0408] The fourth MoPU 232G obtains the fourth high FR camera image 214B1 from the fourth high FR camera 214B and obtains the fourth radar signal 214C1 from the fourth radar 214C at time intervals specified by the fourth highest frame rate. Moreover, the fourth MoPU 232G identifies the second rear object based on the fourth high FR camera image 214B1 and the fourth radar signal 214C1, and generates fourth point information 266 indicating the identified result. For example, here, the first rear object is identified as a point. The fourth point information 266 is information of the same concept as the point information described in the first embodiment and the like. That is, the fourth point information 266 is point information (for example, three-dimensional coordinates) obtained by capturing the second rear object as a point.

[0409] The fifth high FR camera 216B generates an image showing the left side of the intermediate vehicle 204B, that is, the fifth high FR camera image 216B1, by photographing the left side of the intermediate vehicle 204B at the fifth highest frame rate.

[0410] The fifth radar 216C receives the reflected waves of the left side object at time intervals specified by the fifth highest frame rate. Moreover, each time the fifth radar 216C receives the reflected waves of the left side object, it generates a fifth radar signal 216C1 capable of determining the existence position of the left side object based on the received reflected waves of the left side object.

[0411] The fifth MoPU 232H obtains the fifth high FR camera image 216B1 from the fifth high FR camera 216B and obtains the fifth radar signal 216C1 from the fifth radar 216C at time intervals specified by the fifth highest frame rate. Moreover, the fifth MoPU 232H identifies the left side object based on the fifth high FR camera image 216B1 and the fifth radar signal 216C1, and generates fifth point information 268 indicating the identified result. For example, here, the left side object is identified as a point. The fifth point information 268 is information of the same concept as the point information described in the first embodiment and the like. That is, the fifth point information 268 is point information (for example, three-dimensional coordinates) obtained by capturing the left side object as a point.

[0412] The sixth highest frame rate (FR) camera 218B captures the right side of the middle vehicle 204B at the sixth highest frame rate, generating an image showing the situation on the right side of the middle vehicle 204B, i.e., the sixth highest FR camera image 218B1.

[0413] The sixth radar 218C receives reflected waves from objects on the right side at time intervals specified by the sixth highest frame rate. Moreover, each time the sixth radar 218C receives reflected waves from objects on the right side, it generates a sixth radar signal 218C1 capable of determining the existence position of the object on the right side based on the received reflected waves.

[0414] The sixth MoPU 232I obtains the sixth highest FR camera image 218B1 from the sixth highest FR camera 218B and the sixth radar signal 218C1 from the sixth radar 218C at time intervals specified by the sixth highest frame rate. Moreover, the sixth MoPU 232I identifies the object on the right side based on the sixth highest FR camera image 218B1 and the sixth radar signal 218C1, and generates sixth point information 270 indicating the identified result. For example, here, the object on the right side is identified as a point. The sixth point information 270 is information of the same concept as the point information described in the above first embodiment and the like. That is, the sixth point information 270 is point information (e.g., three-dimensional coordinates) that captures the object on the right side as a point.

[0415] In Figure 21 the example shown, the fifth point information 268 and the sixth point information 270 are examples of the "lateral point information" of the present disclosure. In addition, the fifth highest frame rate and the sixth highest frame rate are examples of the "first frame rate" of the present disclosure.

[0416] Figure 22 is a conceptual diagram showing an example of the configurations of the third information processing device 206C, the seventh aspect sensor 220, and the eighth aspect sensor 222 mounted on the trailing vehicle 204C.

[0417] In the third information processing device 206C, the trailing vehicle processor 240 includes a third central brain 240A, a seventh IPU 240B, an eighth IPU 240C, a seventh MoPU 240D, and an eighth MoPU 240E. The third central brain 240A is a processing device corresponding to the central brain 15 described in the above first embodiment and the like. Each of the seventh IPU 240B and the eighth IPU 240C is a processing device corresponding to the IPU 11 described in the above first embodiment and the like. Each of the seventh MoPU 240D and the eighth MoPU 240E is a processing device corresponding to the MoPU 12 described in the above first embodiment and the like. In Figure 22In the example shown, the eighth IPU 240C is an example of the "rear recognition processor" of the present disclosure.

[0418] The seventh aspect sensor 220 includes a seventh low FR camera 220A, a seventh high FR camera 220B, and a seventh radar 220C.

[0419] The seventh low FR camera 220A is a camera having the same specifications as the first low FR camera 208A, and the seventh high FR camera 220B is a camera having the same specifications as the first high FR camera 208B. The seventh low FR camera 220A photographs the front of the trailing vehicle 204C (i.e., the side of the middle vehicle 204B as viewed from the trailing vehicle 204C) at the seventh low frame rate. The seventh high FR camera 220B photographs the front of the trailing vehicle 204C at the seventh high frame rate. The seventh low frame rate is the same as the first low frame rate, and the seventh high frame rate is the same as the first high frame rate.

[0420] The photographing direction and photographing range of the seventh low FR camera 220A are the same as those of the seventh high FR camera 220B. The seventh low FR camera 220A and the seventh high FR camera 220B photograph the front of the trailing vehicle 204C at the viewing angle θ1.

[0421] The seventh radar 220C is a radar having the same specifications as the first radar 208C, irradiates electromagnetic waves toward the front of the trailing vehicle 204C, and receives a third front object reflected wave, which is a reflected wave obtained by reflecting the irradiated electromagnetic waves at a third front object.

[0422] The eighth aspect sensor 222 includes an eighth low FR camera 222A, an eighth high FR camera 222B, and an eighth radar 222C. The eighth low FR camera 222A is an example of the "rear camera" of the present disclosure.

[0423] The eighth low FR camera 222A is a camera having the same specifications as the first low FR camera 208A, and the eighth high FR camera 222B is a camera having the same specifications as the first high FR camera 208B. The eighth low FR camera 222B photographs the rear of the trailing vehicle 204C at the eighth low frame rate. The eighth high FR camera 222B photographs the rear of the trailing vehicle 204C at the eighth high frame rate. The eighth low frame rate is the same as the first low frame rate, and the eighth high frame rate is the same as the first high frame rate.

[0424] The photographing direction and range of the eighth lowest FR camera 222A are the same as those of the eighth highest FR camera 222B. The eighth lowest FR camera 222A and the eighth highest FR camera 222B photograph the rear of the trailing vehicle 204C at a viewing angle θ2.

[0425] The eighth radar 222C is a radar having the same specifications as the first radar 208C, irradiates electromagnetic waves toward the rear of the trailing vehicle 204C, and receives a third rear object reflected wave, which is a reflected wave obtained by reflecting the irradiated electromagnetic waves at a third rear object.

[0426] Figure 23 It is a conceptual diagram showing an example of the processing contents of the seventh aspect sensor 220, the eighth aspect sensor 222, the seventh IPU 240B, the eighth IPU 240C, the seventh MoPU 240D, and the eighth MoPU 240E.

[0427] The seventh lowest FR camera 220A generates an image showing the situation in front of the trailing vehicle 204C, that is, the seventh lowest FR camera image 220A1, by photographing the front of the trailing vehicle 204C at the seventh lowest frame rate.

[0428] The seventh IPU 240B acquires the seventh lowest FR camera image 220A1 from the seventh lowest FR camera 220A at time intervals specified by the seventh lowest frame rate. Moreover, each time the seventh IPU 240B acquires the seventh lowest FR camera image 220A1, based on the seventh lowest FR camera image 220A1, it identifies the situation in front of the trailing vehicle 204C and generates seventh tag information 272 indicating the identified result. The seventh tag information 272 is information corresponding to the tag information described in the above first embodiment and the like. As an example of the seventh tag information 272, information that is tagged so as to be able to determine the type of the third front object can be cited.

[0429] The eighth lowest FR camera 222A generates an image showing the situation in the rear of the trailing vehicle 204C, that is, the eighth lowest FR camera image 222A1, by photographing the rear of the trailing vehicle 204C at the eighth lowest frame rate. The eighth lowest FR camera image 222A1 is an example of the "rear image" of the present disclosure.

[0430] The eighth IPU 240C acquires the eighth low FR camera image 222A1 from the eighth low FR camera 222A at time intervals specified by the eighth low frame rate. Further, each time the eighth IPU 240C acquires the eighth low FR camera image 222A1, it identifies the situation behind the trailing vehicle 204C based on the eighth low FR camera image 222A1, and generates eighth tag information 274 indicating the identified result. The eighth tag information 274 is information corresponding to the tag information described in the first embodiment and the like. As an example of the eighth tag information 274, information that is tagged so as to be able to determine the type of the third rear object can be cited.

[0431] The seventh high FR camera 220B generates an image showing the situation ahead, i.e., the seventh high FR camera image 220B1, by photographing the front of the trailing vehicle 204C at the seventh high frame rate.

[0432] The seventh radar 220C receives the third front object reflected wave at time intervals specified by the seventh high frame rate. Further, each time the seventh radar 220C receives the third front object reflected wave, it generates a seventh radar signal 220C1 that can determine the existence position of the third front object based on the received third front object reflected wave.

[0433] The seventh MoPU 240D acquires the seventh high FR camera image 220B1 from the seventh high FR camera 220B and the seventh radar signal 220C1 from the seventh radar 220C at time intervals specified by the seventh high frame rate. Further, the seventh MoPU 240D identifies the third front object based on the seventh high FR camera image 220B1 and the seventh radar signal 220C1, and generates seventh point information 276 indicating the identified result. For example, here, the third front object is identified as a point. The seventh point information 276 is information of the same concept as the point information described in the first embodiment and the like. That is, the seventh point information 276 is point information (for example, three-dimensional coordinates) that captures the third front object as a point.

[0434] The eighth high FR camera 222B generates an image showing the situation behind the trailing vehicle 204C, i.e., the eighth high FR camera image 222B1, by photographing the rear of the trailing vehicle 204C at the eighth high frame rate.

[0435] The eighth radar 222C receives the third rear object reflected wave at time intervals specified by the eighth high frame rate. Further, each time the eighth radar 222C receives the third rear object reflected wave, it generates an eighth radar signal 222C1 that can determine the existence position of the third rear object based on the received third rear object reflected wave.

[0436] The eighth MoPU 240E obtains the eighth high FR camera image 222B1 from the eighth high FR camera 222B at time intervals specified by the eighth high frame rate, and obtains the eighth radar signal 222C1 from the eighth radar 222C. Moreover, the eighth MoPU 240E identifies a third rear object based on the eighth high FR camera image 222B1 and the eighth radar signal 222C1, and generates eighth point information 278 indicating the identified result. For example, here, the third rear object is identified as a point. The eighth point information 278 is information of the same concept as the point information described in the above first embodiment and the like. That is, the eighth point information 278 is point information (for example, three-dimensional coordinates) that captures the third rear object as a point.

[0437] In Figure 23 the example shown, the eighth high FR camera image 222B1 is an example of the "second image" of the present disclosure. In addition, the eighth high frame rate is an example of the "fifth frame rate" of the present disclosure.

[0438] Figure 24 is a conceptual diagram showing an example of the processing content for the first central brain 224A to obtain necessary information for controlling the autonomous driving of the queue 220.

[0439] The first central brain 224A obtains first tag information 248 from the first IPU 224B and obtains first point information 252 from the first MoPU 224D. Then, the first central brain 224A generates first correspondence information 280 based on the first tag information 248 and the first point information 252. The first correspondence information 280 is information in which the first tag information 248 and the first point information 252 are corresponded in the same manner as described in the above first embodiment and the like.

[0440] In a similar manner, the first central brain 224A generates second correspondence information 282 based on the second tag information 250 obtained from the second IPU 224C and the second point information 254 obtained from the second MoPU 224E.

[0441] In Figure 24 the example shown, the first correspondence information 280 is an example of the "front correspondence information" of the present disclosure. In addition, the first point information 252 is an example of the "front point information" of the present disclosure. In addition, the first tag information 248 is an example of the "front object information" of the present disclosure.

[0442] Figure 25 is a conceptual diagram showing an example of the processing content for the second central brain 232A to obtain necessary information for controlling the autonomous driving of the queue 220.

[0443] The second central brain 232A obtains third tag information 256 from the third IPU 232C and obtains third point information 264 from the third MoPU 232F. Then, the second central brain 232A generates third corresponding information 284 based on the third tag information 258 and the third point information 264. The third corresponding information 284 is information in which the third tag information 256 and the third point information 264 are made to correspond in the same manner as described in the first embodiment and the like above.

[0444] In a similar manner, the second central brain 232A generates fourth corresponding information 286 based on fourth tag information 258 obtained from the fourth IPU 232C and fourth point information 266 obtained from the fourth MoPU 232G. In addition, the second central brain 232A generates fifth corresponding information 288 based on fifth tag information 260 obtained from the fifth IPU 232D and fifth point information 268 obtained from the fifth MoPU 232H. Further, the second central brain 232A generates sixth corresponding information 290 based on sixth tag information 262 obtained from the sixth IPU 232E and sixth point information 270 obtained from the sixth MoPU 232I.

[0445] Figure 26 It is a conceptual diagram showing an example of the processing content in which the third central brain 240A obtains necessary information in order to achieve control of the queue 220 for autonomous driving.

[0446] The third central brain 240A obtains seventh tag information 272 from the seventh IPU 240B and obtains seventh point information 276 from the seventh MoPU 240D. Then, the third central brain 240A generates seventh corresponding information 292 based on the seventh tag information 272 and the seventh point information 276. The seventh corresponding information 292 is information in which the seventh tag information 272 and the seventh point information 276 are made to correspond in the same manner as described in the first embodiment and the like above.

[0447] In a similar manner, the third central brain 240A generates eighth corresponding information 294 based on eighth tag information 274 obtained from the eighth IPU 240C and eighth point information 278 obtained from the eighth MoPU 240E.

[0448] In Figure 26 In the example shown, the eighth corresponding information 294 is an example of the "rear corresponding information" of the present disclosure. In addition, the eighth point information 278 is an example of the "rear point information" of the present disclosure. In addition, the eighth tag information 274 is an example of the "rear object information" of the present disclosure.

[0449] Figure 27This is a conceptual diagram showing an example of the processing content of the first central brain 224A for controlling the autonomous driving of the leading vehicle 204A to achieve the control of the autonomous driving of the platoon 200.

[0450] The first central brain 224A obtains the fifth corresponding information 288 and the sixth corresponding information 290 from the second central brain 232A. Then, the first central brain 224A derives the first control variable 298 based on the sensor information 296, the first corresponding information 280, the second corresponding information 282, the fifth corresponding information 288, and the sixth corresponding information 290.

[0451] The sensor information 296 is information of the same concept as the sensor information described in the above first embodiment, etc., and is obtained from various types of sensors mounted on the leading vehicle 204A. The first control variable 298 is a variable of the same concept as the control variable described in the above first embodiment, etc., and is a variable for controlling the autonomous driving of the leading vehicle 204A in order to achieve the control of the autonomous driving of the platoon 200.

[0452] The first central brain 224A has a deep learning model 300 and derives the first control variable 298 using the deep learning model 300.

[0453] The deep learning model 300 is a pre-trained model obtained by performing deep learning on a neural network using teacher data. As an example of the teacher data used here, a data set corresponding to example problem data and correct answer data assuming the first control variable 298 can be cited. As an example of the problem data, data obtained in advance through tests of actual devices and / or computer simulations, etc., as data assuming the sensor information 296, and each data obtained in advance through tests of actual devices and / or computer simulations, etc., as data assuming each of the first corresponding information 280, the second corresponding information 282, the fifth corresponding information 288, and the sixth corresponding information 290 can be cited.

[0454] The first central brain 224A inputs the sensor information 296, the first corresponding information 280, the second corresponding information 282, the fifth corresponding information 288, and the sixth corresponding information 290 into the deep learning model 300. The deep learning model 300 outputs the first control variable 298 (for example, the control variable with the highest confidence) corresponding to the input sensor information 296, the first corresponding information 280, the second corresponding information 282, the fifth corresponding information 288, and the sixth corresponding information 290.

[0455] In addition, although a derivation method for deriving the first control variable 298 by using the deep learning model 300 is illustrated herein, this is merely an example, and the first control variable 298 can also be derived by using various derivation methods (e.g., multivariate analysis based on the integration method) described in the above fourth embodiment.

[0456] The first central brain 224A controls the autonomous driving of the leading vehicle 204A based on the first control variable 298 in the same manner as described in the above first embodiment and the like.

[0457] Figure 27 It is a conceptual diagram showing an example of the processing content for the second central brain 232A to control the autonomous driving of the middle vehicle 204B in order to control the autonomous driving of the platoon 200.

[0458] The second central brain 232A derives a second control variable 304 based on the sensor information 302, the third corresponding information 284, the fourth corresponding information 286, the fifth corresponding information 288, and the sixth corresponding information 290.

[0459] The sensor information 302 is information of the same concept as the sensor information described in the above first embodiment and the like, and is obtained from various types of sensors mounted on the middle vehicle 204B. The second control variable 304 is a variable of the same concept as the control variable described in the above first embodiment and the like, and is a variable used to control the autonomous driving of the middle vehicle 204B in order to control the autonomous driving of the platoon 200.

[0460] The second central brain 232A has a deep learning model 306 and uses the deep learning model 306 to derive the second control variable 304.

[0461] The deep learning model 306 is a pre-trained model obtained by performing deep learning on a neural network using teacher data. As an example of the teacher data used herein, a data set corresponding to sample problem data and correct answer data assuming the second control variable 304 can be cited. As an example of the problem data, data obtained in advance through tests of actual devices and / or computer simulations, etc., as data assuming the sensor information 302, and data obtained in advance through tests of actual devices and / or computer simulations, etc., as data assuming each of the third corresponding information 284, the fourth corresponding information 286, the fifth corresponding information 288, and the sixth corresponding information 290 can be cited.

[0462] The second central brain 232A inputs the sensor information 302, the third corresponding information 284, the fourth corresponding information 286, the fifth corresponding information 288, and the sixth corresponding information 290 into the deep learning model 306. The deep learning model 306 outputs a second control variable 304 (e.g., the control variable with the highest confidence) corresponding to the input sensor information 302, the third corresponding information 284, the fourth corresponding information 286, the fifth corresponding information 288, and the sixth corresponding information 290.

[0463] In addition, although an example of a derivation method for deriving the second control variable 304 by using the deep learning model 306 is illustrated here, this is merely an example, and the second control variable 304 can also be derived by using various derivation methods (e.g., multivariate analysis based on the integration method) described in the above fourth embodiment.

[0464] The second central brain 232A controls the autonomous driving of the middle vehicle 204B based on the second control variable 304 in the same manner as described in the above first embodiment and the like.

[0465] Figure 29 It is a conceptual diagram showing an example of the processing content for the third central brain 240A to control the autonomous driving of the trailing vehicle 204C in order to control the autonomous driving of the platoon 200.

[0466] The third central brain 240A obtains the fifth corresponding information 288 and the sixth corresponding information 290 from the second central brain 232A. Then, the third central brain 240A derives a third control variable 310 based on the sensor information 308, the fifth corresponding information 288, the sixth corresponding information 290, the seventh corresponding information 292, and the eighth corresponding information 294.

[0467] The sensor information 308 is information of the same concept as the sensor information described in the above first embodiment and the like, and is obtained from various types of sensors mounted on the trailing vehicle 204C. The third control variable 310 is a variable of the same concept as the control variable described in the above first embodiment and the like, and is a variable used to control the autonomous driving of the trailing vehicle 204C in order to control the autonomous driving of the platoon 200.

[0468] The third central brain 240A has a deep learning model 312 and uses the deep learning model 312 to derive the third control variable 310.

[0469] The deep learning model 312 is a pre-trained model obtained by performing deep learning on a neural network using teacher data. As an example of the teacher data used herein, a data set corresponding to example problem data and correct answer data assuming the third control variable 310 can be cited. As an example of the problem data, data obtained in advance through tests of actual devices and / or computer simulations, etc., as data assuming the sensor information 308, and each data obtained in advance through tests of actual devices and / or computer simulations, etc., as data assuming each of the fifth corresponding information 288, the sixth corresponding information 290, the seventh corresponding information 292, and the eighth corresponding information 294 can be cited.

[0470] The third central brain 240A inputs the sensor information 308, the fifth corresponding information 288, the sixth corresponding information 290, the seventh corresponding information 292, and the eighth corresponding information 294 into the deep learning model 312. The deep learning model 312 outputs the third control variable 310 (for example, the control variable with the highest confidence) corresponding to the input sensor information 308, the fifth corresponding information 288, the sixth corresponding information 290, the seventh corresponding information 292, and the eighth corresponding information 294.

[0471] In addition, although a derivation method for deriving the third control variable 310 by using the deep learning model 312 is illustrated herein, this is merely an example, and the third control variable 310 can also be derived by using various derivation methods (for example, multivariate analysis based on the integration method) described in the above fourth embodiment.

[0472] The third central brain 240A controls the autonomous driving of the trailing vehicle 204C based on the third control variable 310 in the same manner as described in the above first embodiment, etc.

[0473] Next, reference will be made to Figures 30 to 38 to describe the operation of the information processing apparatus 206 according to the eleventh embodiment.

[0474] Figure 30 is a flowchart showing an example of the process of the leading vehicle IPU process executed by the leading vehicle processor 224.

[0475] In Figure 30 In the leading vehicle IPU process shown, first, in step ST10, the first IPU 224B acquires the first low FR camera image 208A1 from the first low FR camera 208A. In addition, the second IPU 224C acquires the second low FR camera image 210A1 from the second low FR camera 210A. After performing the process of step ST10, the leading vehicle IPU process proceeds to step ST12.

[0476] In step ST12, the first IPU 224B identifies the situation in front of the leading vehicle 204A (e.g., the type of the first front object) based on the first low FR camera image 208A1. In addition, the second IPU 224C identifies the situation behind the leading vehicle 204A (e.g., the type of the first rear object) based on the second low FR camera image 210A1. After performing the process of step ST12, the leading vehicle IPU process proceeds to step ST14.

[0477] In step ST14, the first IPU 224B generates first tag information 248, and the first tag information 248 indicates the result of identifying the situation in front of the leading vehicle 204A. In addition, the second IPU 224C generates second tag information 250, and the second tag information 250 indicates the result of identifying the situation behind the leading vehicle 204A. After performing the process of step ST14, the leading vehicle IPU process proceeds to step ST16.

[0478] In step ST16, the leading vehicle processor 224 determines whether the condition for ending the front vehicle IPU process is satisfied. As an example of the condition for ending the leading vehicle IPU process, a condition that an instruction to end the leading vehicle IPU process is issued to the leading vehicle processor 224 can be cited. In step ST16, when the condition for ending the leading vehicle IPU process is not satisfied, the determination is negative, and the leading vehicle IPU process proceeds to step ST10. In step ST16, when the condition for ending the leading vehicle IPU process is satisfied, the determination is positive, and the leading vehicle IPU process ends.

[0479] Figure 31 is a flowchart showing an example of the process flow of the leading vehicle MoPU process executed by the leading vehicle processor 224.

[0480] In Figure 31 In the leading vehicle MoPU process shown, first, in step ST50, the first MoPU 224D obtains a first high FR camera image 208B1 from the first high FR camera 208A. In addition, the first MoPU 224D obtains a first radar signal 208C1 from the first radar 208C. In addition, the second MoPU 224E obtains a second high FR camera image 210B1 from the second high FR camera 210B. Further, the second MoPU 224E obtains a second radar signal 210C1 from the second radar 210C. After performing the process of step ST50, the leading vehicle MoPU process proceeds to step ST52.

[0481] In step ST52, the first MoPU 224D identifies the situation in front of the leading vehicle 204A (e.g., the first front object) as points based on the first high FR camera image 208B1 and the first radar signal 208C1. Additionally, the second MoPU 224E identifies the situation behind the leading vehicle 204A (e.g., the first rear object) as points based on the second high FR camera image 210B1 and the second radar signal 210C1. After performing the processing of step ST52, the leading vehicle MoPU processing proceeds to step ST54.

[0482] In step ST54, the first MoPU 224D generates first point information 252, which indicates the result of identifying the situation in front of the leading vehicle 204A as points. Additionally, the second MoPU 224E generates second point information 254, which indicates the result of identifying the situation behind the leading vehicle 204A as points. After performing the processing of step ST54, the leading vehicle MoPU processing proceeds to step ST56.

[0483] In step ST56, the leading vehicle processor 224 determines whether the condition for ending the leading vehicle MoPu processing is satisfied. As an example of the condition for ending the leading vehicle MoPU processing, the condition that an instruction to end the leading vehicle MoPU processing is issued to the leading vehicle processor 224 can be cited. In step ST56, when the condition for ending the leading vehicle MoPU processing is not satisfied, the determination is negative, and the leading vehicle MoPU processing proceeds to step ST50. In step ST56, when the condition for ending the leading vehicle MoPU processing is satisfied, the determination is positive, and the leading vehicle MoPU processing ends.

[0484] Figure 32 It is a flowchart showing an example of the process of the first central brain processing executed by the leading vehicle processor 224.

[0485] In Figure 32 In the first central brain processing shown, first, in step ST100, the first central brain 224A acquires first tag information 248 from the first IPU 224B and second tag information 250 from the second IPU 224C. After performing the processing of step ST100, the first central brain processing proceeds to step ST102.

[0486] In step ST102, the first central brain 224A acquires first point information 252 from the first MoPU 224D and second point information 254 from the second MoPU 224E. After performing the processing of step ST102, the first central brain processing proceeds to step ST104.

[0487] In step ST104, the first central brain 224A generates first correspondence information 280 by making the first tag information 248 correspond to the first point information 252. Additionally, the first central brain 224A generates second correspondence information 282 by making the second tag information 250 correspond to the second point information 254. After performing the process of step ST104, the first central brain proceeds to step ST106.

[0488] In step ST106, the first central brain 224A obtains fifth correspondence information 288 and sixth correspondence information 290 from the second central brain 232A. After performing the process of step ST106, the first central brain proceeds to step ST108.

[0489] In step ST108, the first central brain 224A obtains sensor information 296 from various sensors of the leading vehicle 204A. After performing the process of step ST108, the first central brain proceeds to step ST110.

[0490] In step ST110, the first central brain 224A derives a first control variable 298 based on the sensor information 296, the first correspondence information 280, the second correspondence information 282, the fifth correspondence information 288, and the sixth correspondence information 290. After performing the process of step ST110, the first central brain proceeds to step ST112.

[0491] In step ST112, the first central brain 224A controls the autonomous driving of the leading vehicle 204A based on the first control variable 298. After performing the process of step ST112, the first central brain proceeds to step ST114.

[0492] In step ST114, the first central brain 224A determines whether the condition for ending the first central brain process is satisfied. As an example of the condition for ending the first central brain process, a condition where an instruction to end the first central brain process is issued to the leading vehicle processor 224 can be cited. In step ST114, when the condition for ending the first central brain process is not satisfied, the determination is negative, and the first central brain process proceeds to step ST100. In step ST114, when the condition for ending the first central brain process is satisfied, the determination is positive, and the first central brain process ends.

[0493] Figure 33 It is a flowchart showing an example of the process of the intermediate vehicle IPU process executed by the intermediate vehicle processor 232.

[0494] In Figure 33In the intermediate vehicle IPU processing shown, first, in step ST150, the third IPU 232B obtains the third low FR camera image 212A1 from the third low FR camera 212A. Additionally, the fourth IPU 232C obtains the fourth low FR camera image 214A1 from the fourth low FR camera 214A. Additionally, the fifth IPU 232D obtains the fifth low FR camera image 216A1 from the fifth low FR camera 216A. Furthermore, the sixth IPU 232E obtains the sixth low FR camera image 218A1 from the sixth low FR camera 218A. After performing the processing of step ST150, the intermediate vehicle IPU processing proceeds to step ST152.

[0495] In step ST152, the third IPU 232B identifies the situation in front of the intermediate vehicle 204B (e.g., the type of the second front object) based on the third low FR camera image 212A1. Additionally, the fourth IPU 232C identifies the situation behind the intermediate vehicle 204B (e.g., the type of the second rear object) based on the fourth low FR camera image 214A1. Additionally, the fifth IPU 232D identifies the situation to the left of the intermediate vehicle 204B (e.g., the type of the left-side object) based on the fifth low FR camera image 216A1. Furthermore, the sixth IPU 232E identifies the situation to the right of the intermediate vehicle 204B (e.g., the type of the right-side object) based on the sixth low FR camera image 218A1. After performing the processing of step ST152, the intermediate vehicle IPU processing proceeds to step ST154.

[0496] In step ST154, the third IPU 232B generates the third label information 256, which indicates the result of identifying the situation in front of the intermediate vehicle 204B. Additionally, the fourth IPU 232C generates the fourth label information 258, which indicates the result of identifying the situation behind the intermediate vehicle 204B. Additionally, the fifth IPU 232D generates the fifth label information 260, which indicates the result of identifying the situation to the left of the intermediate vehicle 204B. Furthermore, the sixth IPU 232E generates the sixth label information 262, which indicates the result of identifying the situation to the right of the intermediate vehicle 204B. After performing the processing of step ST154, the intermediate vehicle IPU processing proceeds to step ST156.

[0497] In step ST156, the intermediate vehicle processor 232 determines whether the condition for the end of the intermediate vehicle IPU processing is satisfied. As an example of the condition for the end of the intermediate vehicle IPU processing, a condition that an instruction to end the intermediate vehicle IPU processing is issued to the intermediate vehicle processor 232 can be cited. In step ST156, when the condition for the end of the intermediate vehicle IPU processing is not satisfied, the determination is negative, and the intermediate vehicle IPU processing proceeds to step ST150. In step ST156, when the condition for the end of the intermediate vehicle IPU processing is satisfied, the determination is positive, and the intermediate vehicle IPU processing ends.

[0498] Figure 34 FIG. is a flowchart showing an example of the process of the intermediate vehicle MoPU processing executed by the intermediate vehicle processor 232.

[0499] In Figure 34 In the intermediate vehicle MoPU processing shown, first, in step ST200, the third MoPU 232F acquires the third high FR camera image 212B1 from the third high FR camera 212B. In addition, the third MoPU 232F acquires the third radar signal 212C1 from the third radar 212C. In addition, the fourth MoPU 232G acquires the fourth high FR camera image 214B1 from the fourth high FR camera 214B. In addition, the fourth MoPU 232G acquires the fourth radar signal 214C1 from the fourth radar 214C. In addition, the fifth MoPU 232H acquires the fifth high FR camera image 216B1 from the fifth high FR camera 216B. In addition, the fifth MoPU 232H acquires the fifth radar signal 216C1 from the fifth radar 216C. In addition, the sixth MoPU 232I acquires the sixth high FR camera image 218B1 from the sixth high FR camera 218B. Further, the sixth MoPU 232I acquires the sixth radar signal 218C1 from the sixth radar 218C. After performing the processing of step ST200, the intermediate vehicle MoPU processing proceeds to step ST202.

[0500] In step ST202, the third MoPU 232F identifies the situation in front of the intermediate vehicle 204B (e.g., the second front object) as a point based on the third highest FR camera image 212B1 and the third radar signal 212C1. Additionally, the fourth MoPU 232G identifies the situation behind the intermediate vehicle 204B (e.g., the second rear object) as a point based on the fourth highest FR camera image 214B1 and the fourth radar signal 214C1. Additionally, the fifth MoPU 232H identifies the situation to the left of the intermediate vehicle 204B (e.g., the left-side object) as a point based on the fifth highest FR camera image 216B1 and the fifth radar signal 216C1. Furthermore, the sixth MoPU 232I identifies the situation to the right of the intermediate vehicle 204B (e.g., the right-side object) as a point based on the sixth highest FR camera image 218B1 and the sixth radar signal 218C1. After performing the processing of step ST202, the intermediate vehicle MoPU processing proceeds to step ST204.

[0501] In step ST204, the third MoPU 232F generates third point information 264, which indicates the result of identifying the situation in front of the intermediate vehicle 204B as a point. Additionally, the fourth MoPU 232G generates fourth point information 266, which indicates the result of identifying the situation behind the intermediate vehicle 204B as a point. Additionally, the fifth MoPU 232H generates fifth point information 268, which indicates the result of identifying the situation to the left of the intermediate vehicle 204B as a point. Furthermore, the sixth MoPU 232I generates sixth point information 270, which indicates the result of identifying the situation to the right of the intermediate vehicle 204B as a point. After performing the processing of step ST200, the intermediate vehicle MoPU processing proceeds to step ST206.

[0502] In step ST206, the intermediate vehicle processor 232 determines whether the condition for ending the intermediate vehicle MoPU processing is satisfied. As an example of the condition for ending the intermediate vehicle MoPU processing, there can be cited the condition that an instruction to end the intermediate vehicle MoPU processing is issued to the intermediate vehicle processor 232. In step ST206, when the condition for ending the intermediate vehicle MoPU processing is not satisfied, the determination is negative, and the intermediate vehicle MoPU processing proceeds to step ST200. In step ST206, when the condition for ending the intermediate vehicle MoPU processing is satisfied, the determination is positive, and the intermediate vehicle MoPU processing ends.

[0503] Figure 35 It is a flowchart showing an example of the process of the second central brain processing executed by the intermediate vehicle processor 232.

[0504] In Figure 35In the second central brain processing shown, first, in step ST250, the second central brain 232A obtains the third tag information 256 from the third IPU 232B, the fourth tag information 258 from the fourth IPU 232C, the fifth tag information 260 from the fifth IPU 232D, and the sixth tag information 262 from the sixth IPU 232E. After performing the processing of step ST250, the second central brain processing proceeds to step ST252.

[0505] In step ST252, the second central brain 232A obtains the third point information 264 from the third MoPU 232F, the fourth point information 266 from the fourth MoPU 232G, the fifth point information 268 from the fifth MoPU 232H, and the sixth point information 270 from the sixth MoPU 232I. After performing the processing of step ST252, the second central brain processing proceeds to step ST254.

[0506] In step ST254, the second central brain 232A generates the third correspondence information 284 by making the third tag information 256 correspond to the third point information 264. Additionally, the second central brain 232A generates the fourth correspondence information 286 by making the fourth tag information 258 correspond to the fourth point information 266. Additionally, the second central brain 232A generates the fifth correspondence information 288 by making the fifth tag information 260 correspond to the fifth point information 268. Further, the second central brain 232A generates the sixth correspondence information 290 by making the sixth tag information 262 correspond to the sixth point information 270. After performing the processing of step ST254, the second central brain processing proceeds to step ST256.

[0507] In step ST256, the second central brain 232A obtains the sensor information 302 from various sensors of the middle vehicle 204B. After performing the processing of step ST256, the second central brain processing proceeds to step ST258.

[0508] In step ST258, the second central brain 232A derives the second control variable 304 based on the sensor information 302, the third correspondence information 284, the fourth correspondence information 286, the fifth correspondence information 288, and the sixth correspondence information 290. After performing the processing of step ST258, the second central brain processing proceeds to step ST260.

[0509] In step ST260, the second central brain 232A controls the autonomous driving of the middle vehicle 204B based on the second control variable 304. After performing the processing of step ST260, the second central brain processing proceeds to step ST262.

[0510] In step ST262, the second central brain 232A determines whether the condition for the end of the second central brain processing is satisfied. As an example of the condition for the end of the second central brain processing, a condition that an instruction to end the second central brain processing is issued to the intermediate vehicle processor 232 can be cited. In step ST262, when the condition for the end of the second central brain processing is not satisfied, the determination is negative, and the second central brain processing proceeds to step ST250. In step ST262, when the condition for the end of the second central brain processing is satisfied, the determination is positive, and the second central brain processing ends.

[0511] Figure 36 FIG. is a flowchart showing an example of the process of the end vehicle IPU process executed by the end vehicle processor 240.

[0512] In Figure 36 In the end vehicle IPU process shown, first, in step ST300, the seventh IPU 240B acquires a seventh low FR camera image 220A1 from the seventh low FR camera 220A. In addition, the eighth IPU 240C acquires an eighth low FR camera image 222A1 from the eighth low FR camera 222A. After performing the process of step ST300, the end vehicle IPU process proceeds to step ST302.

[0513] In step ST302, the seventh IPU 240B identifies the situation in front of the end vehicle 204C (e.g., the type of the third front object) based on the seventh low FR camera image 220A1. In addition, the eighth IPU 240C identifies the situation behind the end vehicle 204C (e.g., the type of the third rear object) based on the eighth low FR camera image 222A1. After performing the process of step ST302, the end vehicle IPU process proceeds to step ST304.

[0514] In step ST304, the seventh IPU 240B generates seventh tag information 272, and the seventh tag information 272 indicates the result of identifying the situation in front of the end vehicle 204C. In addition, the eighth IPU 240C generates eighth tag information 274, and the eighth tag information 274 indicates the result of identifying the situation behind the end vehicle 204C. After performing the process of step ST304, the end vehicle IPU process proceeds to step ST306.

[0515] In step ST306, the trailing vehicle processor 240 determines whether the condition for the end of the trailing vehicle IPU processing is satisfied. As an example of the condition for the end of the trailing vehicle IPU processing, the condition that an instruction to end the trailing vehicle IPU processing is issued to the trailing vehicle processor 240 can be cited. In step ST306, when the condition for the end of the trailing vehicle IPU processing is not satisfied, the determination is negative, and the trailing vehicle IPU processing proceeds to step ST300. In step ST306, when the condition for the end of the trailing vehicle IPU processing is satisfied, the determination is positive, and the trailing vehicle IPU processing ends.

[0516] Figure 37 is a flowchart showing an example of the process of the trailing vehicle MoPU processing executed by the trailing vehicle processor 240.

[0517] In Figure 37 In the trailing vehicle MoPU processing shown, first, in step ST350, the seventh MoPU 240D acquires the seventh high FR camera image 220B1 from the seventh high FR camera 220B. In addition, the seventh MoPU 240D acquires the seventh radar signal 220C1 from the seventh radar 220C. In addition, the eighth MoPU 240E acquires the eighth high FR camera image 222B1 from the eighth high FR camera 222B. Further, the eighth MoPU 240E acquires the eighth radar signal 222C1 from the eighth radar 222C. After performing the processing of step ST350, the trailing vehicle MoPU processing proceeds to step ST352.

[0518] In step ST352, the seventh MoPU 240D identifies the situation in front of the trailing vehicle 204C (e.g., the third front object) as points based on the seventh high FR camera image 220B1 and the seventh radar signal 220C1. In addition, the eighth MoPU 240E identifies the situation behind the trailing vehicle 204C (e.g., the third rear object) as points based on the eighth high FR camera image 222B1 and the eighth radar signal 222C1. After performing the processing of step ST352, the trailing vehicle MoPU processing proceeds to step ST354.

[0519] In step ST354, the seventh MoPU 240D generates the seventh point information 276, and the seventh point information 276 indicates the result of identifying the situation in front of the trailing vehicle 204C as points. In addition, the eighth MoPU 240E generates the eighth point information 278, and the eighth point information 278 indicates the result of identifying the situation behind the trailing vehicle 204C as points. After performing the processing of step ST354, the trailing vehicle MoPU processing proceeds to step ST356.

[0520] In step ST356, the trailing vehicle processor 240 determines whether the condition for the end of the trailing vehicle MoPU process is satisfied. As an example of the condition for the end of the trailing vehicle MoPU process, the condition that an instruction to end the trailing vehicle MoPU process is issued to the trailing vehicle processor 240 can be cited. In step ST356, when the condition for the end of the trailing vehicle MoPU process is not satisfied, the determination is negative, and the trailing vehicle MoPU process proceeds to step ST350. In step ST356, when the condition for the end of the trailing vehicle MoPU process is satisfied, the determination is positive, and the trailing vehicle MoPU process ends.

[0521] Figure 38 is a flowchart showing an example of the process of the third central brain process executed by the trailing vehicle processor 240.

[0522] In Figure 38 In the third central brain process shown, first, in step ST400, the third central brain 240A obtains the seventh tag information 272 from the seventh IPU 240B and obtains the eighth tag information 274 from the eighth IPU 240C. After executing the process of step ST400, the third central brain process proceeds to step ST402.

[0523] In step ST402, the third central brain 240A obtains the seventh point information 276 from the seventh MoPU 240D and obtains the eighth point information 278 from the eighth MoPU 240E. After executing the process of step ST402, the third central brain process proceeds to step ST404.

[0524] In step ST404, the third central brain 240A generates the seventh correspondence information 292 by making the seventh tag information 272 correspond to the seventh point information 276. In addition, the third central brain 240A generates the eighth correspondence information 294 by making the eighth tag information 274 correspond to the eighth point information 278. After executing the process of step ST404, the third central brain process proceeds to step ST406.

[0525] In step ST406, the third central brain 240A obtains the fifth correspondence information 288 and the sixth correspondence information 290 from the second central brain 232A. After executing the process of step ST406, the third central brain process proceeds to step ST408.

[0526] In step ST408, the third central brain 232A obtains the sensor information 308 from various sensors of the trailing vehicle 204C. After executing the process of step ST408, the third central brain process proceeds to step ST410.

[0527] In step ST410, the third central brain 240A derives a third control variable 310 based on the sensor information 308, the fifth corresponding information 288, the sixth corresponding information 290, the seventh corresponding information 292, and the eighth corresponding information 294. After the process of step ST410 is executed, the third central brain process proceeds to step ST412.

[0528] In step ST412, the third central brain 240A controls the autonomous driving of the trailing vehicle 204C based on the third control variable 310. After the process of step ST412 is executed, the third central brain process proceeds to step ST414.

[0529] In step ST414, the third central brain 240A determines whether the condition for ending the third central brain process is satisfied. As an example of the condition for ending the third central brain process, a condition that an instruction to end the third central brain process is issued to the trailing vehicle processor 240 can be cited. In step ST414, when the condition for ending the third central brain process is not satisfied, the determination is negative, and the third central brain process proceeds to step ST400. In step ST414, when the condition for ending the third central brain process is satisfied, the determination is positive, and the third central brain process ends.

[0530] As described above, in this eleventh embodiment, through the first IPU 224B, the situation in front of the platoon 200 is recognized based on the first low FR camera image 208A1. In addition, through the eighth IPU 240D, the situation behind the platoon 200 is recognized based on the eighth low FR camera image 222A1. Furthermore, through the fifth MoPU 232H1, the situation on the left side of the platoon 200 is recognized based on the fifth high FR camera image 216B1, and through the sixth MoPU 232I, the situation on the right side of the platoon 200 is recognized based on the sixth high FR camera image 218B1.

[0531] Here, the fifth high FR camera image 216B is an image obtained by the fifth high FR camera 216B mounted on the middle vehicle 204B photographing the left side of the platoon 200, and the sixth high FR camera image 218B1 is an image obtained by the sixth high FR camera 218B mounted on the middle vehicle 204B photographing the right side of the platoon 200. That is, instead of the cameras provided in each of all the vehicles 204 constituting the platoon 200 photographing the sides of the platoon 200, the fifth high FR camera 216B and the sixth high FR camera 218B mounted on one middle vehicle 204B constituting the platoon 200 photograph the sides of the platoon 200 (here, as an example, the left side and the right side of the platoon 200).

[0532] Therefore, according to the eleventh embodiment, compared with the case where the information processing device 206 identifies the situation on the side of the platoon 200 based on all the images obtained by photographing the side of the platoon 200 by each camera provided in each of all the vehicles 204 constituting the platoon 200, the information processing device 206 can identify the situations in the front, rear, and side of the platoon 200 without imposing a processing burden.

[0533] In addition, in the eleventh embodiment, the fifth highest FR camera 216A and the sixth highest FR camera 218B are provided in the middle vehicle 204B instead of the leading vehicle 204A and the trailing vehicle 204C. Then, the situation on the left side of the platoon 200 is identified based on the fifth highest FR camera image 218B1 obtained by photographing the left side of the platoon 200 by the fifth highest FR camera 216A, and the situation on the right side of the platoon 200 is identified based on the sixth highest FR camera image 218B1 obtained by photographing the right side of the platoon 200 by the sixth highest FR camera 218B. In the eleventh embodiment, the autonomous driving of the platoon 200 is controlled based on the identified results of the situation on the left side of the platoon 200 and the situation on the right side of the platoon 200. Therefore, when the platoon 200 is under autonomous driving, since the information processing device 206 grasps the side situation of the middle position of the entire platoon 200, compared with the case where the fifth highest FR camera 216A and the sixth highest FR camera 218B are provided in the leading vehicle 204A or the trailing vehicle 204C instead of the middle vehicle 204B, the entire platoon 200 can move with high precision.

[0534] Further, in the eleventh embodiment, the fifth highest FR camera 216B and the sixth highest FR camera 218B photograph the side of the platoon 200 at the fifth highest frame rate and the sixth highest frame rate, which are higher than the frame rates of the first lowest FR camera 208A and the eighth lowest FR camera 222A. Therefore, according to the eleventh embodiment, the fifth MoPU 232H and the sixth MoPU 232I can identify the situation on the side of the platoon 200 at a time interval shorter than the time interval at which the first IPU 224B identifies the situation in the front of the platoon 200 based on the first lowest FR camera image 208A1 and the time interval at which the eighth IPU 240C identifies the situation in the rear of the platoon 200 based on the eighth lowest FR camera image 222A1.

[0535] In addition, in this eleventh embodiment, when a side camera that captures the side of the queue 200 at the same frame rate as the fifth highest FR camera 216B and the sixth highest FR camera 218B is provided in each of all the vehicles 204 that make up the queue 200, not all the images obtained by capturing with all the side cameras will be processed. That is, the fifth MoPU 232H and the sixth MoPU 232I identify the side conditions of the queue 200 based on the fifth highest FR camera image 216B1 and the sixth highest FR camera image 218B1 obtained by capturing the side of the queue 200 only with the fifth highest FR camera 216B and the sixth highest FR camera 218B provided only in the middle vehicle 204B. Therefore, according to this eleventh embodiment, compared with the case where the information processing device 206 identifies the side of the queue 200 based on all the images obtained by capturing the side of the queue 200 with each side camera provided in each of all the vehicles 204 that make up the queue 200, the processing burden imposed on the information processing device 206 can be reduced.

[0536] In addition, in this eleventh embodiment, whenever the fifth highest FR camera image 216B1 is obtained by capturing the left side of the queue 200 at the fifth highest frame rate, which is higher than the frame rates of the first low FR camera 208A and the eighth low FR camera 222A, the fifth MoPU 232H identifies the left side conditions of the queue 200 based on the obtained fifth highest FR camera image 216B1. In addition, whenever the sixth highest FR camera image 218B1 is obtained by capturing the right side of the queue 200 at the sixth highest frame rate, which is higher than the frame rates of the first low FR camera 208A and the eighth low FR camera 222A, the sixth MoPU 232I identifies the right side conditions of the queue 200 based on the obtained sixth highest FR camera image 218B1. Therefore, according to this eleventh embodiment, the information processing device 206 can identify the side of the queue 200 in detail.

[0537] In addition, in this eleventh embodiment, the front conditions of the queue 200 are identified by the first IPU 224B identifying the type of the first front object based on the first low FR camera image 208A1, and the rear conditions of the queue 200 are identified by the eighth IPU 240C identifying the type of the third rear object based on the eighth low FR camera image 222A1. Therefore, according to this eleventh embodiment, the information processing device 206 and the like can accurately grasp the respective conditions of the front and rear of the queue 200.

[0538] In addition, in this eleventh embodiment, the first IPU 224B identifies the front situation of the queue 200, the eighth IPU 240C identifies the rear situation of the queue 200, and the fifth MoPU 232H and the sixth MoPU 232I identify the side situations of the queue 200. Then, the autonomous driving of the queue 200 (for example, the autonomous driving of each of the leading vehicle 204A, the intermediate vehicle 204B, and the trailing vehicle 204C) is controlled based on the front situation of the queue 200, the rear situation of the queue 200, and the side situations of the queue 200. Therefore, according to this eleventh embodiment, safe autonomous driving of the queue 200 can be achieved.

[0539] In addition, in this eleventh embodiment, the autonomous driving of the queue 200 is controlled based on the first correspondence information 280 and the eighth correspondence information 294. The first correspondence information 280 indicates the recognition result for the front situation of the queue 200, and is information in which the first label information 248 capable of determining the type of the first front object corresponds to the first point information 252 representing the first front object as a point. In addition, the eighth correspondence information 294 indicates the recognition result for the rear situation of the queue 200, and is information in which the eighth label information 274 capable of determining the type of the third rear object corresponds to the eighth point information 278 representing the third rear object as a point. Therefore, according to this eleventh embodiment, by controlling the autonomous driving of the queue 200 based on the first correspondence information 280 and the eighth correspondence information 294, safe autonomous driving of the queue 200 in the front-rear direction can be achieved.

[0540] In addition, in this eleventh embodiment, the autonomous driving of the queue 200 is controlled based on the fifth correspondence information 288 and the sixth correspondence information 290. The fifth correspondence information 288 indicates the recognition result for the left-side situation of the queue 200, and is information in which the fifth label information 260 capable of determining the type of the left-side object corresponds to the fifth point information 268 representing the left-side object as a point. In addition, the sixth correspondence information 290 indicates the recognition result for the right-side situation of the queue 200, and is information in which the sixth label information 262 capable of determining the type of the right-side object corresponds to the eighth point information 270 representing the right-side object as a point. Therefore, according to this eleventh embodiment, by controlling the autonomous driving of the queue 200 based on the fifth correspondence information 288 and the sixth correspondence information 290, the safety of the left side and the right side of the queue 200 moving by autonomous driving can be ensured.

[0541] In addition, although an example of controlling the autonomous driving of the queue 200 based on the fifth corresponding information 288 and the sixth corresponding information 290 is shown here, this is merely an example, and the autonomous driving of the queue 200 may also be controlled based on the fifth tag information 260 or the fifth point information 268, and the sixth tag information 262 or the sixth point information 270.

[0542] In addition, in the eleventh embodiment, the first IPU 224B recognizes the front situation of the queue 200, the eighth IPU 240C recognizes the rear situation of the queue 200, and the fifth MoPU 232H and the sixth MoPU 232I recognize the side situation of the queue 200. Therefore, according to the eleventh embodiment, compared with the case where the front situation, the rear situation, and the side situation of the queue 200 are recognized by a single processor mounted on the information processing device 206, the processing load imposed on one processor mounted on the information processing device 206 can be reduced.

[0543] In addition, in the eleventh embodiment, the fifth MoPU 232H and the sixth MoPU 232I recognize the side situation of the queue 200 based on the fifth high FR camera image 216B1 and the sixth high FR camera image 218B1 by performing processing faster than the first IPU 224B and the eighth IPU 240C. Therefore, according to the eleventh embodiment, the fifth MoPU 232H and the sixth MoPU 232I can recognize the side situation of the queue 200 in a time shorter than the time required for the first IPU 224B to recognize the front situation of the queue 200 and the time required for the eighth IPU 240C to recognize the rear situation of the queue 200.

[0544] In addition, although in the above eleventh embodiment, the case where the third high frame rate and the fourth high frame rate are the same as the first high frame rate and the second high frame rate has been described, the technology of the present disclosure is not limited thereto. For example, the third high frame rate and the fourth high frame rate may be lower than the first high frame rate and the second high frame rate. In this case, as an example of the third high frame rate and the fourth high frame rate, frame rates equal to or lower than the first low frame rate and the second low frame rate can be cited. In this way, by making the third high frame rate and the fourth high frame rate lower than the first high frame rate and the second high frame rate, the processing load imposed on the intermediate vehicle processor 232 can be reduced. As a result, the processing load of the entire information processing device 206 is also reduced.

[0545] Although in the above-described eleventh embodiment, an example of the manner in which the first control variable 298 is derived by the first central brain 224A, the second control variable 304 is derived by the second central brain 232A, and the third control variable 310 is derived by the third central brain 240A is shown, the technology of the present disclosure is not limited thereto. For example, two or more of the first control variable 298, the second control variable 304, and the third control variable 310 may be derived by one processor.

[0546] In Figure 39 it is shown an example of the manner in which the first control variable 298, the second control variable 304, and the third control variable 310 are derived by the first central brain 224A. In Figure 39 the example shown, the first central brain 224A has a deep learning model 314, and uses the deep learning model 314 to derive the first control variable 298, the second control variable 304, and the third control variable 310.

[0547] The deep learning model 314 is a pre-trained model obtained by performing deep learning on a neural network using teacher data. As an example of the teacher data used herein, a data set corresponding to example problem data and correct answer data assuming the first control variable 298, the second control variable 304, and the third control variable 310 can be cited. As an example of the problem data, each data obtained in advance through tests of actual devices and / or computer simulations, etc., as each data assuming each of the sensor information 296, 302, and 308, and each data obtained in advance through tests of actual devices and / or computer simulations, etc., as each data assuming each of the first corresponding information 280, the second corresponding information 282, the third corresponding information 284, the fourth corresponding information 286, the fifth corresponding information 288, the sixth corresponding information 290, the seventh corresponding information 292, and the eighth corresponding information 294 can be cited.

[0548] The first central brain 224A inputs the sensor information 296, 302, and 308, and the first corresponding information 280, the second corresponding information 282, the third corresponding information 284, the fourth corresponding information 286, the fifth corresponding information 288, the sixth corresponding information 290, the seventh corresponding information 292, and the eighth corresponding information 294 into the deep learning model 314. The deep learning model 314 outputs the first control variable 298, the second control variable 304, and the third control variable 310 (for example, the control variable with the highest confidence) corresponding to the input sensor information 296, 302, and 308, and the first corresponding information 280, the second corresponding information 282, the third corresponding information 284, the fourth corresponding information 286, the fifth corresponding information 288, the sixth corresponding information 290, the seventh corresponding information 292, and the eighth corresponding information 294.

[0549] In addition, although this example illustrates a derivation method for deriving the first control variable 298, the second control variable 304, and the third control variable 310 by using the deep learning model 314, this is merely an example, and the first control variable 298, the second control variable 304, and the third control variable 310 can also be derived by using various derivation methods (for example, multivariable analysis based on the integration method) described in the above fourth embodiment.

[0550] When the first central brain 224A derives the first control variable 298, the second control variable 304, and the third control variable 310, similar to the above-described eleventh embodiment, the first control variable 298 is used by the first central brain 224A for controlling the autonomous driving of the leading vehicle 204A, the second control variable 304 is used by the second central brain 232A for controlling the autonomous driving of the middle vehicle 204B, and the third control variable 310 is used by the third central brain 240A for controlling the autonomous driving of the trailing vehicle 204C.

[0551] In the above-described eleventh embodiment, an example is shown in which the type of the left-side object existing on the left side of the queue 200 is identified based on the fifth low FR camera image 216A1, and the left-side object existing on the left side of the queue 200 is identified as a point based on the fifth high FR camera image 216B1. In addition, in the above-described eleventh embodiment, an example is shown in which the type of the right-side object existing on the right side of the queue 200 is identified based on the sixth low FR camera image 218A1, and the right-side object existing on the right side of the queue 200 is identified as a point based on the sixth high FR camera image 218B1. However, the technology of the present disclosure is not limited thereto. For example, instead of performing the process of identifying the type of the left-side object existing on the left side of the queue 200, the process of identifying the left-side object existing on the left side of the queue 200 as a point based on the fifth high FR camera image 216B1 may be performed. In addition, instead of performing the process of identifying the type of the right-side object existing on the right side of the queue 200, the process of identifying the right-side object existing on the right side of the queue 200 as a point based on the sixth high FR camera image 218B1 may be performed. Thus, compared with the case where the type of the left-side object existing on the left side of the queue 200 is identified as the left-side situation of the queue 200 and the type of the right-side object existing on the right side of the queue 200 is identified as the right-side situation of the queue 200, the information processing device 206 can identify each situation on the left side and the right side of the queue 200 with a lighter processing load.

[0552] In addition, in this way, for example, as Figure 40 shown, for the control variables required for controlling autonomous driving (in Figure 40In the example shown, as an example, when the first control variable 298, the second control variable 304, and the third control variable 310 are derived, the fifth piece of information 268 can be used to replace the fifth corresponding information 288, and the sixth piece of information 270 can be used to replace the sixth corresponding information 290.

[0553] In Figure 40 In the example shown, the first control variable 298, the second control variable 304, and the third control variable 310 are derived by using the deep learning model 316. The deep learning model 316 is a pre-trained model obtained by performing deep learning on a neural network using teacher data. As an example of the teacher data used herein, a data set corresponding to example problem data and correct answer data assuming the first control variable 298, the second control variable 304, and the third control variable 310 can be cited. As an example of the example problem data, each data obtained in advance through tests of actual devices and / or computer simulations, etc., as each data assuming each of the sensor information 296, 302, and 308, each data obtained in advance through tests of actual devices and / or computer simulations, etc., as each data assuming each of the first corresponding information 280, the second corresponding information 282, the third corresponding information 284, the fourth corresponding information 286, the seventh corresponding information 292, and the eighth corresponding information 294, and each data obtained in advance through tests of actual devices and / or computer simulations, etc., as each data assuming each of the fifth piece of information 268 and the sixth piece of information 270 can be cited.

[0554] In this way, by using the fifth piece of information 268 to replace the fifth corresponding information 288 and using the sixth piece of information 270 to replace the sixth corresponding information 290, the processing load involved in the derivation of the control variable can be reduced. In addition, compared with the case of constructing the deep learning model 314, the load required to construct the deep learning model 316 (for example, the load involved in deep learning) can also be reduced. In addition, by using the fifth piece of information 268 to replace the fifth corresponding information 288 and using the sixth piece of information 270 to replace the sixth corresponding information 290, the fifth label information 260 and the sixth label information 262 are not required. In this case, the processing of the fifth low FR camera image 216A1 and the sixth low FR camera image 218A1 is not required, so the fifth IPU 232D and the sixth IPU 232E are not required either. In addition, the fifth low FR camera 2216A and the sixth low FR camera 218A are not required either. As a result, the processing load imposed on the second information processing device 206B can be reduced. In addition, the number of components mounted on the intermediate vehicle 204B can be reduced, which can contribute to cost reduction.

[0555] In addition, although an example of using the fifth point information 268 to replace the fifth corresponding information 288 and using the sixth point information 270 to replace the sixth corresponding information 290 is shown here, it is also possible to use the fifth tag information 260 to replace the fifth corresponding information 288 and use the sixth tag information 262 to replace the sixth corresponding information 290.

[0556] Although in the example as Figure 40 shown, an example of the method of deriving the first control variable 298, the second control variable 304, and the third control variable 310 based on the sensor information 296, 302, and 308, the first corresponding information 280, the second corresponding information 282, the third corresponding information 284, the fourth corresponding information 286, the seventh corresponding information 292, and the eighth corresponding information 294, and the fifth point information 268 and the sixth point information 270 is shown, the technology of the present disclosure is not limited thereto. For example, as Figure 41 shown, it is possible to derive the first control variable 298, the second control variable 304, and the third control variable 310 based on the sensor information 296, 302, and 308, the first corresponding information 280, the second corresponding information 282, the seventh corresponding information 292, and the eighth corresponding information 294, and the fifth point information 268 and the sixth point information 270.

[0557] In this case, the first control variable 298, the second control variable 304, and the third control variable 310 are derived by using the deep learning model 318. The deep learning model 318 is a pre-trained model obtained by performing deep learning on a neural network using teacher data. As an example of the teacher data used here, a data set corresponding to example problem data and correct answer data assuming the first control variable 298, the second control variable 304, and the third control variable 310 can be cited. As an example of the problem data, each data obtained in advance through tests of actual devices and / or computer simulations, etc., as data assuming each of the sensor information 296, 302, and 308, each data obtained in advance through tests of actual devices and / or computer simulations, etc., as data assuming each of the first corresponding information 280, the second corresponding information 282, the seventh corresponding information 292, and the eighth corresponding information 294, and each data obtained in advance through tests of actual devices and / or computer simulations, etc., as data assuming each of the fifth point information 268 and the sixth point information 270 can be cited.

[0558] Thus, in Figure 41In the example shown, the third correspondence information 284 and the fourth correspondence information 286 are not used, so the third tag information 256, the third point information 264, the fourth tag information 258, and the fourth point information 266 are not required. Accordingly, it is not necessary to process the third low FR camera image 212A1, the third high FR camera image 212B1, the fourth low FR camera image 214A1, the fourth high FR camera image 214B1, the third radar signal 212C1, and the fourth radar signal 214C1, so the third IPU 232B, the third MoPU 232F, the fourth IPU 232C, and the fourth MoPU 232G are not required. In addition, since the third low FR camera image 212A1, the third high FR camera image 212B1, the fourth low FR camera image 214A1, the fourth high FR camera image 214B1, the third radar signal 212C1, and the fourth radar signal 214C1 are not required, the third low FR camera 212A, the third high FR camera 212B, the fourth low FR camera 214A, the fourth high FR camera 214B, the third radar 212C, and the fourth radar 214C are not required. As a result, the processing load imposed on the second information processing device 206B can be reduced. In addition, the number of components mounted on the intermediate vehicle 204B can be reduced, which can contribute to cost reduction.

[0559] Although in Figure 41 the example shown, an example of a manner of not using the third correspondence information 284 and the fourth correspondence information 286 is shown, this is merely an example, and the third correspondence information 284 or the fourth correspondence information 286 may be used. In addition, the third tag information 256 or the third point information 264 may be used instead of the third correspondence information 284, and the fourth tag information 258 or the fourth point information 266 may be used instead of the fourth correspondence information 286.

[0560] Although in as Figure 41 the example shown, an example of a manner of deriving the first control variable 298, the second control variable 304, and the third control variable 310 based on the sensor information 296, 302, and 308, the first correspondence information 280, the second correspondence information 282, the seventh correspondence information 292, and the eighth correspondence information 294, and the fifth point information 268 and the sixth point information 270 is shown, the technology of the present disclosure is not limited thereto. For example, as Figure 42 shown, the first control variable 298, the second control variable 304, and the third control variable 310 may be derived based on the sensor information 296, 302, and 308, the first correspondence information 280, the eighth correspondence information 294, and the fifth point information 268 and the sixth point information 270.

[0561] In this case, the first control variable 298, the second control variable 304, and the third control variable 310 are derived by using the deep learning model 320. The deep learning model 320 is a pre-trained model obtained by performing deep learning on a neural network using teacher data. As an example of the teacher data used herein, a data set corresponding to example problem data and correct answer data assuming the first control variable 298, the second control variable 304, and the third control variable 310 can be cited. As an example of the example problem data, each data obtained in advance through tests of actual devices and / or computer simulations, etc., as data assuming each of the sensor information 296, 302, and 308, each data obtained in advance through tests of actual devices and / or computer simulations, etc., as data assuming each of the first corresponding information 280 and the eighth corresponding information 294, and each data obtained in advance through tests of actual devices and / or computer simulations, etc., as data assuming each of the fifth point information 268 and the sixth point information 270 can be cited.

[0562] Thus, in Figure 42 In the example shown, the second corresponding information 282 and the seventh corresponding information 292 are not used, so the second label information 250, the second point information 254, the seventh label information 272, and the seventh point information 276 are not required. Accordingly, the second low FR camera image 210A1, the second high FR camera image 210B1, the second radar signal 210C1, the seventh low FR camera image 220A1, the seventh high FR camera image 220B1, and the seventh radar signal 220C1 are not required, so the second IPU 224C, the second MoPU 224E, the seventh IPU 240B, and the seventh MoPU 240D are not required. The second low FR camera image 210A1, the second high FR camera image 210B1, the second radar signal 210C1, the seventh low FR camera image 220A1, the seventh high FR camera image 220B1, and the seventh radar signal 220C1 are not required, so the second low FR camera 210A, the second high FR camera 210B, the second radar 210C, the seventh low FR camera 220A, the seventh high FR camera 220B, and the seventh radar 220C are not required. Thereby, the processing load imposed on the first information processing device 206B and the third information processing device 206C can be reduced. In addition, the number of components mounted on the leading vehicle 204A and the trailing vehicle 204C can be reduced, which can contribute to cost reduction.

[0563] In addition, although in Figure 42In the example shown, exemplary embodiments are listed in which the second corresponding information 282 and the seventh corresponding information 292 are not used, but the second corresponding information 282 or the seventh corresponding information 292 may also be used. Additionally, the second tag information 250 or the second point information 254 may be used instead of the second corresponding information 282. Additionally, the seventh tag information 272 or the seventh point information 276 may be used instead of the seventh corresponding information 292.

[0564] Although in the above-described exemplary embodiments, the first corresponding information 280, the second corresponding information 282, the third corresponding information 284, the fourth corresponding information 286, the fifth corresponding information 288, the sixth corresponding information 290, the seventh corresponding information 292, and the eighth corresponding information 294 are illustrated, the technology of the present disclosure is not limited thereto. For example, even if the first tag information 248 or the first point information 252 is used instead of the first corresponding information 280, the technology of the present disclosure can be implemented. Additionally, even if the second tag information 250 or the second point information 254 is used instead of the second corresponding information 282, the technology of the present disclosure can be implemented. Additionally, even if the third tag information 256 or the third point information 264 is used instead of the third corresponding information 284, the technology of the present disclosure can be implemented. Additionally, even if the fourth tag information 258 or the fourth point information 266 is used instead of the fourth corresponding information 286, the technology of the present disclosure can be implemented. Additionally, even if the fifth tag information 260 or the fifth point information 268 is used instead of the fifth corresponding information 288, the technology of the present disclosure can be implemented. Additionally, even if the sixth tag information 262 or the sixth point information 270 is used instead of the sixth corresponding information 290, the technology of the present disclosure can be implemented. Additionally, even if the seventh tag information 272 or the seventh point information 276 is used instead of the seventh corresponding information 292, the technology of the present disclosure can be implemented. Additionally, even if the eighth tag information 274 or the eighth point information 278 is used instead of the eighth corresponding information 294, the technology of the present disclosure can be implemented.

[0565] Although one intermediate vehicle 204B is shown in the above-described exemplary embodiments, the technology of the present disclosure is not limited thereto, and there may be a plurality of intermediate vehicles 204B. In this case, the fifth aspect sensor 216 and the sixth aspect sensor 218 are mounted on at least one of the plurality of intermediate vehicles 204B. For example, the fifth aspect sensor 216 and the sixth aspect sensor 218 are mounted on at least one intermediate vehicle 204B located at the center of the queue 200. The vehicle 206 on which the fifth aspect sensor 216 and the sixth aspect sensor 218 are mounted may be a number of vehicles 206 that is less than the number of all the vehicles 206 constituting the queue 200. The vehicle 206 on which the fifth aspect sensor 216 and the sixth aspect sensor 218 are mounted is preferably the intermediate vehicle 204B.

[0566] Although in the above-described exemplary embodiments, the queue 200 is composed of the leading vehicle 204A, the intermediate vehicle 204B, and the trailing vehicle 204C, the intermediate vehicle 204B may be absent. In this case, it is sufficient to mount the fifth aspect sensor 216 and the sixth aspect sensor 218 on the leading vehicle 204A or the trailing vehicle 204C, and the processing of the information obtained by the fifth aspect sensor 216 and the sixth aspect sensor 218 may be executed by the first information processing device 206A or the second information processing device 206C.

[0567] Although in the above-described exemplary embodiments, the leading vehicle processor 224 performs the leading vehicle control process, at least one processor other than the leading vehicle processor 224 may perform the leading vehicle control process, or the leading vehicle processor 224 and at least one processor other than the leading vehicle processor 224 may perform the leading vehicle control process distributively. Further, although in the above-described exemplary embodiments, the intermediate vehicle processor 232 performs the intermediate vehicle control process, at least one processor other than the leading vehicle processor 224 may perform the intermediate vehicle control process, or the intermediate vehicle processor 232 and at least one processor other than the intermediate vehicle processor 232 may perform the intermediate vehicle control process distributively. Further, although in the above-described exemplary embodiments, the trailing vehicle processor 240 performs the trailing vehicle control process, at least one processor other than the trailing vehicle processor 240 may perform the trailing vehicle control process, or the trailing vehicle processor 240 and at least one processor other than the trailing vehicle processor 240 may perform the trailing vehicle control process distributively.

[0568] Although in the above-described exemplary embodiments, the case where the information processing device 10 is mounted on the queue 200 has been described, the technology of the present disclosure is not limited thereto. At least a part of the information processing device 10 may be an external device (e.g., a server) provided at a position outside the queue 200, and at least a part of the leading vehicle control process, the intermediate vehicle control process, and the trailing vehicle control process may be performed by the external device.

[0569] Although the vehicle 204 has been illustrated in the above-described exemplary embodiments, this is merely an example, and the technology of the present disclosure can be applied to a moving body other than the vehicle 204. Examples of the moving body other than the vehicle 204 include an airplane, a ship, or a walking robot (e.g., a walking robot for carrying goods or the like, or a walking robot for cleaning or the like).

[0570] (Twelfth Embodiment)

[0571] Next, regarding the twelfth embodiment, a description will be given while omitting or simplifying parts that overlap with the above-described embodiments. The information processing apparatus according to the twelfth embodiment can accurately obtain an index value required for driving control based on a large amount of information associated with the control of a vehicle. Therefore, at least a part of the information processing apparatus of the present disclosure can be mounted on a vehicle to achieve vehicle control.

[0572] As Figure 2 shown, the information processing apparatus 10 includes an IPU (Image Processing Unit) 11, a MoPU (Motion Processing Unit) 12, a central brain 15, and a memory 16. The central brain 15 is configured to include a GNPU (Graphics Neural Network Processing Unit) 13 and a CPU (Central Processing Unit) 14.

[0573] The IPU 11 can be built into a super high-resolution camera (not shown) provided in the vehicle. The IPU 11 performs predetermined image processing such as Bayer conversion, demosaicing, denoising, and sharpening on an object image existing around the vehicle, and outputs the processed object image at a frame rate of, for example, 10 frames per second and a resolution of 12 million pixels. The image output from the IPU 11 is supplied to the central brain 15 and the memory 16. The IPU 11 is an example of the "second processor" in the technology of the present disclosure.

[0574] The MoPU 12 can be built into a camera different from the super high-resolution camera provided in the vehicle. The MoPU 12 outputs action information indicating the action of an object captured from an image of the object captured at a frame rate of 1000 frames per second or more, for example, at a frame rate of 1000 frames per second or more. That is, the frame rate of the output of the MoPU 12 is 100 times the frame rate of the output of the IPU 11. The MoPU 12 outputs vector information of the action along a predetermined coordinate axis of a point indicating the existence position of the object as the action information. That is, the action information output from the MoPU 12 does not include information required to identify what the captured object is (for example, a person or an obstacle), but only includes information indicating the action (moving direction and moving speed) of the center point (or center of gravity point) of the object on the coordinate axes (x-axis, y-axis, z-axis). The information output from the MoPU 12 is supplied to the central brain 15 and the memory 16. By making the action information not include image information, the amount of information transmitted to the central brain 15 and the memory 16 can be significantly reduced. The MoPU 12 is an example of the "first processor" in the technology of the present disclosure.

[0575] The central brain 15 performs driving control of the vehicle based on the images output from the IPU 11 and the motion information output from the MoPU 12 as response control for the object. For example, the central brain 15 identifies objects (people, animals, roads, signals, signs, crosswalks, obstacles, buildings, etc.) existing around the vehicle based on the images output from the IPU 11. In addition, the central brain 15 identifies the actions of the objects whose identities have been recognized and that exist around the vehicle based on the motion information output from the MoPU 12. Based on the recognized information, the central brain 15 controls, for example, the motors of the drive wheels (speed control), braking control, and steering control. In the central brain 15, the GNPU 13 can be responsible for the processing related to image recognition, and the CPU 14 can be responsible for the processing related to vehicle control. The central brain 15 is an example of the "third processor" in the technology of the present disclosure.

[0576] Generally, ultra-high-resolution cameras are used for image recognition in autonomous driving. From the images captured by high-resolution cameras, it is possible to identify what the objects contained in the images are. However, in autonomous driving in the Level 6 era, this alone is not enough. In the Level 6 era, it is also necessary to identify the actions of objects with higher accuracy. By more accurately identifying the actions of objects through the MoPU 12, it is possible to more accurately implement, for example, the avoidance action of a vehicle traveling through autonomous driving to avoid obstacles. However, a high-resolution camera can only acquire approximately 10 frames of images per second, and the accuracy of analyzing the actions of objects is lower compared to the camera equipped with the MoPU 12. On the other hand, the camera equipped with the MoPU 12 can output at a high frame rate of, for example, 1000 frames per second.

[0577] Therefore, in the technology of the present disclosure, two independent processors, the IPU 11 and the MoPU 12, are used. The high-resolution camera (IPU 11) is made to have the function of acquiring the image information required to identify what the captured objects are, and the MoPU 12 is made to have the function of detecting the actions of objects. The MoPU 12 captures the object as a point and analyzes in which direction and at what speed the coordinates of this point move along the x-axis, y-axis, and z-axis. Since the overall outline of the object and what the object is can be detected from the images from the ultra-high-resolution camera, through the MoPU 12, for example, as long as it is known how the center point of the object moves, the overall behavior of the object can be known.

[0578] According to the means of analyzing only the movement and speed of the center point of the object, compared with judging how the entire image of the object moves, the amount of information transmitted to the central brain 15 can be significantly reduced, and the amount of calculation in the central brain 15 can be significantly reduced. For example, in the case of sending an image of 1000 pixels × 1000 pixels to the central brain 15 at a frame rate of 1000 frames per second, if color information is included, 4 billion bits per second of data will be sent to the central brain 15. The MoPU 12 can compress the amount of data transmitted to the central brain 15 to 20,000 bits per second by only sending the motion information indicating the action of the center point of the object. That is, the amount of data transmitted to the central brain 15 is compressed to 1 / 200,000.

[0579] In this way, by combinatorially using the low frame rate and high-resolution images output from the IPU 11 and the high frame rate and lightweight motion information output from the MoPU 12, object recognition including object motion can be achieved with a small amount of data.

[0580] In addition, in the case of using one MoPU 12, vector information of the motion of a point indicating the existence position of the object along each of two coordinate axes (x-axis and y-axis) in the three-dimensional orthogonal coordinate system can be obtained. The principle of a stereo camera can be utilized, and two MoPU 12s can be used to output vector information of the motion of a point indicating the existence position of the object along each of the three coordinate axes (x-axis, y-axis, and z-axis) in the three-dimensional orthogonal coordinate system. The z-axis is the axis along the depth direction (vehicle driving).

[0581] In addition, as Figure 43 shown, the images from the camera installed on the left side of the vehicle and the images from the camera installed on the right side of the vehicle can be input into the core 17A of the MoPU 12. Each of these images is, for example, an image including color information of 1000 pixels × 1000 pixels and can be input into the core 17A at a frame rate of 1000 frames per second. The core 17A of the MoPU 12 can transmit vector information of the motion along each of the three coordinate axes (x-axis, y-axis, z-axis) in the three-dimensional orthogonal coordinate system to the central brain 15 at a frame rate of 1000 frames per second based on these images. In addition, as Figure 44 shown, separate cores 17A1 and 17A2 can be used to separately process the images from the camera installed on the left side of the vehicle and the images from the camera installed on the right side of the vehicle.

[0582] In addition, in the above description, a method of outputting motion information in which the MoPU 12 outputs an action indicating the center point of an object is illustrated. However, the technology of the present disclosure is not limited to this method. The MoPU 12 may output motion information for at least two diagonal coordinate points among the vertices of a quadrilateral that encloses the contour of an object recognized from an image captured by a camera. In Figure 5 , a method is illustrated in which the MoPU 12 sets bounding boxes 21, 22, 23, and 24 that enclose the contour of each of the four objects included in the image, and outputs motion information for two diagonal coordinate points among the vertices of the bounding boxes 21, 22, 23, and 24. In this way, the MoPU 12 can capture an object as an object having a certain size rather than a point. When capturing an object as an object having a certain size, it is not necessary to extract only at least two diagonal coordinate points among the vertices of a quadrilateral that encloses the contour of an object recognized from an image captured by a camera, but a plurality of coordinate points including the contour can be extracted.

[0583] In addition, as Figure 15 shown, the MoPU 12 may output motion information based on at least one of a visible light image and an infrared image. The visible light image is an image captured by a visible light camera, and the infrared image is an image captured by an infrared camera. The visible light image and the infrared image are each input to the core 17A at a frame rate of 1000 frames per second or more. Preferably, the visible light image and the infrared image are synchronized with each other. By using the infrared image in object detection performed by the MoPU 12, object detection can be performed even in cases where it is difficult to perform object detection using the visible light image, such as at night. Among the visible light image and the infrared image, the MoPU 12 may output motion information based only on the infrared image, or may output motion information based on both the visible light image and the infrared image.

[0584] In addition, as Figure 46 shown, the MoPU 12 may output motion information based on an image and a radar signal. The radar signal is a signal of a reflected wave from an object based on an electromagnetic wave irradiated onto the object. The MoPU 12 may derive the distance to the object based on the image and the radar signal, and output vector information indicating the motion of a point indicating the existence position of the object along each of the three axes in a three-dimensional orthogonal coordinate system as motion information. The image may include at least one of a visible light image and an infrared image. The image and the radar signal are input to the core 17A at a frame rate of 1000 frames per second or more.

[0585] In addition, although in the above description, an example is given where the central brain 15 performs driving control of a vehicle based on the images output from the IPU 11 and the motion information output from the MoPU 12, the technology of the disclosed content is not limited to this aspect. The central brain 15 can perform motion control of a robot based on the images output from the IPU 11 and the motion information output from the MoPU 12 as a response control to the object. The robot can be, for example, a humanoid intelligent robot that performs operations in place of a human. For example, the central brain 15 can perform motion control of the robot's arms, palms, fingers, and feet based on the images output from the IPU 11 and the motion information output from the MoPU 12, and perform motion control such as grasping, holding, hugging, carrying on the back, moving, transporting, throwing, kicking, and avoiding the object. The IPU 11 and the MoPU 12 can be mounted, fo...

Claims

1. An information processing apparatus, comprising: a first processor that outputs point information capturing the object to be imaged as points from an image of the object captured by a first camera; a second processor that outputs identification information obtained by identifying the object captured from an image of the object captured by a second camera facing a direction corresponding to that of the first camera; and a third processor that correlates the point information output from the first processor with the identification information output from the second processor, wherein a frame rate of the first camera is higher than that of the second camera.

2. The information processing apparatus according to claim 1, wherein the frame rate of the first camera is 10 times or more that of the second camera.

3. The information processing apparatus according to claim 2, wherein the frame rate of the first camera is not less than 100 frames per second and the frame rate of the second camera is 10 frames per second.

4. An information processing method, comprising causing a computer to perform the following processes: outputting point information capturing the object to be imaged as points from an image of the object captured by a first camera; outputting identification information obtained by identifying the object captured from an image of the object captured by a second camera having a frame rate lower than that of the first camera and facing a direction corresponding to that of the first camera; and correlating the point information with the identification information.

5. An information processing program for causing a computer to perform the following processes: outputting point information capturing the object to be imaged as points from an image of the object captured by a first camera; outputting identification information obtained by identifying the object captured from an image of the object captured by a second camera having a frame rate lower than that of the first camera and facing a direction corresponding to that of the first camera; and correlating the point information with the identification information.

6. An information processing apparatus, comprising: a first processor that outputs point information capturing the object to be imaged as points from an image of the object captured by a first camera; a second processor that outputs identification information obtained by identifying the object captured from an image of the object captured by a second camera facing a direction corresponding to that of the first camera; and a third processor that correlates the point information output from the first processor with the identification information output from the second processor, wherein the first processor calculates a danger level related to the movement of a specified moving body as a score related to the external environment based on detection information detected by a detection unit and the point information.

7. The information processing apparatus according to claim 6, wherein the frame rate of the first camera is variable, and the first processor changes the frame rate of the first camera according to the calculated danger level.

8. The information processing apparatus according to claim 6, wherein the danger level indicates the degree of how dangerous a place the moving body will travel in the future.

9. An information processing apparatus includes: A first processor that outputs point information capturing the object to be photographed as points from an image of the object photographed by a first camera; A second processor that outputs identification information obtained by identifying the photographed object from an image of the object photographed by a second camera facing a direction corresponding to the first camera; And A third processor that correlates the point information output from the first processor with the identification information output from the second processor, The third processor calculates a risk related to the movement of a specified moving body as a score related to the external environment for the specified moving body based on detection information detected by a detection unit and the point information.

10. The information processing apparatus according to claim 9, Wherein, The frame rate of the first camera is variable, The third processor outputs an instruction to change the frame rate of the first camera to the first processor according to the calculated risk.

11. An information processing method causes a computer to perform the following processing: Output point information capturing the object to be photographed as points from an image of the object photographed by a first camera; Output identification information obtained by identifying the photographed object from an image of the object photographed by a second camera facing a direction corresponding to the first camera; Correlate the point information with the identification information; And Calculate a risk related to the movement of a specified moving body as a score related to the external environment for the specified moving body based on detection information detected by a detection unit and the point information.

12. An information processing program causes a computer to perform the following processing: Output point information capturing the object to be photographed as points from an image of the object photographed by a first camera; Output identification information obtained by identifying the photographed object from an image of the object photographed by a second camera facing a direction corresponding to the first camera; Correlate the point information with the identification information; And Calculate a risk related to the movement of a specified moving body as a score related to the external environment for the specified moving body based on detection information detected by a detection unit and the point information.

13. An information processing apparatus includes: A first processor that outputs point information capturing the object to be photographed as points from an image of the object photographed by a first camera capable of changing the frame rate; and A second processor that outputs identification information obtained by identifying the photographed object from an image of the object photographed by a second camera facing a direction corresponding to the first camera, The first processor changes the frame rate of the first camera according to the category of the object based on the identification information.

14. The information processing apparatus according to claim 13, Wherein, When the object is an agile object, the first processor increases the frame rate. When the object is a sluggish object or a stationary object, the first processor decreases the frame rate.

15. The information processing apparatus according to claim 13, wherein, the first processor also changes the frame rate of the first camera according to the number of the objects.

16. The information processing apparatus according to claim 15, wherein, the first processor makes the frame rate higher as the number of the objects is larger, and makes the frame rate lower as the number of the objects is smaller.

17. The information processing apparatus according to claim 13, wherein, the first processor calculates a score related to the external environment according to the category of the object, and changes the frame rate according to the score related to the external environment.

18. The information processing apparatus according to claim 15 or 16, wherein, the first processor calculates a score related to the external environment according to the category and the number of the objects, and changes the frame rate according to the score related to the external environment.

19. The information processing apparatus according to claim 13, wherein, the first processor extracts points indicating the existence positions of the objects from the images captured by the first camera, and outputs the points indicating the existence positions of the objects.

20. The information processing apparatus according to claim 13, comprising: a third processor that makes the point information output from the first processor correspond to the identification information output from the second processor.

21. An information processing method, wherein, point information that captures the object to be photographed as points is output from an image of the object captured by a first camera capable of changing the frame rate; identification information obtained by identifying the object captured by a second camera facing the direction corresponding to the first camera is output from an image of the object captured by the second camera; and the frame rate of the first camera is changed according to the category of the object based on the identification information.

22. An information processing program for causing a computer to execute the following processing: Point information that captures the object to be photographed as points is output from an image of the object captured by a first camera capable of changing the frame rate; Identification information obtained by identifying the object captured by a second camera facing the direction corresponding to the first camera is output from an image of the object captured by the second camera; and the frame rate of the first camera is changed according to the category of the object based on the identification information.

23. An information processing apparatus, comprising: a first processor that outputs point information that captures the object to be photographed as points from an image of the object captured by a first camera capable of changing the frame rate; a second processor that outputs identification information obtained by identifying the object captured by a second camera facing the direction corresponding to the first camera from an image of the object captured by the second camera; and A third processor that causes the point information output from the first processor to correspond to the identification information output from the second processor. The first processor derives, as the point information, coordinate values in the depth direction of the object in a three-dimensional rectangular coordinate system of a point indicating the existence position of the object from an image of the object captured by the first camera. Change the frame rate of the first camera according to the coordinate values in the depth direction.

24. The information processing apparatus according to claim 23, wherein, The first processor derives the coordinate values in the depth direction as the point information from images of the object captured by a plurality of the first cameras.

25. The information processing apparatus according to claim 23 or 24, wherein, The first processor derives coordinate values in the width direction, height direction, and depth direction of the object as the point information from an image of the object captured by the first camera and a radar signal from a reflected wave of the object based on electromagnetic waves irradiated onto the object by a radar.

26. The information processing apparatus according to claim 23 or 24, wherein, The first processor derives coordinate values in the width direction, height direction, and depth direction of the object as the point information from an image of the object captured by the first camera and a result obtained by photographing structured light irradiated onto the object by an irradiating device.

27. The information processing apparatus according to claim 23 or 24, wherein, The first processor derives, as the point information, coordinate values in the depth direction at a second time point based on coordinate values in the width direction, height direction, and depth direction of the object in the three-dimensional rectangular coordinate system at a first time point and coordinate values in the width direction and height direction at a second time point, which is the next time point after the first time point.

28. An information processing method, wherein, Output point information that captures the object to be photographed as a point from an image of the object captured by a first camera capable of changing the frame rate, Output identification information obtained by identifying the object captured from an image of the object captured by a second camera facing a direction corresponding to the first camera, Make the point information correspond to the identification information, Derive, as the point information, coordinate values in the depth direction of the object in a three-dimensional rectangular coordinate system of a point indicating the existence position of the object from an image of the object captured by the first camera, Change the frame rate of the first camera according to the coordinate values in the depth direction.

29. An information processing program for causing a computer to execute the following processing: Output point information that captures the object to be photographed as a point from an image of the object captured by a first camera capable of changing the frame rate; Output recognition information obtained by recognizing the object captured in the image of the object captured by the second camera facing the direction corresponding to the first camera; Correlate the point information with the recognition information; Derive, from the image of the object captured by the first camera, the coordinate value in the depth direction of the object in a three-dimensional rectangular coordinate system of a point indicating the position where the object exists as the point information; And Change the frame rate of the first camera according to the coordinate value in the depth direction.

30. An information processing device mounted on a vehicle, comprising: A first processor that outputs point information obtained by capturing the object captured in the image of the object captured by a first camera capable of changing the frame rate as a point, The first processor changes the frame rate of the first camera according to the position of the vehicle.

31. The information processing device according to claim 30, wherein, The first processor calculates a score related to the external environment according to the position of the vehicle, and changes the frame rate according to the score related to the external environment.

32. The information processing device according to claim 30 or 31, wherein, The first processor extracts a point indicating the position where the object exists from the image captured by the first camera, and outputs the point indicating the position where the object exists.

33. The information processing device according to claim 30 or 31, wherein, The first processor changes the frame rate of the first camera according to the category of the position of the vehicle.

34. The information processing device according to claim 30 or 31, further comprising: A second processor that outputs recognition information obtained by recognizing the object captured in the image of the object captured by a second camera facing the direction corresponding to the first camera; and A third processor that correlates the point information output from the first processor with the recognition information output from the second processor.

35. An information processing method in an information processing device mounted on a vehicle, wherein, Output point information obtained by capturing the object captured in the image of the object captured by a first camera capable of changing the frame rate as a point, Change the frame rate of the first camera according to the position of the vehicle.

36. An information processing program for executing the information processing method in an information processing device mounted on a vehicle, for causing a computer to execute the following processing: Output point information obtained by capturing the object captured in the image of the object captured by a first camera capable of changing the frame rate as a point, Change the frame rate of the first camera according to the position of the vehicle.

37. An information processing device, comprising: A first processor that outputs point information obtained by capturing the object captured in the image of the object captured by a first camera as a point; and A second processor that outputs identification information obtained by identifying the object captured in the image of the object captured by a second camera facing the direction corresponding to the first camera. The frame rate of the first camera is variable. The first processor changes the frame rate of the first camera according to the position information.

38. The information processing apparatus according to claim 37, comprising: A third processor that correlates the point information output from the first processor with the identification information output from the second processor.

39. The information processing apparatus according to claim 37, wherein, The first processor generates a heat map based on the frequency of previously detecting the object at each position around the first camera.

40. The information processing apparatus according to claim 39, wherein, The first processor changes the frame rate of the first camera based on the position information and the heat map.

41. An information processing method that causes a computer to perform the following processing: Changing the frame rate of a first camera according to position information; Outputting point information that captures the object captured by the first camera as a point from the image of the object; and Outputting identification information obtained by identifying the object captured in the image of the object captured by a second camera facing the direction corresponding to the first camera.

42. An information processing program for causing a computer to perform the following processing: Changing the frame rate of a first camera according to position information; Outputting point information that captures the object captured by the first camera as a point from the image of the object; and Outputting identification information obtained by identifying the object captured in the image of the object captured by a second camera facing the direction corresponding to the first camera.

43. An information processing apparatus, comprising: A first processor that outputs point information that captures the object captured by a first camera as a point from the image of the object; and A second processor that outputs identification information obtained by identifying the object captured in the image of the object captured by a second camera facing the direction corresponding to the first camera, The frame rate of the first camera is variable, The first processor changes the frame rate of the first camera based on the information of the user obtained from the user.

44. The information processing apparatus according to claim 43, comprising: A third processor that correlates the point information output from the first processor with the identification information output from the second processor.

45. The information processing apparatus according to claim 43, The information of the user includes: At least one of voice information from the user, image information of the user captured, and heart rate information of the user.

46. The information processing apparatus according to claim 43, wherein, The user is an occupant of a vehicle in which at least a part of the information processing apparatus is mounted.

47. An information processing method, which causes a computer to perform the following processing: Based on the information of the user obtained from the user, change the frame rate of the first camera; Output point information that captures the object to be photographed as a point from the image of the object photographed by the first camera; And Output recognition information obtained by recognizing the object photographed from the image of the object photographed by the second camera facing the direction corresponding to the first camera.

48. An information processing program for causing a computer to perform the following processing: Based on the information of the user obtained from the user, change the frame rate of the first camera; Output point information that captures the object to be photographed as a point from the image of the object photographed by the first camera; And Output recognition information obtained by recognizing the object photographed from the image of the object photographed by the second camera facing the direction corresponding to the first camera.

49. An information processing apparatus, comprising: A processor, The processor Identifies the situation in the front based on a front image obtained by photographing the front with a front camera provided by a leading moving body among a plurality of moving bodies moving in a queue and capable of photographing the front of the queue; Identifies the situation in the rear based on a rear image obtained by photographing the rear with a rear camera provided by a trailing moving body among the plurality of moving bodies and capable of photographing the rear of the queue; Identifies the situation in the side based on a side image obtained by photographing the side with a side camera provided by a specific moving body among the plurality of moving bodies and capable of photographing the side of the queue, where the specific moving body is a moving body fewer than the number of the plurality of moving bodies.

50. The information processing apparatus according to claim 49, Wherein, The side camera photographs the side at a first frame rate, and the first frame rate is higher than the frame rates of the front camera and the rear camera.

51. The information processing apparatus according to claim 50, Wherein, Whenever a side image is obtained by photographing the side at the first frame rate, the processor identifies the situation in the side based on the obtained side image.

52. The information processing apparatus according to claim 49, Wherein, The plurality of moving bodies are three or more moving bodies, The specific moving body is an intermediate moving body located between the leading moving body and the trailing moving body.

53. The information processing apparatus according to claim 52, Wherein, Each of the plurality of moving bodies is a moving body capable of autonomous driving, In the intermediate moving body, at least one of a leading-side camera capable of photographing the leading moving body side and a trailing-side camera capable of photographing the trailing moving body side is provided. The processor controls the autonomous driving for the middle moving body based on at least one of a leading moving body side image obtained by photographing the leading moving body side with the leading side camera and a trailing moving body side image obtained by photographing the trailing moving body side with the trailing side camera. A second frame rate that is the frame rate of the leading side camera and a third frame rate that is the frame rate of the trailing side camera are lower than the frame rate of the front camera and the frame rate of the rear camera.

54. The information processing apparatus according to claim 52, wherein, each of the plurality of moving bodies is a moving body capable of autonomous driving, the processor controls the autonomous driving for the middle moving body without using at least one of a leading moving body side image obtained by photographing the leading moving body side from the middle moving body side and a trailing moving body side image obtained by photographing the trailing moving body side from the middle moving body side.

55. The information processing apparatus according to claim 49, wherein, the processor identifies the situation in the front by identifying the type of a front object existing in the front based on the front image, identifies the situation in the rear by identifying the type of a rear object existing in the rear based on the rear image.

56. The information processing apparatus according to claim 49, wherein, the processor identifies the situation in the side by recognizing a side object existing in the side as a point based on the side image.

57. The information processing apparatus according to claim 49, wherein, each of the plurality of moving bodies is a moving body capable of autonomous driving, the processor controls the autonomous driving based on the situation in the front, the situation in the rear, and the situation in the side.

58. The information processing apparatus according to claim 49, wherein, each of the plurality of moving bodies is a moving body capable of autonomous driving, the processor obtains front object information capable of determining the type of a front object by identifying the type of a front object existing in the front based on the front image, obtains rear object information capable of determining the type of a rear object by identifying the type of a rear object existing in the rear based on the rear image, controls the autonomous driving based on front correspondence information and rear correspondence information, the front correspondence information is information that makes front point information representing the front object correspond to the front object information based on a first image obtained by photographing the front at a fourth frame rate higher than the frame rate of the front camera, the rear correspondence information is information that makes rear point information representing the rear object correspond to the rear object information based on a second image obtained by photographing the rear at a fifth frame rate higher than the frame rate of the rear camera.

59. The information processing apparatus according to claim 58, wherein, the processor Based on the side image, lateral point information representing the lateral object as points is obtained by recognizing lateral objects existing on the side as points. Based on the front correspondence information, the rear correspondence information, and the lateral point information, the autonomous driving is controlled.

60. The information processing device according to claim 49, wherein, the processor includes a front recognition processor, a rear recognition processor, and a lateral recognition processor, the front recognition processor recognizes the situation in the front based on the front image, the rear recognition processor recognizes the situation in the rear based on the rear image, the lateral recognition processor recognizes the situation in the side based on the side image.

61. The information processing device according to claim 60, wherein, the lateral recognition processor recognizes the situation in the side based on the side image by performing processing at a higher speed than the front recognition processor and the rear recognition processor.

62. An information processing method, comprising: recognizing the situation in the front based on a front image obtained by photographing the front with a front camera provided by a leading moving body among a plurality of moving bodies moving in a queue and capable of photographing the front of the queue; recognizing the situation in the rear based on a rear image obtained by photographing the rear with a rear camera provided by a trailing moving body among the plurality of moving bodies and capable of photographing the rear of the queue; and recognizing the situation in the side based on a side image obtained by photographing the side with a side camera provided by a specific moving body among the plurality of moving bodies and capable of photographing the side of the queue, the specific moving body being a moving body fewer in number than the plurality of moving bodies.

63. An information processing program for causing a computer to execute processing including the following steps: recognizing the situation in the front based on a front image obtained by photographing the front with a front camera provided by a leading moving body among a plurality of moving bodies moving in a queue and capable of photographing the front of the queue; recognizing the situation in the rear based on a rear image obtained by photographing the rear with a rear camera provided by a trailing moving body among the plurality of moving bodies and capable of photographing the rear of the queue; and recognizing the situation in the side based on a side image obtained by photographing the side with a side camera provided by a specific moving body among the plurality of moving bodies and capable of photographing the side of the queue, the specific moving body being a moving body fewer in number than the plurality of moving bodies.

64. An information processing device, comprising: a camera capable of changing the frame rate; and a processor, the processor detects an object reflected in an image photographed by the camera; controls in such a manner as to change the frame rate of the camera according to at least one of the number of the detected objects, the acceleration of the objects, and the size of the objects.

65. The information processing apparatus according to claim 64, wherein, the processor when changing the frame rate according to the number of the objects, controls in such a manner that the higher the number of the objects, the higher the frame rate, and controls in such a manner that the lower the number of the objects, the lower the frame rate.

66. The information processing apparatus according to claim 64, wherein, the processor when changing the frame rate according to the acceleration of the objects, controls in such a manner that the greater the acceleration of the objects, the higher the frame rate, and controls in such a manner that the smaller the acceleration of the objects, the lower the frame rate.

67. The information processing apparatus according to claim 64, wherein, the processor when changing the frame rate according to the size of the objects, controls in such a manner that the larger the size of the objects, the higher the frame rate, and controls in such a manner that the smaller the size of the objects, the lower the frame rate.

68. The information processing apparatus according to claim 64, wherein, the processor calculates a score related to the external environment based on at least one of the number of the objects, the acceleration of the objects, and the size of the objects, and controls in such a manner as to change the frame rate based on the score related to the external environment and a preset threshold value.

69. The information processing apparatus according to claim 64, wherein, the processor extracts points indicating the existence positions of the objects from the images captured by the camera, and outputs the points indicating the existence positions of the objects.

70. The information processing apparatus according to claim 64, wherein, the information processing apparatus uses two of the processors, and outputs, as the motion information, vector information of the motion of the points indicating the existence positions of the objects along each of three coordinate axes in a three-dimensional orthogonal coordinate system.

71. An information processing method executed by an information processing apparatus, the information processing apparatus including: a camera capable of changing the frame rate; and a processor, wherein the processor detects objects reflected in the images captured by the camera; and controls in such a manner as to change the frame rate of the camera based on at least one of the number of the detected objects, the acceleration of the objects, and the size of the objects.

72. An information processing program for causing the processor of an information processing apparatus to execute, the information processing apparatus including: a camera capable of changing the frame rate; and a processor, wherein the processor detects objects reflected in the images captured by the camera; and controls in such a manner as to change the frame rate of the camera based on at least one of the number of the detected objects, the acceleration of the objects, and the size of the objects.

73. An information processing apparatus including: a camera capable of changing the frame rate; and a processor, wherein the processor detects objects reflected in the images captured by the camera at each moment. Control is performed in such a manner that the frame rate of the camera is changed based on at least one of a time series of the number of the detected objects, a time series of the acceleration of the objects, and a time series of the size of the objects.

74. The information processing apparatus according to claim 73, wherein, the processor when changing the frame rate based on the time series of the number of the objects, controls in such a manner that the frame rate is increased when the number of the objects reflected in the image at the current moment is larger than the number of the objects reflected in the image at the previous moment, and controls in such a manner that the frame rate is decreased when the number of the objects reflected in the image at the current moment is smaller than the number of the objects reflected in the image at the previous moment.

75. The information processing apparatus according to claim 73, wherein, the processor when changing the frame rate based on the time series of the acceleration of the objects, controls in such a manner that the frame rate is increased when the acceleration of the objects reflected in the image at the current moment is larger than the acceleration of the objects reflected in the image at the previous moment, and controls in such a manner that the frame rate is decreased when the acceleration of the objects reflected in the image at the current moment is smaller than the acceleration of the objects reflected in the image at the previous moment.

76. The information processing apparatus according to claim 73, wherein, the processor when changing the frame rate based on the time series of the size of the objects, controls in such a manner that the frame rate is increased when the size of the objects reflected in the image at the current moment is larger than the size of the objects reflected in the image at the previous moment, and controls in such a manner that the frame rate is decreased when the size of the objects reflected in the image at the current moment is smaller than the size of the objects reflected in the image at the previous moment.

77. The information processing apparatus according to claim 73, wherein, the processor calculates a score related to the external environment based on at least one of the time series of the number of the objects, the time series of the acceleration of the objects, and the time series of the size of the objects, and controls in such a manner that the frame rate is changed based on the score related to the external environment.

78. The information processing apparatus according to claim 73, wherein, the processor extracts points indicating the existence positions of the objects from the images captured by the camera, and outputs the points indicating the existence positions of the objects.

79. The information processing apparatus according to claim 73, wherein, the information processing apparatus uses two of the processors, and outputs, as the motion information, vector information of the motion of the points indicating the existence positions of the objects along each of three coordinate axes in a three-dimensional orthogonal coordinate system.

80. An information processing method executed by an information processing apparatus, the information processing apparatus including: a camera capable of changing the frame rate; and a processor, the processor detects objects reflected in the images captured by the camera at respective moments; Control is performed in such a manner that the frame rate of the camera is changed based on at least one of the time series of the number of the detected objects, the time series of the acceleration of the objects, and the time series of the size of the objects.

81. An information processing program for causing a processor of an information processing apparatus to execute, the information processing apparatus including: a camera capable of changing the frame rate; and a processor, wherein the processor: detects objects reflected in an image captured by the camera at each moment; controls the frame rate of the camera in such a manner that the frame rate is changed based on at least one of the time series of the number of the detected objects, the time series of the acceleration of the objects, and the time series of the size of the objects.

Images