Information processing device, vehicle, information processing method, and information processing program

Abstract

An information processing device is provided with: a first processor that outputs, on the basis of an image of an object captured by a first camera, point information obtained by capturing the captured object as a point; and a second processor that outputs identification information obtained by identifying the captured object on the basis of an image of the object captured by a second camera oriented in a direction corresponding to the first camera.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, a vehicle, an information processing method, and an information processing program. Background Art

[0002] JP-A-2022-035198 describes a vehicle having an autonomous driving function. Summary of the Invention

[0003] Problems to be Solved by the Invention

[0004] In the case of an autonomous driving vehicle as described in JP-A-2022-035198, control of autonomous driving is performed by using a plurality of images obtained by photographing the surroundings of the vehicle with a camera. Therefore, in existing autonomous driving, there are the following problems: the amount of data acquired by the processor that controls autonomous driving increases, and the amount of calculation required for controlling autonomous driving increases.

[0005] Therefore, an object of the present disclosure is to provide an information processing apparatus, a vehicle, an information processing method, and an information processing program that can reduce the amount of data output to a specified output destination when outputting the photographed information of an object photographed by a camera to the specified output destination.

[0006] Means for Solving the Problems

[0007] The information processing apparatus according to the first aspect includes: a first processor that outputs point information obtained by capturing the photographed object as a point based on an image of the object photographed by a first camera; and a second processor that outputs identification information obtained by identifying the photographed object based on an image of the object photographed by a second camera facing a direction corresponding to the first camera.

[0008] The information processing apparatus according to the second aspect includes, based on the information processing apparatus according to the first aspect, a third processor that associates the point information output from the first processor with the identification information output from the second processor.

[0009] The information processing apparatus according to the third aspect includes, based on the information processing apparatus according to the first or second aspect, wherein a frame rate of the first camera is variable, and the first processor changes the frame rate of the first camera according to a specified factor.

[0010] The information processing apparatus according to the fourth aspect includes, based on the information processing apparatus according to the third aspect, wherein the first processor calculates a score related to an external environment for a specified object.

[0011] The information processing apparatus according to the fifth aspect is based on the information processing apparatus according to the fourth aspect. The first processor changes the frame rate of the first camera according to the calculated score related to the external environment.

[0012] The information processing apparatus according to the sixth aspect is based on the information processing apparatus according to any one of the first aspect to the fifth aspect. The first processor outputs coordinate values of at least two coordinate axes in a three-dimensional orthogonal coordinate system of a point indicating the existence position of the object captured by the first camera. The information processing apparatus includes a third processor, and the third processor associates the point information output from the first processor with the recognition information output from the second processor.

[0013] The information processing apparatus according to the seventh aspect is based on the information processing apparatus according to the sixth aspect. The first processor outputs coordinate values of at least two points that are diagonally opposite among the vertices of a polygon that encloses the contour of the object recognized from the image captured by the first camera.

[0014] The information processing apparatus according to the eighth aspect is based on the information processing apparatus according to the seventh aspect. The first processor outputs coordinate values of a plurality of vertices of a polygon that encloses the contour of the object recognized from the image captured by the first camera.

[0015] The information processing apparatus according to the ninth aspect is based on the information processing apparatus according to any one of the first aspect to the eighth aspect. It includes a third processor, and the third processor associates the point information output from the first processor with the recognition information output from the second processor, and controls the autonomous driving of the moving body based on the point information and the recognition information.

[0016] The information processing apparatus according to the tenth aspect is based on the information processing apparatus according to the ninth aspect. The third processor calculates a control variable for controlling the autonomous driving of the moving body based on the detection information detected by the detection unit, and controls the autonomous driving of the moving body based on the calculated control variable, the point information, and the recognition information.

[0017] The information processing apparatus according to the eleventh aspect is based on the information processing apparatus according to any one of the first aspect to the tenth aspect. The first processor outputs the point information based on at least one of a visible light image and an infrared image of the object captured by the first camera. The information processing apparatus includes a third processor, and the third processor associates the point information output from the first processor with the recognition information output from the second processor.

[0018] The information processing device according to the twelfth aspect is based on the information processing device according to the eleventh aspect. When the object cannot be captured from the visible light image of the object captured by the visible light camera included in the first camera due to a specified factor, the first processor outputs the point information based on the infrared image of the object captured by the infrared camera included in the first camera.

[0019] The information processing device according to the thirteenth aspect is based on the information processing device according to the twelfth aspect. The first processor synchronizes the timing of capturing the visible light image by the visible light camera with the timing of capturing the infrared image by the infrared camera.

[0020] The information processing device according to the fourteenth aspect is based on the information processing device according to any one of the first aspect to the thirteenth aspect. The first processor outputs the point information based on the image of the object captured by the first camera and the radar signal based on the electromagnetic wave reflected from the object irradiated by the radar. The information processing device includes a third processor, and the third processor associates the point information output from the first processor with the recognition information output from the second processor.

[0021] The information processing device according to the fifteenth aspect is based on the information processing device according to the fourteenth aspect. The first processor synchronizes the timing of capturing the image by the first camera with the timing of the radar acquiring the three-dimensional point cloud data of the object based on the radar signal.

[0022] The information processing device according to the sixteenth aspect is based on the information processing device according to the fourteenth aspect or the fifteenth aspect. The number of images per unit time captured by the first camera and the number of three-dimensional point cloud data per unit time acquired by the radar are more than the number of images per unit time captured by the second camera.

[0023] The information processing device according to the seventeenth aspect is based on the information processing device according to any one of the first aspect to the sixteenth aspect. The second processor outputs label information indicating the category of the captured object based on the image of the object captured by the second camera. The information processing device includes a third processor, and the third processor associates the point information output from the first processor with the label information output from the second processor.

[0024] The information processing device according to the eighteenth aspect is based on the information processing device according to the seventeenth aspect. The third processor associates the position information of the object indicated by the point information with the label information related to the object existing at the position indicated by the position information.

[0025] The information processing apparatus according to the nineteenth aspect is based on the information processing apparatus according to the eighteenth aspect. The third processor associates the point information output from the first processor at the same timing as the timing when the second processor outputs the tag information with the tag information.

[0026] The information processing apparatus according to the twentieth aspect is based on the information processing apparatus according to the eighteenth or nineteenth aspect. When new point information is output from the first processor after associating the point information with the tag information, the third processor also associates the new point information with the tag information.

[0027] The information processing apparatus according to the twenty - first aspect is based on the information processing apparatus according to any one of the first to twentieth aspects. It includes a third processor that associates the point information output from the first processor with the identification information output from the second processor. The first processor derives the coordinate value in the depth direction of the object in a three - dimensional orthogonal coordinate system of a point representing the existence position of the object from an image of the object captured by the first camera as the point information.

[0028] The information processing apparatus according to the twenty - second aspect is based on the information processing apparatus according to the twenty - first aspect. The first processor derives the coordinate value in the depth direction as the point information from images of the object captured by a plurality of the first cameras.

[0029] The information processing apparatus according to the twenty - third aspect is based on the information processing apparatus according to the twenty - first or twenty - second aspect. The first processor derives the coordinate values in the width direction, height direction, and depth direction of the object as the point information from an image of the object captured by the first camera and a radar signal based on an electromagnetic wave radiated by a radar and reflected from the object.

[0030] The information processing apparatus according to the twenty - fourth aspect is based on the information processing apparatus according to any one of the twenty - first to twenty - third aspects. The first processor derives the coordinate values in the width direction, height direction, and depth direction of the object as the point information from an image of the object captured by the first camera and a result of photographing structured light irradiated onto the object by an irradiating device.

[0031] The information processing apparatus according to the twenty-fifth aspect is the information processing apparatus according to any one of the twenty-first aspect to the twenty-fourth aspect. The first processor derives the coordinate value in the depth direction of the object in the three-dimensional orthogonal coordinate system at the second time point as the point information from the coordinate values in the width direction, height direction, and depth direction of the object in the three-dimensional orthogonal coordinate system at the first time point and the coordinate values in the width direction and height direction at the second time point. The second time point is the next time point after the first time point.

[0032] The information processing apparatus according to the twenty-sixth aspect is the information processing apparatus according to the first aspect, and includes a third processor. The third processor associates the point information output from the first processor with the recognition information output from the second processor. The third processor derives the coordinate value in the depth direction of the object in the three-dimensional orthogonal coordinate system of the point indicating the existence position of the object from the image of the object captured by the first camera as the point information.

[0033] The information processing apparatus according to the twenty-seventh aspect is the information processing apparatus according to any one of the first aspect to the twenty-sixth aspect. The first processor outputs the point information based on the image of the object captured by the event camera. The second processor outputs the recognition information based on the image of the object captured by the second camera facing the direction corresponding to the event camera. The information processing apparatus includes a third processor, and the third processor associates the point information output from the first processor with the recognition information output from the second processor.

[0034] The information processing apparatus according to the twenty-eighth aspect is the information processing apparatus according to the twenty-seventh aspect. When the object cannot be captured from the visible light image of the object captured by the visible light camera due to a specified factor, the first processor outputs the point information based on the image of the object captured by the event camera.

[0035] The information processing apparatus according to the twenty-ninth aspect is the information processing apparatus according to the twenty-eighth aspect. The specified factor includes at least one of a case where the moving speed of the object is equal to or higher than a specified value and a case where the change in the amount of light per unit time of the ambient light is equal to or higher than a specified value.

[0036] The information processing apparatus according to the thirtieth aspect is the information processing apparatus according to any one of the twenty-seventh aspect to the twenty-ninth aspect. The event camera is a camera that outputs an event image, and the event image represents the difference part between the image captured at the current moment and the image captured at the previous moment.

[0037] The information processing apparatus according to the thirty-first aspect is the information processing apparatus according to any one of the first aspect to the thirtieth aspect. The second processor outputs an image of the photographed object at a first frame rate, and the first processor outputs motion information indicating the motion of the photographed object at a second frame rate higher than the first frame rate. The information processing apparatus includes a third processor, and the third processor performs driving control of the vehicle based on the image and the motion information.

[0038] The information processing apparatus according to the thirty-second aspect is the information processing apparatus according to the thirty-first aspect, and the second frame rate is 10 times or more the first frame rate.

[0039] The information processing apparatus according to the thirty-third aspect is the information processing apparatus according to the thirty-first aspect or the thirty-second aspect, and the second frame rate is 100 frames per second or more.

[0040] The information processing apparatus according to the thirty-fourth aspect is the information processing apparatus according to any one of the thirty-first aspect to the thirty-third aspect. The first processor outputs vector information indicating the motion of a point representing the existence position of the object along a specified coordinate axis.

[0041] The information processing apparatus according to the thirty-fifth aspect is the information processing apparatus according to the thirty-fourth aspect. Using two of the first processors, vector information indicating the motion of a point representing the existence position of the object along each of the three coordinate axes in a three-dimensional orthogonal coordinate system is output.

[0042] The information processing apparatus according to the thirty-sixth aspect is the information processing apparatus according to the thirty-fifth aspect, and the third processor has the ability to process a plurality of information in units of one billionth of a second.

[0043] The information processing apparatus according to the thirty-seventh aspect is the information processing apparatus according to the first aspect. From an image of an object photographed by a camera with a frame rate of 100 frames per second or more, only a point representing the existence position of the object is extracted, and vector information indicating the motion of the point representing the existence position of the object along a specified coordinate axis is output from the processor.

[0044] The information processing apparatus according to the thirty-eighth aspect is the information processing apparatus according to the thirty-fifth aspect. The first processor outputs the vector information for at least two diagonal points among the vertices of a quadrilateral surrounding the contour of the object.

[0045] The information processing apparatus according to the thirty-ninth aspect is the information processing apparatus according to any one of the first aspect to the thirty-eighth aspect. The first processor extracts points representing the existence position of the object from the image of the object and outputs motion information at a frame rate of 1000 frames per second or more. The motion information represents the motion of the points representing the existence position of the object along a specified coordinate axis.

[0046] The information processing apparatus according to the fortieth aspect is the information processing apparatus according to the thirty-ninth aspect. The first processor outputs vector information representing the motion of the center point or the center of gravity point of the object along a specified coordinate axis as the motion information.

[0047] The information processing apparatus according to the forty-first aspect is the information processing apparatus according to the thirty-ninth aspect or the fortieth aspect. The first processor outputs, as the motion information, vector information representing the motion along a specified coordinate axis for at least two diagonal points among the vertices of the quadrilateral surrounding the contour of the object.

[0048] The information processing apparatus according to the forty-second aspect is the information processing apparatus according to any one of the thirty-ninth aspect to the forty-first aspect. The image includes an infrared image.

[0049] The information processing apparatus according to the forty-third aspect is the information processing apparatus according to any one of the thirty-ninth aspect to the forty-second aspect. The image includes a visible light image and an infrared image that are synchronized with each other.

[0050] The information processing apparatus according to the forty-fourth aspect is the information processing apparatus according to any one of the thirty-ninth aspect to the forty-third aspect. Using two of the first processors, vector information representing the motion of the points representing the existence position of the object along each of the three coordinate axes in a three-dimensional orthogonal coordinate system is output as the motion information.

[0051] The information processing apparatus according to the forty-fifth aspect is the information processing apparatus according to any one of the thirty-ninth aspect to the forty-fourth aspect. The first processor derives the distance to the object based on the reflected wave obtained by reflecting an electromagnetic wave irradiated onto the object from the object, and outputs vector information representing the motion of the points representing the existence position of the object along each of the three coordinate axes in a three-dimensional orthogonal coordinate system as the motion information.

[0052] The information processing apparatus according to the forty-sixth aspect is the information processing apparatus according to any one of the thirty-ninth aspect to the forty-fifth aspect. It further includes: a second processor that outputs an image of the object at a frame rate less than 1000 frames per second; and a third processor that performs response control on the object based on the motion information and the image output from the second processor.

[0053] The information processing apparatus according to the forty-seventh aspect is the information processing apparatus according to any one of the first to forty-sixth aspects. The first processor extracts points indicating the existence position of the object from the image that reflects the object, and outputs points indicating the existence position of the object.

[0054] The information processing apparatus according to the forty-eighth aspect is the information processing apparatus according to the forty-seventh aspect. The information processing apparatus includes a camera capable of changing the frame rate. The first processor calculates a score related to the external environment, determines the frame rate of the camera according to the score, outputs a control signal indicating to capture an image at the determined frame rate to the camera, extracts points indicating the existence position of the object from the image captured by the camera, and outputs points indicating the existence position of the object.

[0055] The information processing apparatus according to the forty-ninth aspect is the information processing apparatus according to the forty-eighth aspect. The information processing apparatus is mounted on a vehicle. The first processor calculates the risk related to the driving of the vehicle as the score related to the external environment, determines the frame rate of the camera according to the risk, outputs a control signal instructing to capture an image at the determined frame rate to the camera, extracts points indicating the existence position of the object from the image captured by the camera, and outputs points indicating the existence position of the object.

[0056] The information processing apparatus according to the fiftieth aspect is the information processing apparatus according to any one of the forty-seventh to forty-ninth aspects. The first processor extracts an object from the image. When the existence position of the object is in a specified area, the first processor extracts points indicating the existence position of the object and outputs points indicating the existence position of the object.

[0057] The information processing apparatus according to the fifty-first aspect is the information processing apparatus according to any one of the forty-seventh to fiftieth aspects. The first processor extracts an object from the image, calculates a score for each object, extracts points indicating the existence position of the object for which the score is equal to or higher than a specified threshold, and outputs points indicating the existence position of the object.

[0058] The information processing apparatus according to the fifty-second aspect is the information processing apparatus according to any one of the first to fifty-first aspects, and includes: a first camera with a first horizontal viewing angle; a second camera with a second horizontal viewing angle wider than the first horizontal viewing angle; and an adjustment unit that adjusts the shooting direction of the first camera. When the movement of an object located in the blind spot of the first camera is detected in the image obtained by the second camera, the first processor controls the adjustment unit to direct the shooting direction of the first camera toward the detected object.

[0059] The information processing apparatus according to the fifty-third aspect is based on the information processing apparatus according to the fifty-second aspect, and the horizontal viewing angle of the aforementioned second camera is 360°.

[0060] The information processing apparatus according to the fifty-fourth aspect is based on the information processing apparatus according to the fifty-third aspect, and the adjustment range of the shooting direction of the aforementioned adjustment unit in the horizontal direction is within a range of ±135° with respect to the reference direction.

[0061] The information processing apparatus according to the fifty-fifth aspect is based on the information processing apparatus according to any one of the fifty-second aspect to the fifty-fourth aspect, and the resolution per unit viewing angle of the aforementioned first camera is higher than that of the second camera.

[0062] The information processing apparatus according to the fifty-sixth aspect is based on the information processing apparatus according to any one of the fifty-second aspect to the fifty-fifth aspect, and the aforementioned first processor uses the image acquired by the aforementioned first camera and the image acquired by the aforementioned second camera to obtain information on the distance to an object in the image.

[0063] The information processing apparatus according to the fifty-seventh aspect is based on the information processing apparatus according to any one of the first aspect to the fifty-sixth aspect, and includes: the aforementioned first processor that outputs motion information indicating the motion of an object extracted from an image; the aforementioned second processor that outputs category information indicating the category of an object extracted from an image; a third processor that performs response control on the aforementioned object based on the aforementioned motion information and the aforementioned category information; and an acquisition unit that acquires an image acquired by an external camera provided outside the moving body on which this apparatus is mounted and position information of the external camera, and the aforementioned second processor associates and outputs the category information indicating the category of an object extracted from the image acquired by the external camera with the position information of the external camera.

[0064] The information processing apparatus according to the fifty-eighth aspect is based on the information processing apparatus according to the fifty-seventh aspect. When an object extracted from the image acquired by the external camera is an object that may affect the travel of the aforementioned moving body, the aforementioned second processor attaches identification information indicating that it is an object that may affect travel and outputs the category information of the object.

[0065] In the information processing apparatus according to the fifty-ninth aspect, which is based on the information processing apparatus according to the fifty-eighth aspect, an object that may affect the travel of the aforementioned moving body is a person or an animal.

[0066] The information processing apparatus according to the sixtieth aspect is based on the information processing apparatus according to any one of the fifty-seventh aspect to the fifty-ninth aspect, and the aforementioned acquisition unit acquires an image and position information from an external camera located in front of the travel route of the aforementioned moving body.

[0067] The information processing apparatus according to the sixty-first aspect is based on the information processing apparatus according to the sixtieth aspect. The acquisition unit acquires an image and position information only from the external camera that is the closest to the apparatus in front of the traveling route of the moving body.

[0068] The information processing apparatus according to the sixty-second aspect is based on the information processing apparatus according to the sixtieth or sixty-first aspect. The acquisition unit determines an external camera located in front of the traveling route of the moving body based on the route information of the navigation system that guides the traveling route of the moving body.

[0069] The information processing apparatus according to the sixty-third aspect is based on the information processing apparatus according to any one of the first aspect to the sixty-second aspect. It includes: a photographing unit that sequentially moves the camera to a first position and a second position, where at least the horizontal position of the second position is different from that of the first position, and causes the camera to photograph images at the first position and the second position respectively; and a processing unit that calculates the three-dimensional positions of the objects reflected in the first image and the second image respectively based on the first image photographed at the first position and the second image photographed at the second position.

[0070] The information processing apparatus according to the sixty-fourth aspect is based on the information processing apparatus according to the sixty-third aspect. The photographing unit includes a moving unit that rotates the member on which the camera is mounted so that the camera moves in a circular path passing through each of the first position and the second position.

[0071] The information processing apparatus according to the sixty-fifth aspect is based on the information processing apparatus according to the sixty-fourth aspect. The processing unit calculates a score related to the external environment and determines the frame rate of the camera according to the calculated score. The moving unit changes the rotation speed of the member according to the frame rate determined by the processing unit.

[0072] The information processing apparatus according to the sixty-sixth aspect is based on the information processing apparatus according to any one of the sixty-third aspect to the sixty-fifth aspect. The processing unit corrects the deviation of the two-dimensional positions of the objects in the first image and the second image caused by the shooting time difference between the first image and the second image, and calculates the three-dimensional positions of the objects.

[0073] The information processing apparatus according to the sixty-seventh aspect is based on the information processing apparatus according to the sixty-sixth aspect. The aforementioned processing unit performs matching between the first image captured at the first position at the first time and the first image captured at the first position at the second time, and estimates the two-dimensional position of the aforementioned object in the virtual first image based on the matching result, thereby correcting the deviation of the two-dimensional position of the aforementioned object. The virtual first image is an image obtained when the first position is captured at the third time, which is the shooting time at the second position.

[0074] The information processing apparatus according to the sixty-eighth aspect is based on the information processing apparatus according to any one of the first aspect to the sixty-seventh aspect. It includes a third processor. The third processor associates the aforementioned point information output from the first processor with the aforementioned recognition information output from the second processor. When the image captured by the first camera is unclear, the third processor uses the image captured by the second camera as a substitute for the image captured by the first camera.

[0075] The information processing apparatus according to the sixty-ninth aspect is based on the information processing apparatus according to the sixty-eighth aspect. When the image captured by the first camera is unclear, the third processor performs processing to clarify the image captured by the first camera.

[0076] The information processing apparatus according to the seventieth aspect is based on the information processing apparatus according to any one of the first aspect to the sixty-ninth aspect. The second processor uses the image of the aforementioned object captured by another device existing near this device and the position information of the aforementioned object to obtain the position information of the aforementioned image captured by the second camera.

[0077] The information processing apparatus according to the seventy-first aspect is based on the information processing apparatus according to the seventieth aspect. The second processor considers the transmission time of the image of the aforementioned object and the position information of the aforementioned object from the aforementioned other device, and synchronizes the frame of the second camera with the frame of the image of the aforementioned object from the aforementioned other device.

[0078] The information processing apparatus according to the seventy-second aspect is based on the information processing apparatus according to any one of the first aspect to the seventy-first aspect. The first processor switches and outputs the first coordinate value or the second coordinate value at a specified timing according to the image of the aforementioned object captured by the first camera. Among at least two coordinate axes of the three-dimensional orthogonal coordinate system constituting the points representing the existence position of the captured aforementioned object, the first coordinate value represents the existence position of the center point or the centroid point of the aforementioned object, and the second coordinate value represents the existence position of at least two diagonal points among the vertices of the polygon surrounding the contour of the aforementioned object.

[0079] The information processing apparatus according to the seventy-third aspect, based on the information processing apparatus according to the seventy-second aspect, when the second processor cannot recognize the photographed object from the image of the object photographed by the second camera due to a specified factor, the first processor switches the output coordinate value from the first coordinate value to the second coordinate value.

[0080] The information processing apparatus according to the seventy-fourth aspect, based on the information processing apparatus according to the seventy-second aspect or the seventy-third aspect, when the moving speed of the object is equal to or lower than a specified threshold value, or when the moving direction of the object is a specified direction, the first processor switches the output coordinate value from the second coordinate value to the first coordinate value.

[0081] The information processing apparatus according to the seventy-fifth aspect, based on the information processing apparatus according to any one of the seventy-second aspect to the seventy-fourth aspect, includes a third processor, and the third processor associates the first coordinate value or the second coordinate value output from the first processor with the recognition information output from the second processor.

[0082] The information processing apparatus according to the seventy-sixth aspect, based on the information processing apparatus according to any one of the first aspect to the seventy-fifth aspect, when the first processor can determine that the possibility of contact between the object existing in the passage path through which the moving body passes and the moving body is low, the first processor stops outputting the point information related to the object.

[0083] The information processing apparatus according to the seventy-seventh aspect, based on the information processing apparatus according to the seventy-sixth aspect, when the object does not exist on the moving route of the moving body in the passage path and the object gradually moves away from the moving route, the first processor determines that the possibility of contact between the object and the moving body is low.

[0084] The information processing apparatus according to the seventy-eighth aspect, based on the information processing apparatus according to any one of the first aspect to the seventy-seventh aspect, the first processor outputs the point information based on the image of the object photographed by the first camera, and calculates and outputs motion information indicating the motion of the point based on the time series of the point information. The information processing apparatus includes a third processor, and the third processor associates the motion information output from the first processor with the recognition information output from the second processor. When motion information of a plurality of objects corresponding to the detected motion direction is detected, the third processor removes the motion information of the plurality of objects and associates the remaining motion information with the recognition information.

[0085] The information processing apparatus according to the seventy-ninth aspect is based on the information processing apparatus according to the seventy-eighth aspect. The first processor calculates the direction of the movement of the points using the Hough transform according to the time series of the point information, and calculates the speed of the movement according to the change in the point information in the calculated direction of the movement of the points.

[0086] The information processing apparatus according to the eightieth aspect is based on the information processing apparatus according to the seventy-eighth or seventy-ninth aspect. The third processor associates the movement information output from the first processor with the recognition information output from the second processor, and controls the autonomous driving of the moving body based on the movement information and the recognition information.

[0087] The information processing apparatus according to the eighty-first aspect is based on the information processing apparatus according to any one of the first aspect to the eightieth aspect, and includes at least one processor. The processor outputs the point information based on the image of the object captured by the camera. When a certain number or more of points are detected whose moving direction is either the up or down direction and is the same direction, and the moving amount in the up and down direction is within a certain range, among the points included in the point information, only the points other than the detected certain number or more of points are output.

[0088] The information processing apparatus according to the eighty-second aspect is based on the information processing apparatus according to the eighty-first aspect. When the moving body provided with the camera is going straight, the processor also outputs the points whose moving amount in the left and right direction is above the threshold among the detected certain number or more of points.

[0089] The information processing apparatus according to the eighty-third aspect is based on the information processing apparatus according to any one of the first aspect to the eighty-second aspect, and includes a first moving body processor for use with the first moving body. The first moving body processor identifies the types of the first objects included in the surroundings of the first moving body based on the first image obtained by photographing the surroundings of the first moving body. A second moving body processor for use with the second moving body identifies the types of the second objects included in the surroundings of the second moving body based on the second image obtained by photographing the surroundings of the second moving body. When the second moving body is identified by the first moving body processor as the type of the first object, and the first moving body is identified by the second moving body processor as the type of the second object, the first moving body processor sets an identifier indicating that the first moving body is recognized by the second moving body processor.

[0090] The information processing apparatus according to the eighty-fourth aspect is based on the information processing apparatus according to the eighty-third aspect. The second moving body is a moving body capable of autonomous driving. When the first moving body processor identifies the second moving body as the type of the first object, the first moving body is identified as the type of the second object by the second moving body processor, and the second moving body is in autonomous driving, the first moving body processor sets the identifier.

[0091] The information processing apparatus according to the eighty-fifth aspect is based on the information processing apparatus according to the eighty-third or eighty-fourth aspect. The first moving body and the second moving body are moving bodies capable of autonomous driving. When the identifier is set and the second moving body is in autonomous driving, the first moving body processor controls the autonomous driving of the first moving body based on the behavior of the second moving body.

[0092] The information processing apparatus according to the eighty-sixth aspect is based on the information processing apparatus according to the eighty-fifth aspect. The second moving body processor controls the autonomous driving of the second moving body. The first moving body processor obtains the control information for the second moving body processor to control the autonomous driving of the second moving body as the information representing the behavior, and controls the autonomous driving of the first moving body based on the control information.

[0093] The information processing apparatus according to the eighty-seventh aspect is based on the information processing apparatus according to the eighty-sixth aspect. A first priority is assigned to the first moving body, and a second priority is assigned to the second moving body. The first moving body processor obtains the control information on the condition that the first priority is higher than the second priority.

[0094] The information processing apparatus according to any one of the eighty-fifth to eighty-seventh aspects. The first moving body processor outputs first point information obtained by capturing the first object as a point based on a third image. The third image is an image obtained by photographing the surroundings of the first moving body at a frame rate higher than the frame rate for obtaining the first image. The second moving body information is associated with the first point information. The second moving body information can identify the second moving body identified as the type of the first object by the first moving body processor, and controls the autonomous driving of the first moving body based on the associated second moving body information and the first point information.

[0095] The information processing apparatus according to the eighty-ninth aspect is based on the information processing apparatus according to the eighty-eighth aspect. The first mobile body processor has a first processor, a second processor, and a third processor. The first processor outputs the first point information. The second processor identifies the type of the first object based on the first image. The third processor associates the second mobile body information with the first point information and controls the autonomous driving of the first mobile body.

[0096] The information processing apparatus according to the ninetieth aspect is the information processing apparatus according to any one of the eighty-third aspect to the eighty-ninth aspect. The second mobile body is a mobile body capable of autonomous driving. The second mobile body processor outputs second point information obtained by capturing the second object as a point based on a fourth image. The fourth image is an image obtained by photographing the surroundings of the second mobile body at a frame rate higher than the frame rate of the photographing for obtaining the second image. The first mobile body information is associated with the second point information. The first mobile body information can determine the first mobile body whose type is identified as the second object by the second mobile body processor. Based on the associated first mobile body information and the second point information, a control process for controlling the autonomous driving of the second mobile body is performed. When the second mobile body is identified as the type of the first object by the first mobile body processor, the first mobile body is identified as the type of the second object by the second mobile body processor, and the control process is performed by the second mobile body processor, the first mobile body processor sets the identifier.

[0097] The information processing apparatus according to the ninety-first aspect is the information processing apparatus according to any one of the eighty-fifth aspect to the ninetieth aspect. The first mobile body processor controls the autonomous driving of the first mobile body using control content that can avoid contact between the first mobile body and the second mobile body.

[0098] The information processing apparatus according to the ninety-second aspect is the information processing apparatus according to any one of the eighty-third aspect to the ninety-first aspect. When the identifier is set, the first mobile body processor notifies the notification device that the first mobile body has been recognized by the second mobile body processor.

[0099] The information processing apparatus according to the ninety-third aspect, which is the information processing apparatus according to any one of the first aspect to the ninety-second aspect, is an information processing apparatus including at least one processor. The foregoing processor outputs an image of an object taken at a first frame rate. For each of the images output at the foregoing first frame rate, label information indicating the category of the object included in the image is derived. Position information indicating the existence position of the object is output according to the image of the object taken at a second frame rate higher than the foregoing first frame rate. The position information at each time point corresponding to the output timing of the image output at the foregoing first frame rate among the position information output sequentially at the foregoing second frame rate is associated with the foregoing label information derived for the same object as the object corresponding to the position information.

[0100] The information processing apparatus according to the ninety-fourth aspect, which is the information processing apparatus according to the ninety-third aspect, for the position information among the position information output at the foregoing second frame rate that has not been associated with the foregoing label information, the foregoing processor associates the foregoing label information after the immediately preceding association.

[0101] The information processing apparatus according to the ninety-fifth aspect, which is the information processing apparatus according to the ninety-third aspect or the ninety-fourth aspect, the foregoing processor outputs at least one point indicating the existence position of the foregoing object as the foregoing position information.

[0102] The information processing apparatus according to the ninety-sixth aspect, which is the information processing apparatus according to any one of the ninety-third aspect to the ninety-fifth aspect, the foregoing processor derives the foregoing label information based on a first image with relatively high resolution output at the foregoing first frame rate, and derives the foregoing position information based on a second image with relatively low resolution.

[0103] The information processing apparatus according to the ninety-seventh aspect, which is the information processing apparatus according to any one of the second aspect to the ninety-sixth aspect, the foregoing third processor determines an object that satisfies a specified condition among the objects reflected in a first image captured by the foregoing first camera and the objects reflected in a second image captured by the foregoing second camera as the same object, and for the objects determined to be the same, associates the foregoing point information with the foregoing identification information.

[0104] The information processing apparatus according to the ninety-eighth aspect is based on the information processing apparatus according to the ninety-seventh aspect. The aforementioned specified conditions include integrating the positional relationship between the object shown in the aforementioned first image and the object shown in the aforementioned second image as the first condition, integrating the outlines of the object shown in the aforementioned first image and the object shown in the aforementioned second image as the second condition, and integrating the category of the object estimated for the object shown in the aforementioned first image and the category of the object estimated for the object shown in the aforementioned second image as the third condition. When at least one of the aforementioned first condition, the aforementioned second condition, and the aforementioned third condition is satisfied, the aforementioned third processor determines that the object shown in the aforementioned first image and the object shown in the aforementioned second image are the same object.

[0105] The information processing apparatus according to the ninety-ninth aspect is based on the information processing apparatus according to any one of the first aspect to the ninety-eighth aspect. The aforementioned information processing apparatus includes an extraction unit, an integration unit, an estimation unit, and an update unit. The aforementioned extraction unit extracts first planar coordinates from an image showing an object, and the first planar coordinates represent the existence position of the aforementioned object in a two-dimensional space at time t. The aforementioned integration unit combines the first planar coordinates with first depth information to generate three-dimensional coordinates of the aforementioned object at time t. The first depth information is information obtained by detecting the depth direction of the aforementioned object in a three-dimensional space. The aforementioned extraction unit extracts second planar coordinates, and the second planar coordinates represent the existence position of the aforementioned object in a two-dimensional space at the next moment after time t. The aforementioned estimation unit estimates second depth information corresponding to the aforementioned next moment based on the shape information of the space where the aforementioned object exists and the change from the first planar coordinates to the second planar coordinates. The aforementioned update unit integrates the second planar coordinates with the estimated second depth information to update the three-dimensional coordinates of the aforementioned object at the aforementioned next moment.

[0106] The information processing apparatus according to the one-hundredth aspect is based on the information processing apparatus according to the ninety-ninth aspect. The aforementioned integration unit detects the first depth information from the point cloud data detected by the sensor.

[0107] The information processing apparatus according to the one-hundred-and-first aspect is based on the information processing apparatus according to the one-hundredth aspect. When the second planar coordinates at the aforementioned next moment after the movement of the aforementioned object are extracted, if the depth information obtained from the point cloud data corresponding to the aforementioned next moment cannot be acquired, the estimation of the aforementioned second depth information and the aforementioned update are executed.

[0108] The information processing apparatus according to any one of the information processing apparatuses of the ninety-ninth aspect to the one-hundred-and-second aspect, wherein the shape information includes the shape of the road surface and the shape of the object set in advance, the shape of the road surface is obtained from the image acquired by the high-resolution camera and the point cloud data detected by the sensor, and the estimation unit estimates the second depth information by calculating the amount of movement in the depth direction when the shape of the object moving relative to the shape of the road surface changes from the first plane coordinates to the second plane coordinates.

[0109] The information processing apparatus according to any one of the information processing apparatuses of the ninety-ninth aspect to the one-hundred-and-third aspect, sets the points corresponding to the coordinates indicating the existence position of the object as each of a plurality of points among the vertices of the frame indicating the contour of the object, the extraction unit extracts the first plane coordinates and the second plane coordinates of each of the points, the integration unit generates the three-dimensional coordinates of the object of each of the points, the estimation unit estimates the second depth information of each of the points, and the update unit updates the three-dimensional coordinates of the object of each of the points.

[0110] The information processing apparatus according to any one of the information processing apparatuses of the first aspect to the one-hundred-and-third aspect, includes a third processor, and the third processor associates the point information output from the first processor with the recognition information output from the second processor. When there are a plurality of pieces of point information, the first processor outputs the point information at the next time point according to the priority determined in advance for the recognition information associated with the point information.

[0111] The information processing apparatus according to the one-hundred-and-fourth aspect, at the first time point, the first processor derives the coordinate value in the depth direction of the object of the point indicating the existence position of the object in the three-dimensional orthogonal coordinate system from the image of the object captured by the first camera as the point information. When there are a plurality of pieces of point information, the first processor outputs the coordinate value in the depth direction at the second time point according to the priority determined in advance for the recognition information associated with the point information, and the second time point is the next time point after the first time point.

[0112] The information processing apparatus according to the one-hundred-and-fourth aspect or the one-hundred-and-fifth aspect, regarding the priority, the first processor determines a high priority for the object for which an action occurs and determines a priority lower than the high priority for the object for which no action occurs according to the presence or absence of the action of the object indicated by the recognition information.

[0113] According to the information processing apparatus of the one hundred and sixth aspect, with respect to the foregoing priority, the foregoing first processor determines a relatively high priority corresponding to the risk factor of the object that has performed the foregoing behavior for the object that has performed the foregoing behavior.

[0114] According to the information processing apparatus of the one hundred and seventh aspect, when the number of the foregoing point information is equal to or greater than a specified number, the foregoing first processor derives the foregoing point information at the next time point according to the foregoing priority.

[0115] According to the information processing apparatus of any one of the first aspect to the one hundred and eighth aspect, with respect to the foregoing point information, at each time point that satisfies the specified conditions related to the foregoing image, the foregoing first processor derives, from the image of the foregoing object captured by the foregoing first camera, the coordinate value in the depth direction of the foregoing object of the point representing the existence position of the foregoing object in a three-dimensional orthogonal coordinate system as the foregoing point information.

[0116] According to the information processing apparatus of the one hundred and ninth aspect, the foregoing first processor sets each time point that satisfies the foregoing specified conditions as a second time point, and derives the coordinate value in the depth direction of the foregoing object at the second time point as the foregoing point information from the coordinate values in the width direction, height direction, and depth direction of the foregoing object in the three-dimensional orthogonal coordinate system at the first time point and the coordinate values in the width direction and height direction at the second time point, and the second time point is the next time point of the first time point.

[0117] According to the information processing apparatus of the one hundred and tenth aspect, the foregoing first processor sets the foregoing specified conditions as the change amount of the coordinate values in the width direction and height direction at the second time point with respect to the coordinate values in the width direction and height direction at the first time point, and derives the coordinate value in the depth direction at a predetermined frequency for the change amount.

[0118] According to the information processing apparatus of any one of the first aspect to the one hundred and eleventh aspect, with respect to the foregoing point information, the foregoing first processor derives, from the image of the foregoing object captured by the foregoing first camera, the coordinate value in the depth direction of the foregoing object of the point representing the existence position of the foregoing object in a three-dimensional orthogonal coordinate system as the foregoing point information, and the foregoing first processor learns the correction information of the coordinate value in the depth direction according to the specified sensor information.

[0119] The information processing apparatus according to the one hundred and thirteenth aspect, based on the information processing apparatus according to the one hundred and twelfth aspect, when the difference between the derived coordinate value in the depth direction and the coordinate value in the depth direction detected by a prescribed sensor is equal to or greater than a threshold value, the first processor collects the sensor information and learns the correction information for a specific driving condition in the collected sensor information.

[0120] The information processing apparatus according to the one hundred and fourteenth aspect, based on the information processing apparatus according to the one hundred and thirteenth aspect, the first processor derives the coordinate value in the depth direction at the second time point as the point information from the coordinate values in the width direction, height direction, and depth direction of the object in the three-dimensional orthogonal coordinate system at the first time point, and the coordinate values in the width direction and height direction at the second time point, where the second time point is the next time point after the first time point.

[0121] The information processing apparatus according to the first aspect includes: an information acquisition unit capable of acquiring a plurality of pieces of information related to a vehicle; an inference unit that uses deep learning to infer a plurality of index values based on the plurality of pieces of information acquired by the information acquisition unit; and a driving control unit that performs driving control of the vehicle based on the plurality of index values.

[0122] The information processing apparatus according to the one hundred and sixteenth aspect, based on the information processing apparatus according to the one hundred and fifteenth aspect, the inference unit infers the plurality of index values based on the plurality of pieces of information through multivariate analysis using an integration method based on the deep learning.

[0123] The information processing apparatus according to the one hundred and seventeenth aspect, based on the information processing apparatus according to the one hundred and fifteenth aspect or the one hundred and sixteenth aspect, the information acquisition unit acquires the plurality of pieces of information in units of one billionth of a second, and the inference unit and the driving control unit use the plurality of pieces of information acquired in units of one billionth of a second to perform inference of the plurality of index values and driving control of the vehicle in units of one billionth of a second.

[0124] The information processing apparatus according to any one of the one hundred and fifteenth aspect to the one hundred and seventeenth aspect further includes a strategy setting unit that sets a driving strategy for the vehicle until it reaches a destination, where the driving strategy includes theoretical values of at least one of an optimal route to the destination, a driving speed, an inclination, and braking, and the driving control unit includes a strategy update unit that updates the driving strategy based on the difference between the plurality of index values and the theoretical values.

[0125] The information processing device according to the one hundred and nineteenth aspect is based on the information processing device according to any one of the one hundred and fifteenth aspect to the one hundred and eighteenth aspect. The information acquisition unit includes a sensor, and the sensor is provided at the lower part of the vehicle and can detect the temperature, material, and inclination of the ground for driving.

[0126] The information processing device according to the one hundred and twentieth aspect is based on the information processing device according to the first aspect, and includes: an acquisition unit that acquires a detection result of an object detected in relation to the operation of a control device for controlling the autonomous driving of a vehicle and mounted on the vehicle; and an execution unit that performs cooling in the control device based on the detection result.

[0127] The information processing device according to the one hundred and twenty-first aspect is based on the information processing device according to the one hundred and twentieth aspect. As the detection result, the acquisition unit extracts points indicating the existence position of the object from one frame image of the object and acquires motion information at a frame rate of 100 frames per second or more. The motion information indicates the motion of the points indicating the existence position of the object along a specified coordinate axis.

[0128] The information processing device according to the one hundred and twenty-second aspect is based on the information processing device according to the one hundred and twenty-first aspect, and further includes a prediction unit that uses the detection result to predict the operation of the control device. The prediction unit uses a learning model generated by machine learning to predict the operation of the control device. The machine learning uses the detection result and the operation status of the control device when the detection result is obtained as learning data.

[0129] The information processing device according to the one hundred and twenty-third aspect is based on the information processing device according to the one hundred and twenty-second aspect. The prediction unit further predicts the respective temperature changes of a plurality of parts in the control device, and the execution unit controls the cooling of the parts in the control device.

[0130] The information processing device according to the one hundred and twenty-fourth aspect is based on the information processing device according to the first aspect, and includes: a calculation unit that calculates, for each of a plurality of combinations of a predetermined number of sensor information among a plurality of sensor information provided in the vehicle, the respective wheel speeds, inclinations of the four wheels for controlling the vehicle, and index values of each suspension supporting the wheels, and sums up the index values to calculate control variables for the wheel speed, the inclination, and each suspension; and a control unit that controls the autonomous driving based on the control variables.

[0131] The information processing device according to the one hundred and twenty-fifth aspect is based on the information processing device according to the one hundred and twenty-fourth aspect. The calculation unit calculates the control variables based on the index values through multivariate analysis based on an integration method using deep learning.

[0132] The information processing apparatus according to the one hundred and twenty-sixth aspect is based on the information processing apparatus according to the one hundred and twenty-fourth or one hundred and twenty-fifth aspect. The control unit controls the aforesaid autonomous driving in units of one billionth of a second based on the aforesaid control variable.

[0133] The information processing apparatus according to the one hundred and twenty-seventh aspect is based on the information processing apparatus according to the first aspect, and includes: an acquisition unit that acquires a plurality of pieces of information related to a vehicle from a detection unit including sensors, and as a cycle for detecting the state of the surroundings of the vehicle, the sensors detect the state of the surroundings of the vehicle at a second cycle shorter than a first cycle for photographing the surroundings of the vehicle; a calculation unit that calculates an index value related to the state of the surroundings of the vehicle based on the acquired plurality of pieces of information, and calculates a control variable for controlling the behavior of the vehicle based on the calculated index value; and a control unit that controls the behavior of the vehicle based on the calculated aforesaid control variable.

[0134] The information processing apparatus according to the one hundred and twenty-eighth aspect is based on the information processing apparatus according to the one hundred and twenty-seventh aspect. The aforesaid calculation unit calculates the aforesaid control variable based on the index value through multivariate analysis using integration based on the aforesaid deep learning.

[0135] The information processing apparatus according to the one hundred and twenty-ninth aspect is based on the information processing apparatus according to the one hundred and twenty-seventh or one hundred and twenty-eighth aspect. The aforesaid acquisition unit acquires the aforesaid plurality of pieces of information in units of one billionth of a second, and the aforesaid calculation unit uses the aforesaid plurality of pieces of information acquired in units of one billionth of a second to perform the calculation of the aforesaid index value and the aforesaid control variable in units of one billionth of a second.

[0136] The information processing apparatus according to the one hundred and thirtieth aspect is based on the information processing apparatus according to any one of the one hundred and twenty-seventh to one hundred and twenty-ninth aspects. The aforesaid calculation unit predicts a collision of an object with the vehicle based on the acquired plurality of pieces of information, and in a case where the predicted result indicates an inevitable collision, calculates a control variable corresponding to damage to the vehicle in the aforesaid inevitable collision that is less than or equal to a predetermined threshold as the aforesaid control variable.

[0137] The information processing apparatus according to the one hundred and thirty-first aspect is based on the information processing apparatus according to the one hundred and thirtieth aspect. The damage to the vehicle is at least one of the deformation position and deformation amount of the vehicle.

[0138] The information processing apparatus according to the one hundred and thirty-second aspect is based on the information processing apparatus according to the one hundred and thirtieth or one hundred and thirty-first aspect. The control variable corresponding to the damage to the vehicle is at least one of the collision angle and vehicle speed of the vehicle.

[0139] The information processing apparatus according to the one hundred and thirty-third aspect, based on the information processing apparatus according to the first aspect, the first processor extracts points representing the existence position of the object from the image showing the object, and outputs points representing the existence position of the object.

[0140] The information processing apparatus according to the one hundred and thirty-fourth aspect, based on the information processing apparatus according to the one hundred and thirty-third aspect, the information processing apparatus includes a camera capable of changing the frame rate. The first processor calculates a score related to the external environment, determines the frame rate of the camera based on the score, outputs a control signal instructing the camera to capture an image at the determined frame rate, extracts points representing the existence position of the object from the image captured by the camera, and outputs points representing the existence position of the object.

[0141] The information processing apparatus according to the one hundred and thirty-fifth aspect, based on the information processing apparatus according to the one hundred and thirty-fourth aspect, the information processing apparatus is mounted on a vehicle. As the score related to the external environment, the first processor calculates the risk level related to the driving of the vehicle, determines the frame rate of the camera based on the risk level, outputs a control signal instructing the camera to capture an image at the determined frame rate, extracts points representing the existence position of the object from the image captured by the camera, and outputs points representing the existence position of the object.

[0142] The information processing apparatus according to the one hundred and thirty-sixth aspect, based on the information processing apparatus according to any one of the one hundred and thirty-third aspect to the one hundred and thirty-fifth aspect, the first processor extracts an object from the image. When the existence position of the object is in a specified area, the first processor extracts points representing the existence position of the object, and outputs points representing the existence position of the object.

[0143] The information processing apparatus according to the one hundred and thirty-seventh aspect, based on the information processing apparatus according to any one of the one hundred and thirty-third aspect to the one hundred and thirty-sixth aspect, the first processor extracts an object from the image, calculates a score for each object, extracts points representing the existence position of the object whose score is equal to or higher than a specified threshold, and outputs points representing the existence position of the object.

[0144] A vehicle according to the one hundred and thirty-eighth aspect includes a pair of camera units including cameras and a processor. The first camera unit extracts points representing the existence position of the object from the image showing the object in front of the vehicle, and outputs points representing the existence position of the object. The second camera unit extracts points representing the existence position of the object from the image showing the object behind the vehicle, and outputs points representing the existence position of the object. The processor controls the driving of the vehicle based on the existence positions of the objects output from the first camera unit and the second camera unit.

[0145] The vehicle according to the one hundred and thirty-ninth aspect is based on the vehicle according to the one hundred and thirty-eighth aspect. Each of the aforementioned camera units has a camera capable of changing the frame rate. One of the aforementioned first camera unit and the aforementioned second camera unit sets the frame rate of the aforementioned camera according to the detection status of an object in the other of the aforementioned first camera unit and the aforementioned second camera unit. Points indicating the existence position of the aforementioned object are extracted from the image of the aforementioned camera captured at the set frame rate, and points indicating the existence position of the aforementioned object are output.

[0146] The vehicle according to the one hundred and fortieth aspect is based on the vehicle according to the one hundred and thirty-eighth aspect or the one hundred and thirty-ninth aspect. It is equipped with a high-resolution camera with a resolution higher than that of the aforementioned camera, and also equipped with other camera units connected to each of the aforementioned camera units. Each of the aforementioned camera units can obtain information about an object determined based on the information of the aforementioned high-resolution camera. One of the aforementioned first camera unit and the second camera unit obtains information about an object possessed by the other of the aforementioned first camera unit and the second camera unit.

[0147] The information processing method according to the one hundred and forty-first aspect is such that a computer performs the following processing: output point information obtained by capturing the photographed object as points based on an image of the object captured by a first camera; and output recognition information obtained by recognizing the photographed object based on an image of the object captured by a second camera facing in the direction corresponding to the first camera.

[0148] The information processing program according to the one hundred and forty-second aspect causes a computer to perform the following processing: output point information obtained by capturing the photographed object as points based on an image of the object captured by a first camera, and output recognition information obtained by recognizing the photographed object based on an image of the object captured by a second camera facing in the direction corresponding to the first camera.

[0149] It should be noted that the above summary of the present disclosure does not list all the necessary features of the present disclosure. In addition, sub-combinations of these feature groups can also form the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0150] Figure 1 It is a schematic diagram showing an example of a vehicle equipped with a Central Brain according to the first embodiment.

[0151] Figure 2 It is a block diagram showing an example of the structure of the information processing device according to the first embodiment.

[0152] Figure 3It is a block diagram showing an example of the structure of the information processing apparatus according to the second embodiment.

[0153] Figure 4 It is an explanatory diagram showing an example of the dot information output by the MoPU according to the third embodiment.

[0154] Figure 5 It is a block diagram showing an example of the structure of the information processing apparatus according to the fifth embodiment.

[0155] Figure 6 It is a block diagram showing an example of the structure of the information processing apparatus according to the sixth embodiment.

[0156] Fig. 7A It is an explanatory diagram showing an example of the association between the dot information and the label information according to the seventh embodiment.

[0157] Figure 7B It is a flowchart showing an example of the process of associating feature points with label information, which is performed by the central brain according to the seventh embodiment.

[0158] Figure 8 It is an explanatory diagram showing a schematic structure of a vehicle according to the eighth embodiment.

[0159] Fig. 9 It is a block diagram showing an example of the functional structure of the cooling execution apparatus according to the eighth embodiment.

[0160] Fig.10 It is the first block diagram showing an example of the structure of the information processing apparatus according to the ninth embodiment.

[0161] Fig.11 It is the second block diagram showing an example of the structure of the information processing apparatus according to the ninth embodiment.

[0162] Fig. 12A It is a diagram schematically showing the coordinate detection of an object in time series in the ninth embodiment.

[0163] Fig. 12B It is a flowchart in the case of determining the update frequency of the z coordinate value according to the change amount in the ninth embodiment.

[0164] Fig. 12C It is a flowchart in the case of learning correction information in the ninth embodiment.

[0165] Fig.13 It is a block diagram showing an example of the structure of the information processing apparatus according to the tenth embodiment.

[0166] Fig.14AThis is the first explanatory diagram for explaining the image of an object captured by the event camera according to the tenth embodiment.

[0167] Fig. 14B This is the second explanatory diagram for explaining the image of an object captured by the event camera according to the tenth embodiment.

[0168] Fig. 14C This is the third explanatory diagram for explaining the image of an object captured by the event camera according to the tenth embodiment.

[0169] Fig.15 This is the first block diagram showing an example of the structure of the information processing apparatus according to the eleventh embodiment.

[0170] Fig.16 This is the second block diagram showing an example of the structure of the information processing apparatus according to the eleventh embodiment.

[0171] Fig.17 This is the first block diagram showing an example of the structure of the information processing apparatus according to the twelfth embodiment.

[0172] Fig.18 This is the second block diagram showing an example of the structure of the information processing apparatus according to the twelfth embodiment.

[0173] Fig.19 This is the first block diagram showing an example of the structure of the information processing apparatus according to the thirteenth embodiment.

[0174] Fig. 20 This is the second block diagram showing an example of the structure of the information processing apparatus according to the thirteenth embodiment.

[0175] Fig.21A This is the first diagram for explaining the processing in the case where feature points are not extracted from an object existing on the sidewalk in the thirteenth embodiment.

[0176] Fig.21B This is the second diagram for explaining the processing in the case where feature points are not extracted from an object existing on the sidewalk in the thirteenth embodiment.

[0177] Fig. 22 This is the block diagram of the information processing apparatus according to the fourteenth embodiment.

[0178] Fig.23 This is the first diagram showing the shooting directions and shooting angle ranges of the high-resolution camera and the omnidirectional camera mounted on the vehicle according to the fourteenth embodiment.

[0179] Fig.24It is the first figure showing the relationship between the shooting ranges of the image obtained by the omnidirectional camera and the image obtained by the high-resolution camera 30L according to the fourteenth embodiment.

[0180] Fig.25 It is the second figure showing the shooting directions and shooting angle ranges of the high-resolution camera and the omnidirectional camera mounted on the vehicle according to the fourteenth embodiment.

[0181] Fig.26 It is the second figure showing the relationship between the shooting ranges of the image obtained by the omnidirectional camera and the image obtained by the high-resolution camera 30L according to the fourteenth embodiment.

[0182] Fig. 27 It is the block diagram of the information processing device according to the fifteenth embodiment.

[0183] Fig.28 It is the figure showing an example of the camera management database according to the fifteenth embodiment.

[0184] Fig.29 It is the first figure for explaining the vehicle according to the fifteenth embodiment.

[0185] Fig.30 It is the figure showing an example of the image obtained from the external camera according to the fifteenth embodiment.

[0186] Fig.31 It is the figure showing an example of the object management database according to the fifteenth embodiment.

[0187] Fig.32 It is the second figure for explaining the vehicle according to the fifteenth embodiment.

[0188] Fig.33 It is the third figure for explaining the vehicle according to the fifteenth embodiment.

[0189] Fig.34 It is the block diagram showing an example of the configuration of the information processing device according to the sixteenth embodiment.

[0190] Fig.35 It is the perspective view of the camera and the moving part according to the sixteenth embodiment.

[0191] Fig.36A It is the first explanatory figure for explaining the process of correcting the deviation of the two-dimensional position of the object according to the sixteenth embodiment.

[0192] Fig.36B It is the second explanatory figure for explaining the process of correcting the deviation of the two-dimensional position of the object according to the sixteenth embodiment.

[0193] Fig.37 It is a block diagram showing the functional structure of the central brain related to the seventeenth embodiment.

[0194] Fig.38A It is the first figure for explaining the method of calculating the movement direction of a point based on the time series of point information in the seventeenth embodiment.

[0195] Fig.38B It is the second figure for explaining the method of calculating the movement direction of a point based on the time series of point information in the seventeenth embodiment.

[0196] Fig.39 It is a figure showing an example of the movement information of multiple objects in the seventeenth embodiment.

[0197] Fig.40 It is a figure showing an example of the movement information after removal in the seventeenth embodiment.

[0198] Fig.41 It is a flowchart showing the process of the association process executed by the central brain related to the seventeenth embodiment.

[0199] Fig.42 It is a figure for explaining the output process of point information executed by MoPU related to the eighteenth embodiment.

[0200] Fig.43 It is a block diagram showing an example of the structure of the information processing device related to the nineteenth embodiment.

[0201] Fig.44 It is a schematic diagram showing an example of the processing contents of the first determination unit, the second determination unit, the third determination unit, and the setting unit of the first vehicle related to the nineteenth embodiment.

[0202] Fig.45 It is a schematic diagram showing an example of the processing contents performed by the inference unit and the driving control unit of the first vehicle in the case where no identifier is set by the setting unit related to the nineteenth embodiment.

[0203] Fig.46 It is a schematic diagram showing an example of the processing contents performed by the inference unit and the driving control unit of the first vehicle in the case where an identifier is set by the setting unit related to the nineteenth embodiment.

[0204] Fig.47 It is a schematic diagram showing an example of the aspect of avoiding a frontal collision between the first vehicle and the second vehicle related to the nineteenth embodiment.

[0205] Fig.48It is a first flowchart showing an example of the process of the automatic driving control process according to the nineteenth embodiment.

[0206] Fig.49 It is a second flowchart showing an example of the process of the automatic driving control process according to the nineteenth embodiment.

[0207] Fig.50 It is a third flowchart showing an example of the process of the automatic driving control process according to the nineteenth embodiment.

[0208] Fig.51 It is a schematic diagram showing an example of the aspect of notifying that the first vehicle is recognized by the second vehicle according to the nineteenth embodiment.

[0209] Fig.52A It is a diagram showing an example of the aspect of object recognition according to the twentieth embodiment.

[0210] Fig.52B It is a diagram showing an example of the aspect of extracting feature points of an object according to the twentieth embodiment.

[0211] Fig.53 It is a diagram showing an example of the aspect of MoPU according to the twentieth embodiment.

[0212] Fig.54 It is a diagram showing an example of a flowchart of the process of MoPU according to the twentieth embodiment.

[0213] Fig.55 It is a diagram for explaining the structure of a vehicle according to the twenty - first embodiment.

[0214] Fig.56 It is a block diagram showing an example of the structure of an information processing device according to the twenty - second embodiment.

[0215] Fig.57 It is a flowchart showing an example of the rate change process according to the twenty - second embodiment.

[0216] Fig.58 It is a diagram for explaining the viewing angle of a camera in the twenty - second embodiment.

[0217] Fig.59 It is a flowchart showing an example of the handover process according to the twenty - second embodiment.

[0218] Fig.60 It is a first block diagram showing an example of the structure of an information processing device according to the twenty - third embodiment.

[0219] Fig.61It is a second block diagram showing an example of the structure of the information processing apparatus according to the twenty-third embodiment.

[0220] Fig.62 It is a diagram schematically showing an example of the system according to the twenty-fourth embodiment.

[0221] Fig.63 It is an explanatory diagram for explaining the learning phase in the system according to the twenty-fourth embodiment.

[0222] Fig.64 It is an explanatory diagram for explaining the cooling execution phase in the system according to the twenty-fourth embodiment.

[0223] Fig.65 It is a first diagram schematically showing an example of the SoCBox (SoCBox) and the cooling unit according to the twenty-fourth embodiment.

[0224] Fig.66 It is a second diagram schematically showing an example of the SoCBox and the cooling unit according to the twenty-fourth embodiment.

[0225] Fig.67 It is a third diagram schematically showing an example of the SoCBox and the cooling unit according to the twenty-fourth embodiment.

[0226] Fig.68 It is a diagram schematically showing the danger prediction ability of the AI for ultra-high performance autonomous driving according to the twenty-fifth embodiment.

[0227] Fig.69 It is a flowchart executed by the central brain according to the twenty-fifth embodiment.

[0228] Fig.70 It is a first explanatory diagram for explaining a control example of autonomous driving based on the central brain according to the twenty-fifth embodiment.

[0229] Fig.71 It is a second explanatory diagram for explaining a control example of autonomous driving based on the central brain according to the twenty-fifth embodiment.

[0230] Fig.72 It is a third explanatory diagram for explaining a control example of autonomous driving based on the central brain according to the twenty-fifth embodiment.

[0231] Fig.73 It is a fourth explanatory diagram for explaining a control example of autonomous driving based on the central brain according to the twenty-fifth embodiment.

[0232] Fig.74 It is a fifth explanatory diagram for explaining a control example of autonomous driving based on the central brain according to the twenty-fifth embodiment.

[0233] Fig.75 As an explanatory diagram showing a control example of central-brain-based autonomous driving according to the twenty-sixth embodiment of the present disclosure, it is a schematic diagram showing a state in which other vehicles are traveling around the vehicle.

[0234] Fig.76 It is a flowchart executed by the central brain according to the twenty-sixth embodiment.

[0235] Fig.77 It is an explanatory diagram schematically showing an example of the hardware configuration of a computer that functions as an information processing device or a cooling execution device. Detailed Embodiments

[0236] Hereinafter, embodiments of the present disclosure will be described, but the following embodiments do not limit the present disclosure. In addition, all combinations of the features described in the embodiments are not essential for the solution of the present disclosure. The information processing device according to the embodiments of the disclosed technology can accurately obtain index values required for driving control based on a large amount of information related to vehicle control. Therefore, at least a part of the information processing device of the present disclosure can be mounted on a vehicle to achieve vehicle control.

[0237] (First Embodiment)

[0238] First, the first embodiment according to the present embodiment will be described. As an example, at least a part of the information processing device according to the present disclosure is mounted on the vehicle 100 to perform autonomous driving control of the vehicle 100. In addition, the information processing device can realize autonomous driving in real time based on data obtained through AI / multivariable analysis / goal seek / strategy formulation / optimal probability solution / optimal speed solution / optimal route management / input of various sensors at the edge at level 6, and can provide a driving system adjusted based on the delta optimal solution. The vehicle 100 is an example of an "object".

[0239] Here, "level 6" refers to a level indicating autonomous driving, which is equivalent to a level higher than level 5 indicating fully autonomous driving. Although level 5 indicates fully autonomous driving, it is a level equivalent to human driving, and there is still a probability of accidents and the like even so. Level 6 indicates a level higher than level 5, which is equivalent to a level with a lower probability of accidents than level 5.

[0240] The computing power in level 6 is about 1000 times that of level 5. Therefore, high-performance driving control that cannot be achieved by level 5 can be realized.

[0241] Figure 1 FIG. 1 is a schematic diagram showing an example of a vehicle 100 equipped with a Central Brain 15. A plurality of Gate Ways are communicably connected to the Central Brain 15. The Central Brain 15 is connected to an external cloud server via the gateway. The Central Brain 15 is configured to be able to access an external cloud server via the gateway. On the other hand, due to the existence of the gateway, it is configured that the Central Brain 15 cannot be directly accessed from the outside.

[0242] Every time a predetermined time elapses, the Central Brain 15 outputs a request signal to the cloud server. Specifically, the Central Brain 15 outputs a request signal indicating an inquiry to the cloud server every one billionth of a second. As an example, the Central Brain 15 controls level 6 autonomous driving based on a plurality of pieces of information acquired via the gateway.

[0243] Figure 2 FIG. 2 is a block diagram showing an example of the structure of an information processing apparatus 10 according to the first embodiment. The information processing apparatus 10 includes an IPU (Image Processing Unit) 11, a MoPU (Motion Processing Unit) 12, a Central Brain 15, and a memory 16. The Central Brain 15 is configured to include a GNPU (Graphics Neural network Processing Unit) 13 and a CPU (Central Processing Unit) 14.

[0244] The IPU 11 is built into a super high-resolution camera (not shown) provided in the vehicle 100. The IPU 11 performs predetermined image processing such as Bayer conversion, demosaicing, denoising, and sharpening on an image of an object existing around the vehicle 100 captured by the super high-resolution camera, and outputs the processed object image at a frame rate of, for example, 10 frames / second and a resolution of 12 million pixels. In addition, the IPU 11 outputs identification information obtained by identifying the object captured by the super high-resolution camera. The identification information is information required to identify what the captured object is (for example, a person or an obstacle). In the present embodiment, as the identification information, the IPU 11 outputs label information indicating the category (type) of the captured object (for example, information indicating that the captured object is a dog, a cat, or a bear). Further, the IPU 11 outputs position information indicating the position of the captured object in the camera coordinate system of the super high-resolution camera. The image, label information, and position information output from the IPU 11 are provided to the Central Brain 15 and the memory 16. The IPU 11 is an example of a "second processor", and the super high-resolution camera is an example of a "second camera".

[0245] The MoPU 12 is built into another camera (not shown) different from the ultra-high resolution camera provided in the vehicle 100. The MoPU 12 outputs point information obtained by capturing the photographed object as a point at a frame rate of 100 frames per second or more based on an image of an object photographed by another camera facing the direction corresponding to the ultra-high resolution camera at a frame rate of 100 frames per second or more. The point information output from the MoPU 12 is provided to the central brain 15 and the memory 16. Thus, the image used by the MoPU 12 to output the point information and the image used by the IPU 11 to output the recognition information refer to images photographed by the other camera and the ultra-high resolution camera facing the corresponding direction (images obtained by photographing). Here, the "corresponding direction" means the direction in which the shooting range of the other camera overlaps with the shooting range of the ultra-high resolution camera. In the above case, the other camera photographs an object facing the direction overlapping with the shooting range of the ultra-high resolution camera. It should be noted that the ultra-high resolution camera and the other camera photograph an object facing the corresponding direction, for example, by previously obtaining the correspondence relationship of the camera coordinate systems between the ultra-high resolution camera and the other camera.

[0246] For example, as the point information, the MoPU 12 outputs the coordinate values of at least two coordinate axes in the three-dimensional orthogonal coordinate system of the point indicating the existence position of the object. As an example, the coordinate values represent the center point (or center of gravity point) of the object. In addition, the MoPU 12 outputs the coordinate value of the axis (x-axis) along the width direction in the three-dimensional orthogonal coordinate system (hereinafter referred to as "x coordinate value") and the coordinate value of the axis (y-axis) along the height direction (hereinafter referred to as "y coordinate value") as the coordinate values of the two coordinate axes. It should be noted that the x-axis is the axis along the vehicle width direction of the vehicle 100, and the y-axis is the axis along the vehicle height direction of the vehicle 100.

[0247] According to the above structure, since the point information output by the MoPU 12 per second includes more than 100 x coordinate values and y coordinate values, based on this point information, the movement (moving direction and moving speed) of the object on the x-axis and y-axis in the above three-dimensional orthogonal coordinate system can be grasped. That is, the point information output by the MoPU 12 includes position information indicating the position of the object in the above three-dimensional orthogonal coordinate system and movement information indicating the movement of the object.

[0248] As described above, in the point information output from the MoPU 12, the information required to identify what the captured object is (for example, a person or an obstacle) is not included, and only the information indicating the movement (moving direction and moving speed) of the center point (or center of gravity point) of the object on the x-axis and y-axis is included. Also, since the point information output from the MoPU 12 does not include image information, it is possible to significantly reduce the amount of data output to the central brain 15 and the memory 16. The MoPU 12 is an example of a "first processor", and the other cameras are examples of "first cameras".

[0249] As described above, in the present embodiment, the frame rate of the other cameras with the MoPU 12 built in is higher than the frame rate of the ultra-high resolution camera with the IPU 11 built in. Specifically, the frame rate of the other cameras is 100 frames per second or more, and the frame rate of the ultra-high resolution camera is 10 frames per second. That is to say, the frame rate of the other cameras is more than 10 times the frame rate of the ultra-high resolution camera.

[0250] The central brain 15 associates the point information output from the MoPU 12 with the label information output from the IPU 11. For example, due to the frame rate difference between the above-mentioned other cameras and the ultra-high resolution camera, the central brain 15 may be in a state where it has acquired the point information related to the object but has not acquired the label information. In this state, the central brain 15 identifies the x-coordinate value and y-coordinate value of the object based on the point information, but does not identify what the object is.

[0251] After that, when the label information related to the above object is acquired, the central brain 15 derives the category of the label information (for example: PERSON (human)). And the central brain 15 associates the label information with the point information acquired above. Thereby, the central brain 15 identifies the x-coordinate value and y-coordinate value of the object based on the point information, and also identifies what the object is. The central brain 15 is an example of a "third processor".

[0252] Here, when there are multiple objects captured by the ultra-high resolution camera and the other cameras, such as object A and object B, the central brain 15 associates the point information and label information related to each object in the following manner. Due to the frame rate difference between the above-mentioned other cameras and the ultra-high resolution camera, the central brain 15 may be in a state where it has acquired the point information related to object A and object B (hereinafter referred to as "point information A" and "point information B") but has not acquired the label information. In this state, the central brain 15 identifies the x-coordinate value and y-coordinate value of object A based on point information A, and identifies the x-coordinate value and y-coordinate value of object B based on point information B, but does not identify what these objects are.

[0253] After that, in the case of obtaining a piece of tag information, the central brain 15 derives the category of the one piece of tag information (for example: PERSON). And, the central brain 15 determines the point information associated with the one piece of tag information based on the position information output from the IPU 11 together with the one piece of tag information and the position information included in the acquired point information A and point information B. For example, the central brain 15 determines the point information including the position information that represents the position closest to the position of the object represented by the position information output from the IPU 11, and associates the point information with the one piece of tag information. In the case where the point information determined above is the point information A, the central brain 15 associates the one piece of tag information with the point information A, identifies the x coordinate value and y coordinate value of the object A based on the point information A, and identifies what the object A is.

[0254] As described above, when there are multiple objects captured by the ultra-high-resolution camera and other cameras, the central brain 15 associates the point information with the tag information based on the position information output from the IPU 11 and the position information included in the point information output from the MoPU 12.

[0255] In addition, the central brain 15 identifies the objects (people, animals, roads, signals, signs, crosswalks, obstacles, buildings, etc.) existing around the vehicle 100 based on the image and tag information output from the IPU 11. In addition, the central brain 15 identifies the position and movement of the objects whose identities have been recognized existing around the vehicle 100 based on the point information output from the MoPU 12. Based on the recognized information, the central brain 15 performs, for example, control (speed control) of the motor driving the wheels, braking control, and steering wheel control to control the autonomous driving of the vehicle 100 as a response control for the object. For example, the central brain 15 controls the autonomous driving of the vehicle 100 according to the position information and movement information included in the point information output from the MoPU 12 to avoid collisions with the objects. In the central brain 15, the GNPU 13 can undertake the processing related to image recognition, and the CPU 14 can undertake the processing related to the control of the vehicle 100.

[0256] Generally, ultra-high-resolution cameras are used for image recognition in autonomous driving. Here, based on the images captured by the ultra-high-resolution camera (the images obtained by capturing with the ultra-high-resolution camera), it is possible to identify what objects are contained in the image. However, in level 6 autonomous driving, this is not enough. In the level 6 era, it is also necessary to identify the movement of objects with higher precision. By using the MoPU 12, the movement of objects can be identified with higher precision, so that, for example, the avoidance action of the vehicle 100 driving through autonomous driving to avoid obstacles can be performed with higher precision. However, through the ultra-high-resolution camera, only about 10 frames of images can be obtained per second, and the precision of analyzing the movement of objects is lower than that of the camera equipped with the MoPU 12. On the other hand, through the camera equipped with the MoPU 12, for example, it can output at a high frame rate of 100 frames per second.

[0257] Therefore, the information processing device 10 according to the first embodiment includes two independent processors, namely, the IPU 11 and the MoPU 12. The information processing device 10 assigns the role of obtaining the information required to identify what the captured object is to the IPU 11 built into the ultra-high-resolution camera, and assigns the role of detecting the position and movement of the object to the MoPU 12 built into other cameras. The MoPU 12 captures the captured object as a point, and analyzes in which direction and at what speed the coordinates of this point move on at least the x-axis and the y-axis in the above three-dimensional orthogonal coordinate system. Since the overall contour of the object and the detection of what the object is can be performed through the images from the ultra-high-resolution camera, as long as it is known how, for example, the center point of the object moves through the MoPU 12, it is known what behavior the whole object makes.

[0258] According to the method of only analyzing the movement and speed of the center point of the object, compared with the case of judging how the whole image of the object moves, the amount of data output to the central brain 15 can be greatly suppressed, and the amount of computation in the central brain 15 can be greatly reduced. For example, in the case of outputting a 1000-pixel × 1000-pixel image to the central brain 15 at a frame rate of 1000 frames per second, if color information is included, 4 billion bits per second of data will be output to the central brain 15. By only outputting the point information representing the movement of the center point of the object, the MoPU 12 can compress the amount of data output to the central brain 15 to 20,000 bits per second. That is, the amount of data output to the central brain 15 is compressed to 1 / 200,000.

[0259] In this way, by combining and using the low frame rate and high-resolution image and label information output from the IPU 11 with the high frame rate and lightweight point information output from the MoPU 12, object recognition including the movement of objects can be achieved with a small amount of data.

[0260] In addition, in the information processing apparatus 10, by associating the point information output from the MoPU 12 with the tag information output from the IPU 11 by the central brain 15, it is possible to grasp information related to what kind of object is performing what kind of movement.

[0261] (Second Embodiment)

[0262] Next, the second embodiment according to the present embodiment will be described by omitting or simplifying the overlapping parts with the above-described embodiments.

[0263] Figure 3 is a block diagram showing an example of the configuration of the information processing apparatus 10 according to the second embodiment. As Figure 3 shown, the information processing apparatus 10 mounted on the vehicle 100 includes a MoPU 12L corresponding to the left eye, a MoPU 12R corresponding to the right eye, an IPU 11, and a central brain 15.

[0264] The MoPU 12L includes a camera 302L, a radar 32L, an infrared camera 34L, and a core 17L. In addition, the MoPU 12R includes a camera 302R, a radar 32R, an infrared camera 34R, and a core 17R. It should be noted that hereinafter, when not distinguishing between the MoPU 12L and the MoPU 12R, it is denoted as "MoPU 12", when not distinguishing between the camera 302L and the camera 302R, it is described as "camera 302", when not distinguishing between the radar 32L and the radar 32R, it is denoted as "radar 32", when not distinguishing between the infrared camera 34L and the infrared camera 34R, it is denoted as "infrared camera 34", and when not distinguishing between the core 17L and the core 17R, it is denoted as "core 17".

[0265] The camera 302 included in the MoPU 12 captures an object at a frame rate greater than that of the ultra-high-resolution camera (e.g., 10 frames per second) included in the IPU 11 (120, 240, 480, 960, or 1920 frames per second). The frame rate of the camera 302 is variable. The camera 302 is an example of the "first camera".

[0266] The radar 32 included in the MoPU 12 acquires a radar signal, which is a signal of a reflected wave based on an electromagnetic wave irradiated onto an object and reflected from the object. The infrared camera 34 included in the MoPU 12 is a camera that captures an infrared image (a camera that captures infrared rays from an object to acquire an infrared image representing the object).

[0267] The core 17 included in the MoPU 12 (e.g., composed of more than one CPU) extracts feature points from each frame image captured by the camera 302 (a frame image obtained by capturing with the camera 302), and outputs the x - coordinate value and y - coordinate value of the object in the above three - dimensional orthogonal coordinate system as point information. The core 17 uses, for example, the center point (center of gravity point) of the object extracted from the image as the feature point. It should be noted that the point information output by the core 17 is similar to that in the above - mentioned embodiment and includes position information and motion information.

[0268] The IPU 11 is equipped with an ultra - high - resolution camera (not shown), and outputs an image of an object captured by the ultra - high - resolution camera (an image obtained by capturing the object with the ultra - high - resolution camera), label information indicating the category of the object, and position information indicating the position of the object in the camera coordinate system of the ultra - high - resolution camera.

[0269] The central brain 15 acquires the point information output from the MoPU 12, and the image, label information, and position information output from the IPU 11. And the central brain 15 associates the label information related to the object existing at the position corresponding to the position information included in the point information output from the MoPU 12 and the position information output from the IPU 11 with the point information. Thus, in the information processing device 10, it is possible to associate the information about what the object represented by the label information is with the position and motion of the object represented by the point information.

[0270] Here, the MoPU 12 changes the frame rate of the camera 302 according to a specified factor. In the second embodiment, as an example of the specified factor, the MoPU 12 changes the frame rate of the camera 302 according to a score related to the external environment. In this case, the MoPU 12 calculates a score related to the external environment for the vehicle 100, and changes the frame rate of the camera 302 according to the calculated score. And the MoPU 12 outputs a control signal for the camera 302 to capture an image at the changed frame rate. Thus, the camera 302 captures an image at the frame rate indicated by the control signal (the camera 302 obtains an image by capturing at the frame rate indicated by the control signal). With this structure, according to the information processing device 10, it is possible to capture an image of an object at a frame rate suitable for the external environment (it is possible to capture an object at a frame rate suitable for the external environment).

[0271] It should be noted that the information processing device 10 mounted on the vehicle 100 is equipped with a variety of sensors (not shown). The MoPU 12 calculates the risk related to the movement of the vehicle 100 as a score related to the external environment for the vehicle 100 based on the sensor information obtained from the variety of sensors (such as the movement of the center of gravity of the weight, the detection of the road material, the detection of the external air temperature, the detection of the external air humidity, the detection of the inclination angles in the vertical, horizontal, and diagonal directions of the slope, the freezing mode of the road, the detection of the moisture content, the material, wear condition, and air pressure of each tire, the road width, the presence or absence of overtaking prohibition, the vehicle types of oncoming vehicles and vehicles in front and behind, the cruising states of these vehicles, or the surrounding conditions (birds, animals, footballs, accident vehicles, earthquakes, fires, winds, typhoons, heavy rains, light rains, snowstorms, fog, etc.)) and the point information. The risk indicates the degree to which the vehicle 100 will drive in a dangerous place in the future. In this case, the MoPU 12 changes the frame rate of the camera 302 according to the calculated risk. The vehicle 100 is an example of a "moving body". With this structure, according to the information processing device 10, the frame rate of the camera 302 can be changed according to the risk related to the movement of the vehicle 100. The sensor is an example of a "detection unit", and the sensor information is an example of "detection information".

[0272] For example, the higher the calculated risk, the higher the frame rate of the camera 302 set by the MoPU 12. When the calculated risk is less than the first threshold, the MoPU 12 changes the frame rate of the camera 302 to 120 frames per second. In addition, when the calculated risk is greater than or equal to the first threshold and less than the second threshold, the MoPU 12 changes the frame rate of the camera 30 to any one of 240, 480, and 960 frames per second. In addition, when the calculated risk is greater than or equal to the second threshold, the MoPU 12 changes the frame rate of the camera 302 to 1920 frames per second. It should be noted that in any of the above cases of the risk, in addition to causing the camera 302 to capture images at the selected frame rate (causing the camera 302 to capture at the selected frame rate), the MoPU 12 can also output control signals to the radar 32 and the infrared camera 34 so as to obtain radar signals and capture infrared images with values corresponding to the frame rate.

[0273] For example, the lower the calculated risk level, the lower the frame rate of the camera 302 set by the MoPU 12. When the frame rate of the camera 302 is set to 1920 frames per second, if the calculated risk level is greater than or equal to the first threshold and less than the second threshold, the MoPU 12 changes the frame rate of the camera 30 to any one of 240, 480, and 960 frames per second. In addition, when the frame rate of the camera 302 is set to 1920 frames per second, if the calculated risk level is less than the first threshold, the MoPU 12 changes the frame rate of the camera 302 to 120 frames per second. Further, when the frame rate of the camera 302 is set to any one of 240, 480, and 960 frames per second, if the calculated risk level is less than the first threshold, the MoPU 12 changes the frame rate of the camera 302 to 120 frames per second. It should be noted that in this case, similar to the above, a control signal can be output to the radar 32 and the infrared camera 34 so that the radar signal and the infrared image are acquired at values corresponding to the changed frame rate of the camera 302 (perform shooting for acquiring the radar signal and obtaining the infrared image at values corresponding to the changed frame rate of the camera 302).

[0274] In addition, the MoPU 12 can calculate the risk level by using big data related to driving, such as long-tail event AI (Artificial Intelligence) DATA (e.g., Trip data of a vehicle equipped with a level 5 autonomous driving control mode) or map information, which is known before the vehicle 100 travels, as information for predicting the risk level.

[0275] In the above, a risk level is calculated as a score related to the external environment, but the index as the score related to the external environment is not limited to the risk level. For example, the MoPU 12 can calculate a score related to the external environment different from the risk level based on the moving direction or speed of an object shown by the camera 302, and change the frame rate of the camera 302 according to the score. Hereinafter, the following case will be described: the MoPU 12 calculates a speed score, which is a score related to the speed of an object shown by the camera 302, and changes the frame rate of the camera 302 according to the speed score. As an example, the speed score is set such that the faster the speed of the object, the higher the speed score, and the slower the speed of the object, the lower the speed score. Also, the higher the calculated speed score, the higher the frame rate of the camera 302 that the MoPU 12 sets, and the lower the calculated speed score, the lower the frame rate of the camera 302 that the MoPU 12 sets. Therefore, when the speed score calculated due to the high speed of the object becomes greater than or equal to the threshold value, the MoPU 12 changes the frame rate of the camera 302 to 1920 frames / second. Further, when the speed score calculated due to the low speed of the object becomes less than the threshold value, the MoPU 12 changes the frame rate of the camera 302 to 120 frames / second. It should be noted that also in this case, similarly to the above, a control signal can be output to the radar 32 and the infrared camera 34 so as to acquire a radar signal and capture an infrared image at a value corresponding to the changed frame rate of the camera 302 (perform shooting for acquiring a radar signal and obtaining an infrared image at a value corresponding to the changed frame rate of the camera 302).

[0276] Next, the following situation will be described: The MoPU 12 calculates a score related to the moving direction of the object shown by the camera 302, that is, the direction score, and changes the frame rate of the camera 302 according to the direction score. As an example, the direction score is set such that the direction score becomes higher when the moving direction of the object is the direction approaching the road, and the direction score becomes lower when the moving direction of the object is the direction away from the road. Moreover, the higher the calculated direction score, the higher the frame rate of the camera 302 that the MoPU 12 makes, and the lower the calculated direction score, the lower the frame rate of the camera 302 that the MoPU 12 makes. Specifically, the MoPU 12 determines the moving direction of the object by using AI or the like, and calculates the direction score based on the determined moving direction. And, in the case where the direction score calculated due to the moving direction of the object being the direction approaching the road becomes greater than or equal to the threshold value, the MoPU 12 changes the frame rate of the camera 302 to 1920 frames / second. In addition, in the case where the direction score calculated due to the moving direction of the object being the direction away from the road becomes less than the threshold value, the MoPU 12 changes the frame rate of the camera 302 to 120 frames / second. It should be noted that, also in this case, similar to the above, a control signal can also be output to the radar 32 and the infrared camera 34 so as to acquire the radar signal and capture the infrared image with a value corresponding to the changed frame rate of the camera 302 (perform the capture for acquiring the radar signal and obtaining the infrared image with a value corresponding to the changed frame rate of the camera 302).

[0277] In addition, the MoPU 12 can output the point information only for the object whose calculated score related to the external environment is greater than or equal to the specified threshold value. In this case, for example, the MoPU 12 can determine whether to output the point information related to the object according to the moving direction or the existing position of the object shown by the camera 302. For example, the MoPU 12 can not output the point information related to the object that has little influence on the driving of the vehicle 100. Specifically, the MoPU 12 calculates the moving direction of the object shown by the camera 302, and does not output the point information related to the object such as a pedestrian gradually moving away from the road. On the other hand, the MoPU 12 outputs the point information related to the object approaching the road (for example, a pedestrian about to rush onto the road) or the object existing in the road (for example, a pedestrian or a bicycle crossing the crosswalk set on the road). With this structure, according to the information processing device 10, the point information related to the object that has little influence on the driving of the vehicle 100 can not be output.

[0278] Note that in the above description, when an object exists within the road on which the vehicle 100 travels, the MoPU 12 is set to output point information related to the object. However, the disclosed technology is not limited to this aspect. For example, when it can be determined that the possibility of contact between the object and the vehicle 100 is low even if the object exists within the road on which the vehicle 100 travels, the MoPU 12 may stop outputting the point information related to the object. The road is an example of a "passage path".

[0279] Here, when the object does not exist on the moving route of the vehicle 100 within the road and the object gradually moves away from the moving route, the MoPU 12 determines that the possibility of contact between the object and the vehicle 100 is low. As an example, the situation where the object does not exist on the moving route of the vehicle 100 within the road is exemplified as follows: in a multi-lane road, the object is located in a lane different from the lane in which the vehicle 100 is traveling. In addition, the situation where the object gradually moves away from the moving route of the vehicle 100 is exemplified as follows: in a multi-lane road, the object is moving in a direction away from the lane in which the vehicle 100 is traveling.

[0280] With the foregoing structure, compared with the case of always outputting the point information related to the object when the object exists within the road on which the vehicle 100 travels, the processing load of the MoPU 12 can be reduced, and at the same time, the amount of data output to the central brain 15 can be reduced.

[0281] In addition, in the above description, the case where the MoPU 12 calculates the risk level is exemplified. However, the disclosed technology is not limited to this method. For example, the central brain 15 may calculate the risk level instead of the MoPU 12. In this case, the central brain 15 calculates the risk level related to the movement of the vehicle 100 based on the sensor information obtained from various sensors and the point information output from the MoPU 12, as a score related to the external environment for the vehicle 100. And the central brain 15 outputs an instruction to change the frame rate of the camera 302 according to the calculated risk level to the MoPU 12.

[0282] In addition, in the above description, a case where the MoPU 12 outputs point information based on an image captured by a camera (an image obtained by capturing with the camera 302) is illustrated, but the disclosed technology is not limited to this method. For example, the MoPU 12 may output point information based on radar signals and infrared images instead of the image captured by the camera 302 (an image obtained by capturing with the camera 302). Similar to the image captured by the camera 302 (an image obtained by capturing with the camera 302), the MoPU 12 can derive the x-coordinate value and y-coordinate value of an object from the infrared image of the object captured by the infrared camera 34. The radar 32 can obtain three-dimensional point cloud data of an object based on radar signals. That is, the radar 32 can detect the coordinate of the z-axis in the above three-dimensional orthogonal coordinate system. Here, the z-axis is an axis along the depth direction of the object and the traveling direction of the vehicle 100. Hereinafter, the coordinate value of the z-axis will be referred to as the "z-coordinate value". In this case, the MoPU 12 uses the principle of a stereo camera and combines the x-coordinate value and y-coordinate value of the object captured by the infrared camera 34 with the z-coordinate value of the object represented by the three-dimensional point cloud data at the same timing as when the radar 32 obtains the three-dimensional point cloud data of the object, and derives the coordinate values of the three axes (x-axis, y-axis, and z-axis) of the object as point information. Then, the MoPU 12 outputs the derived point information to the central brain 15.

[0283] In addition, in the above description, a case where the MoPU 12 derives point information is illustrated, but the disclosed technology is not limited to this method. For example, the central brain 15 may derive point information instead of the MoPU 12. The central brain 15 derives point information by combining information detected by, for example, the camera 302L, the camera 302R, the radar 32, and the infrared camera 34. As a specific example, the central brain 15 performs three-point measurement based on the x-coordinate value and y-coordinate value of the object captured by the camera 302L and the x-coordinate value and y-coordinate value of the object captured by the camera 302R, and thereby derives the coordinate values of the three axes (x-axis, y-axis, and z-axis) of the object as point information.

[0284] In addition, in the above description, an example is given where the central brain 15 controls the autonomous driving of the vehicle 100 based on the images and label information output from the IPU 11 and the point information output from the MoPU 12. However, the disclosed technology is not limited to this method. For example, the central brain 15 can perform motion control of a robot based on the above information output from the IPU 11 and the MoPU 12. The robot can be a humanoid intelligent robot that performs operations instead of humans. In this case, the central brain 15 performs motion control of the robot's arms, palms, fingers, feet, etc. based on the above information output from the IPU 11 and the MoPU 12, so that it can perform actions such as grasping, holding, hugging, carrying on the back, moving, transporting, throwing, kicking, and avoiding objects. When the central brain 15 performs motion control of the robot, the IPU 11 and the MoPU 12 can be mounted at the positions of the right and left eyes of the robot. That is to say, the IPU 11 and the MoPU 12 for the right eye can be mounted on the right eye, and the IPU 11 and the MoPU 12 for the left eye can be mounted on the left eye.

[0285] In addition, when the image captured by the camera 302 is unclear due to dirt adhering to the lens of the camera 302 or dirt adhering to the windshield, there is a risk that the control of the autonomous driving of the vehicle 100 becomes difficult.

[0286] Therefore, when the image captured by the camera 302 is unclear, the central brain 15 can use the image captured by the ultra-high-resolution camera on the IPU 11 side as an alternative to the image captured by the camera 302. The situation where the image captured by the camera 302 is unclear means that although driving outdoors during the day, a specified area of the image captured by the camera 302 appears very dark or the contrast becomes low, etc.

[0287] By using the image captured by the ultra-high-resolution camera on the IPU 11 side as an alternative to the image captured by the camera 302, the central brain 15 can continue to control the autonomous driving of the vehicle 100.

[0288] When the image captured by the camera 302 is unclear, the central brain 15 can perform processing to make the image captured by the camera 302 clear. As such processing, for example, the central brain 15 can spray cleaning liquid and operate the windshield wiper to remove the dirt on the windshield.

[0289] Afterwards, when the image captured by the camera 302 becomes clear, the central brain 15 can return to the process of using the image captured by the camera 302 for the control of the autonomous driving of the vehicle 100.

[0290] The information processing device 10 can acquire an image captured by the camera 302 of another vehicle 100 from the other vehicle 100. In addition, the information processing device 10 can acquire position information included in the point information output from the MoPU 12 of the other vehicle 100 from the other vehicle 100. In this case, the other vehicle 100 that desires to acquire the image and the position information is, for example, present near the vehicle 100 and is traveling in the same direction as the vehicle 100.

[0291] When the information processing device 10 acquires an image captured by the camera 302 of another vehicle 100 (other device image) and position information included in the point information output from the MoPU 12 of the other vehicle 100, the MoPU 12 compares the image captured by the camera 302 of its own vehicle 100 (own device image) with the other device image, and extracts the common range of the two images. Then, the MoPU 12 can use the image of the extracted range and the position information acquired from the other vehicle 100 to acquire the position information included in the point information of the own device image.

[0292] The MoPU 12 extracts the common range of the two images by any method. For example, the MoPU 12 can detect the presence of common subjects (such as people and objects) in the two images, and use the detection result to extract the common range of the two images.

[0293] In addition, the MoPU 12 can synchronize the frame of the own device image with the frame of the other device image in consideration of the transmission time of the image and the position information from the other vehicle 100. By synchronizing the frame of the own device image with the frame of the other device image by the MoPU 12, the common range can be extracted using the two images captured at the same timing. By being able to extract the common range using the two images captured at the same timing, the MoPU 12 can acquire the position information included in the point information of the own device image with high accuracy.

[0294] It should be noted that when the information processing device 10 is provided in a robot, the image and the position information can be acquired from a device having the same structure as the Figure 2 structure shown, for example, a robot present near the robot, instead of from the other vehicle 100.

[0295] (Third Embodiment)

[0296] Next, the third embodiment related to the present embodiment will be described by omitting or simplifying the parts that are repeated with the above embodiments.

[0297] As an example, the information processing device 10 related to the third embodiment has a structure similar to that of the first embodiment Figure 2The structure shown.

[0298] The MoPU 12 according to the third embodiment outputs coordinate values of at least two diagonal points among the vertices of a polygon that encloses the contour of an object identified from an image captured by another camera (an image obtained by capturing with another camera), that is, a plurality of coordinate values. Similar to the first embodiment, the coordinate values represented by each of the plurality of coordinate values are the x coordinate value and the y coordinate value of the object in the above three-dimensional orthogonal coordinate system. The plurality of coordinate values is an example of "second coordinate values".

[0299] Figure 4 It is an explanatory diagram showing an example of the point information output by the MoPU 12 according to the third embodiment. In Figure 4 For each of the four objects included in an image captured by another camera (an image obtained by capturing with another camera), the MoPU 12 shows bounding boxes 21, 22, 23, and 24 formed by enclosing the contour of the object with a quadrilateral. And, Figure 4 An example of a manner in which the MoPU 12 outputs, as a plurality of coordinate values, the coordinate values of two diagonal points among the vertices of the bounding boxes 21, 22, 23, and 24 of the quadrilateral that encloses the contour of the object is illustrated. In this way, the MoPU 12 can capture the object not as a point but as an object having a certain size.

[0300] Furthermore, in the case of capturing the object as an object having a certain size, the MoPU 12 may output, as a plurality of coordinate values, the coordinate values of the vertices of a polygon that encloses the contour of the object, instead of outputting, as point information, the coordinate values of two diagonal points among the vertices of a polygon that encloses the contour of an object identified from an image captured by another camera (an image obtained by capturing with another camera). For example, taking Figure 4 as an example, the MoPU 12 outputs the coordinate values of all four vertices of the bounding boxes 21, 22, 23, and 24 that enclose the contour of the object with a quadrilateral as a plurality of coordinate values.

[0301] In the above description, an example is given where the MoPU 12 outputs multiple coordinate values as point information based on an image of an object captured by another camera. However, the disclosed technology is not limited to this method. The MoPU 12 can switch and output a single coordinate value or the aforementioned multiple coordinate values as point information from an image of an object captured by another camera at a specified timing. The single coordinate value represents the existence position of the center point or the centroid of the object on at least two coordinate axes (e.g., the x-axis and the y-axis) that constitute the three-dimensional orthogonal coordinate system. The single coordinate value is an example of the "first coordinate value".

[0302] For example, as the specified timing, when the IPU 11 cannot recognize the object captured from the image of the object captured by the ultra-high resolution camera due to a specified factor, the MoPU 12 switches the coordinate value output as point information from a single coordinate value to multiple coordinate values. Here, when the IPU 11 cannot recognize the category of the captured object and cannot output label information to the central brain 15, the IPU 11 outputs indication information to the MoPU 12, and this indication information switches the coordinate value output as point information. As an example, when the MoPU 12 obtains the indication information, it regards that the IPU 11 cannot recognize the captured object and switches the coordinate value output as point information from a single coordinate value to multiple coordinate values.

[0303] With the above structure, for example, when the IPU 11 cannot recognize the captured object due to the influence of darkness or bad weather as a specified factor, by switching the coordinate value output as point information to multiple coordinate values, the MoPU 12 can be responsible for the task of grasping the size of the object instead of the IPU 11.

[0304] In addition, as the specified timing, when the moving speed of the object is below a specified threshold value, or when the moving direction of the object is in a specified direction, the MoPU 12 switches the coordinate value output as point information from multiple coordinate values to a single coordinate value.

[0305] The MoPU 12 calculates a speed score, which is a score related to the speed of the object reflected by another camera. As an example, the speed score is set such that the faster the moving speed of the object, the higher the speed score is set, and the slower the moving speed of the object, the lower the speed score is set. Moreover, when the speed score calculated due to the moving speed of the object being below the specified threshold value is less than the threshold value, the MoPU 12 switches the coordinate value output as point information from multiple coordinate values to a single coordinate value.

[0306] In addition, the MoPU 12 calculates a direction score, which is a score related to the moving direction of an object imaged by other cameras. As an example, the direction score is set such that the direction score increases when the moving direction of the object is a direction approaching the road, and the direction score decreases when the moving direction is a direction away from the road. Moreover, when the direction score calculated because the moving direction of the object is a direction away from the road as a specified direction is less than a threshold value, the MoPU 12 switches the coordinate values output as point information from multiple coordinate values to a single coordinate value.

[0307] With the above structure, when the risk level indicating the degree to which the vehicle 100 will travel in a dangerous location in the future is low, by switching the coordinate values output as point information to a single coordinate value, it is possible to reduce the processing load of the MoPU 12 and at the same time reduce the amount of data output to the central brain 15.

[0308] (Fourth Embodiment)

[0309] Next, while omitting or simplifying parts that overlap with the foregoing embodiments, the fourth embodiment related to this embodiment will be described.

[0310] As an example, the information processing device 10 related to the fourth embodiment has a structure similar to that of the first embodiment Figure 2 as shown.

[0311] The vehicle 100 equipped with the information processing device 10 related to the fourth embodiment includes a sensor composed of at least one of a radar, a LiDAR, a high-pixel / long-focus / ultra-wide-angle / 360-degree / high-performance camera, a vision sensor, a sound sensor, an ultrasonic sensor, a vibration sensor, an infrared sensor, a ultraviolet sensor, a radio wave sensor, a temperature sensor, and a humidity sensor. Examples of the sensor information obtained by the information processing device 10 from the sensor include the movement of the center of gravity of the weight, the detection of the road material, the detection of the external air temperature, the detection of the external air humidity, the detection of the up / down / horizontal / oblique inclination angles of the slope, the freezing mode of the road, the moisture content detection, the material, wear condition, and air pressure of each tire, the road width, the presence or absence of overtaking prohibition, the vehicle types of oncoming vehicles and vehicles in front and behind, the cruising states of these vehicles, or the surrounding conditions (birds, animals, footballs, accident vehicles, earthquakes, fires, winds, typhoons, heavy rains, light rains, snowstorms, fog, etc.). The sensor is an example of the "detection unit", and the sensor information is an example of the "detection information".

[0312] The central brain 15 according to the fourth embodiment calculates control variables for controlling the autonomous driving of the vehicle 100 based on the sensor information detected by the sensors. The central brain 15 acquires the sensor information every one billionth of a second. Specifically, the central brain 15 calculates control variables for the wheel speed, tilt, and suspension supporting the wheels of each of the four wheels of the vehicle 100. It should be noted that the tilt of the wheel includes both the tilt of the wheel with respect to the axis horizontal to the road and the tilt of the wheel with respect to the axis perpendicular to the road. In this case, the central brain 15 calculates a total of 16 control variables, and the total 16 control variables are used to control the wheel speed of each of the four wheels, the tilt of each of the four wheels with respect to the axis horizontal to the road, the tilt of each of the four wheels with respect to the axis perpendicular to the road, and the suspensions respectively supporting the four wheels.

[0313] Moreover, the central brain 15 controls the autonomous driving of the vehicle 100 based on the control variables calculated above, the point information output from the MoPU 12, and the label information output from the IPU 11. Specifically, the central brain 15 controls the in-wheel motors respectively mounted on the four wheels based on the above 16 control variables, thereby controlling the wheel speed, tilt, and suspensions respectively supporting the four wheels of the vehicle 100 to perform autonomous driving. In addition, the central brain 15 identifies the positions and movements of the objects that are identified as what in the surroundings of the vehicle 100 based on the point information and the label information, and controls the autonomous driving of the vehicle 100 based on the information obtained by the identification to avoid, for example, colliding with the objects. In this way, the central brain 15 controls the autonomous driving of the vehicle 100 so that, for example, when the vehicle 100 is traveling on a mountain road, it can perform the optimal steering integrated with the mountain road, and when the vehicle 100 is parked in a parking lot, it can travel at the optimal angle integrated with the parking lot.

[0314] Here, the central brain 15 can be a unit that can utilize machine learning, more specifically, use deep learning, to infer control variables based on the above sensor information and information that can be obtained via a network from a server (not shown) or the like. In other words, the central brain 15 can be composed of AI.

[0315] The central brain 15 is the computing power for the above sensor information and long-tail event AI data per billionth of a second. Using the computing power employed at implementation level 6 (hereinafter also referred to as "the computing power of level 6"), by performing multivariate analysis based on the integral method shown in the following formula (1) (for example, refer to formula (2)), the control variables can be obtained. More specifically, while obtaining the integral values of various ultra-high-resolution Delta values with the computing power of level 6, the control variables can be obtained at the edge level and in real time, and the results (i.e., the control variables) that will occur in the next billionth of a second can be obtained with the highest probability value. To achieve this, the integral value obtained by performing time integration on the Delta value (for example, the change value in a tiny time) of a function that can determine various variables such as air resistance, road resistance, road elements (such as garbage), and slip coefficient (for example, the above sensor information and information that can be obtained via the network) (in other words, a function representing the changes in various variables) is input to the deep learning model of the central brain 15 (for example, a learned model obtained by performing deep learning on a neural network). The deep learning model of the central brain 15 outputs the control variable corresponding to the input integral value (for example, the control variable with the highest confidence level (i.e., evaluation value)). The output of the control variable is carried out in units of billionth of a second.

[0316] [Calculation formula 1]

[0317]

[0318] [Calculation formula 2]

[0319] V n = DL(f(A, B, C, D, …, N)(dA n / dt)) (2)

[0320] It should be noted that, as an example, in formula (1), "f(A)" is a formula obtained by simplifying a function representing the changes of various variables such as air resistance, road resistance, road elements (such as garbage), and slip coefficient. In addition, as an example, formula (1) is a formula representing the time integral v of "f(A)" from time a to time b. DL in formula (2) represents deep learning (for example, a deep learning model optimized by deep learning of a neural network), dAn / dt represents the Delta value of f(A,B,C,D,...,N), A, B, C, D,..., N represent various variables such as air resistance, road resistance, road elements (such as garbage), and slip coefficient, f(A,B,C,D,...,N) represents a function used to represent the changes of A, B, C, D,..., N, and Vn represents the value (control variable) output from a deep learning model optimized by deep learning of a neural network.

[0321] It should be noted that an example of the method of inputting the integral value obtained by time-integrating the Delta value of a function into the deep learning model of the central brain 15 is listed here, but this is just an example. It can also be set that, for example, the deep learning model of the central brain 15 infers the integral value obtained by time-integrating the Delta value of a function representing the changes of various variables such as air resistance, road resistance, road elements, and slip coefficient (for example, the result occurring in the next billionth of a second), and the central brain 15 obtains the integral value with the highest confidence level (that is, the evaluation value) per billionth of a second as the inference result.

[0322] In addition, examples of the method of inputting the integral value into the deep learning model or outputting the integral value from the deep learning model are listed here, but this is just an example. Even if the integral value is not used, the technology of the present disclosure is still valid. For example, it can also be that at least one control variable is inferred by using a deep learning model optimized by deep learning of a neural network using the following supervised data. The supervised data uses the values equivalent to A, B, C, D,..., N as example data and the values equivalent to at least one control variable (for example, the result occurring in the next billionth of a second) as correct answer data.

[0323] The control variable obtained by the central brain 15 can be further refined by increasing the number of times of deep learning. For example, a large amount of data or long-tail event AI data such as the rotation of tires or motors, steering angles, road materials, weather, the influence of garbage or quadratic curve deceleration, slip, and the methods of steering or speed control for losing balance or restoring balance can be used to calculate more accurate control variables.

[0324] (Fifth Embodiment)

[0325] Next, while omitting or simplifying parts that overlap with the foregoing embodiments, the fifth embodiment related to the present embodiment will be described.

[0326] Figure 5 FIG. is a block diagram showing an example of the structure of the information processing apparatus 10 related to the fifth embodiment. It should be noted that Figure 5 only a part of the structure of the information processing apparatus 10 is shown.

[0327] As Figure 5 shown, by the MoPU 12, the visible light image and the infrared image of the object captured by the camera 30 are respectively input to the core 17 at a frame rate of 100 frames per second or more. The camera 30 is configured to include a visible light camera 305A capable of capturing a visible light image of an object and an infrared camera 305B capable of capturing an infrared image of an object. Further, the core 17 outputs point information to the central brain 15 based on at least one of the input visible light image and infrared image.

[0328] Here, when an object can be recognized from the visible light image of the object captured by the visible light camera 305A, the core 17 outputs point information based on the visible light image. On the other hand, when the object cannot be captured from the visible light image due to a specified factor, the core 17 outputs point information based on the infrared image of the object captured by the infrared camera 305B. For example, consider a case where the core 17 cannot capture an object from the visible light image due to the influence of darkness as a specified factor. In this case, the core 17 detects the heat of the object using the infrared camera 305B and outputs the point information of the object based on the infrared image as the detection result. It should be noted that, without limitation, the core 17 may also output point information based on the visible light image and the infrared image.

[0329] In addition, the MoPU 12 synchronizes the timing of capturing a visible light image by the visible light camera 305A (capturing for obtaining a visible light image by the visible light camera 305A) with the timing of capturing an infrared image by the infrared camera 305B (capturing for obtaining an infrared image by the infrared camera 305B). Specifically, the MoPU 12 outputs a control signal to the camera 30 so that a visible light image and an infrared image are captured at the same timing (the MoPU 12 performs visible light capture and infrared capture). Thereby, the number of images per second captured by the visible light camera 305A (the number of images per second obtained by capturing with the visible light camera 305A) is synchronized with the number of images per second captured by the infrared camera 305B (the number of images per second obtained by capturing with the infrared camera 305B) (for example, 1920 frames per second).

[0330] (Sixth Embodiment)

[0331] Next, while omitting or simplifying parts that overlap with the foregoing embodiments, the sixth embodiment related to the present embodiment will be described.

[0332] Figure 6 It is a block diagram showing an example of the structure of the information processing apparatus 10 related to the sixth embodiment. It should be noted that Figure 6 only a part of the structure of the information processing apparatus 10 is shown.

[0333] As Figure 6 shown, through the MoPU 12, an image of an object captured by the camera 30 and a radar signal based on a reflected wave obtained by reflecting an electromagnetic wave irradiated onto the object by the radar 32 are respectively input to the core 17 at a frame rate of 100 frames per second or more. And, based on the input image of the object and the radar signal, the core 17 outputs point information to the central brain 15. The core 17 can derive the x coordinate value and y coordinate value of the object from the input image of the object. As described above, the radar 32 can obtain three-dimensional point cloud data of the object based on the radar signal and detect the coordinate of the z axis in the above-mentioned three-dimensional orthogonal coordinate system. In this case, the core 17 uses the principle of a stereo camera to combine the x coordinate value and y coordinate value of the object captured by the camera 30 at the same timing as when the radar 32 obtains the three-dimensional point cloud data of the object, and the z coordinate value of the object represented by the three-dimensional point cloud data, and derives the coordinate values of the three axes (x axis, y axis, and z axis) of the object as point information. It should be noted that the image of the object input to the core 17 described above may also include at least one of a visible light image and an infrared image.

[0334] In addition, the MoPU 12 synchronizes the timing of capturing an image by the camera 30 (the camera 30 captures an image) with the timing of the radar 32 obtaining three-dimensional point cloud data of the object based on the radar signal. Specifically, the MoPU 12 captures images at the same timing and outputs control signals to the camera 30 and the radar 32 to obtain three-dimensional point cloud data of the object. Thereby, the number of images per second captured by the camera 30 (the number of images per second obtained by the camera 30 capturing an image) is synchronized with the number of three-dimensional point cloud data per second obtained by the radar 32 (for example, 1920 frames per second). In this way, the number of images per second captured by the camera 30 (the number of images per second obtained by the camera 30 capturing an image) and the number of three-dimensional point cloud data per second obtained by the radar 32 are greater than the frame rate of the ultra-high-resolution camera provided in the IPU 11, that is, the number of images per second captured by the ultra-high-resolution camera (the number of images per second obtained by the ultra-high-resolution camera capturing an image).

[0335] (Seventh Embodiment)

[0336] Next, while omitting or simplifying the parts that overlap with the foregoing embodiments, the seventh embodiment related to the present embodiment will be described.

[0337] As an example, the information processing apparatus 10 related to the seventh embodiment has a structure similar to that of the first embodiment Figure 2 as shown.

[0338] The central brain 15 related to the seventh embodiment associates the point information output from the MoPU 12 at the same timing as the timing when the IPU 11 outputs the label information with the label information. In addition, when new point information is output from the MoPU 12 after the point information and the label information are associated, the central brain 15 also associates the new point information with the label information. The new point information is the point information of the same object as the object represented by the point information after being associated with the label information, and is one or more pieces of point information during the period from when this association is made until the next label information is output. In the seventh embodiment, similar to the above embodiment, the frame rate of other cameras with the MoPU 12 built-in is 100 frames per second or more (for example, 1920 frames per second), and the frame rate of the ultra-high-resolution camera with the IPU 11 built-in is 10 frames per second.

[0339] Fig. 7A It is an explanatory diagram showing an example of the association between the point information and the label information related to the seventh embodiment. In the following description, the number of pieces of point information output per second from the MoPU 12 is referred to as the "output rate of point information", and the number of pieces of label information output per second from the IPU 11 is referred to as the "output rate of label information".

[0340] Fig. 7A It shows the time series of the output rate of the point information P4 of the object B14. The output rate of the point information P4 related to the object B14 is 1920 frames per second. In addition, the point information P4 moves from right to left in the figure. The output rate of the label information related to the object B14 is 10 frames per second, which is lower than the output rate of the point information P4.

[0341] First, at the time point of time t0, the label information related to the object B14 has not been output from the IPU 11 yet. Therefore, at the time point of time t0, the central brain 15 identifies the coordinate value (position information) of the object B14 based on the point information P4, but does not identify what the object B14 is.

[0342] Next, at the time point of time t1, the tag information related to the object B14 is output from the IPU 11. Therefore, based on this tag information, the central brain 15 derives the tag information "PERSON" for the object B14. And the central brain 15 associates the tag information "PERSON" derived at time t1 with the coordinate value (position information) of the point information P4 output from the MoPU 12 at time t1. Thus, at the time point of time t1, the central brain 15 identifies the coordinate value (position information) of the object B14 based on the point information P4, and also identifies what the object B14 is.

[0343] At Fig. 7A this time, the timing at which the next tag information related to the object B14 is output from the IPU 11 is taken as time t2. Therefore, at the time point of time t2, the central brain 15 derives the tag information "PERSON" for the object B14 based on this tag information output from the IPU 11. And the central brain 15 associates the tag information "PERSON" derived at time t2 with the coordinate value (position information) of the point information P4 output from the MoPU 12 at time t2.

[0344] Here, due to the frame rate difference between the other camera with the MoPU 12 built in and the ultra-high resolution camera with the IPU 11 built in, during the period from time t1 to time t2, the central brain 15 acquires the point information P4 related to the object B14, while the tag information is not acquired. In this case, for the point information P4 acquired during the period from time t1 to time t2, the central brain 15 associates the point information P4 with the tag information "PERSON" associated with the upcoming time t1. Here, the point information P4 acquired by the central brain 15 during the period from time t1 to time t2 is an example of "new point information". In Fig. 7A the example shown, since a plurality of point information P4 is output from the MoPU 12 during the period from time t1 to time t2, the central brain 15 acquires a plurality of point information P4. Therefore, in Fig. 7A the example shown, for any one of the plurality of point information P4 acquired during the period from time t1 to time t2, the central brain 15 associates it with the tag information "PERSON" associated with the upcoming time t1. It should be noted that, different from Fig. 7A the example shown, in the case where one point information P4 is output from the MoPU 12 during the period from time t1 to time t2, for this one point information P4, the central brain 15 associates it with the tag information "PERSON" associated with the upcoming time t1.

[0345] Here, regarding the central brain 15, since the point information of the object being tracked is continuously output at a high frame rate even during a period when the type of the object being tracked for the ongoing tracked motion is uncertain, the risk of losing the coordinate value (position information) of the object is low. Therefore, in the case where the association between the point information and the label information has been performed once, the central brain 15 can presumptively assign the previous label information to the point information acquired during the period until the next label information is obtained.

[0346] The central brain 15 derives the speed and acceleration of the object based on the time series of the coordinate values of the point information. In addition, the central brain 15 predicts the motion of the object based on the time series of the coordinate values of the point information. In the prediction of the motion of the object, the label information of the object can be considered.

[0347] Here, the coordinate values of the label information and the point information that are associated with each other must be related to the same object. That is, it is necessary to avoid associating the label information of object A with the coordinate values of the point information of an object B different from object A. The following conditions 1 to 3 are the conditions for the object of the label information and the object of the coordinate values of the point information to be the same object.

[0348] <Condition 1>

[0349] From the installation positions (viewpoint differences) of the camera equipped with the IPU 11 and the camera equipped with the MoPU 12, the relative positional relationship between the high-resolution image output from the IPU 11 (hereinafter referred to as the IPU image) and the image captured by the camera equipped with the MoPU 12 (hereinafter referred to as the MoPU image) can be determined. When the positional relationship between the object reflected in the IPU image and the object reflected in the MoPU image is integrated, there is a possibility that the two objects are the same object. Therefore, the integration of the positional relationship between the object reflected in the IPU image and the object reflected in the MoPU image is set as Condition 1 for the two objects to be the same object. "Integration of the positional relationship" means that, considering the viewpoint difference between the IPU image and the MoPU image, it can be regarded that the objects reflected in the two images are located at the same position.

[0350] <Condition 2>

[0351] Although the resolution of the MoPU image is low, the outline of the object can be recognized. When the outline of the object reflected in the IPU image is integrated with the outline of the object reflected in the MoPU image, there is a possibility that the two objects are the same object. Therefore, integrating the outline of the object reflected in the IPU image with the outline of the object reflected in the MoPU image is set as the second condition for the two objects to be the same object. "Outline integration" means that the similarity between the outline of the object reflected in the IPU image and the outline of the object reflected in the MoPU image is above a certain level. For example, when the outline of the object reflected in the MoPU image is superimposed on the outline of the object reflected in the IPU image considering the position deviation, and the ratio of the consistent positions of the outlines of the objects in the two images is above a specified threshold, it can be determined that the similarity of the outlines of the objects reflected in the two images is above a certain level.

[0352] <Condition 3>

[0353] From the movement of the point information of the object captured by the MoPU 12, it is possible to estimate to some extent what the object is. For example, the movement of a person is significantly different from that of a car. Buildings such as signs and signals do not move at all. When the category of the object estimated from the movement of the point information of the object captured by the MoPU 12 is integrated with the category (label information) of the object estimated from the IPU image, there is a possibility that the two objects are the same object. Therefore, integrating the category of the object estimated from the movement of the point information of the object captured by the MoPU with the category (label information) of the object estimated from the IPU image is set as the third condition for the two objects to be the same object. "Integration of object categories" means that the category of the object estimated from the movement of the point information of the object captured by the MoPU 12 does not conflict with the category (label information) of the object estimated from the IPU image.

[0354] When all the foregoing Conditions 1 to 3 are satisfied, the central brain 15 determines that the object of the label information and the object of the coordinate value of the point information are the same object, and associates the point information with the label information for the two objects. Figure 7B It is a flowchart showing an example of the process of associating point information with label information implemented by the central brain 15 according to the seventh embodiment.

[0355] In step S1, the central brain 15 extracts objects that satisfy condition 1, i.e., objects with integrated position relationships, from the objects reflected in the IPU image and the objects reflected in the MoPU image. In step S2, the central brain 15 extracts objects that also satisfy condition 2, i.e., objects with integrated contours, from the objects that satisfy condition 1 among the objects reflected in the IPU image and the objects reflected in the MoPU image. In step S3, the central brain 15 extracts objects that also satisfy condition 3, i.e., objects with integrated object categories, from the objects that satisfy both condition 1 and condition 2 among the objects reflected in the IPU image and the objects reflected in the MoPU image. In step S4, the central brain 15 associates the coordinate values of the point information with the label information for the objects extracted in step S3, i.e., the objects that satisfy all of conditions 1 to 3. It should be noted that in the above description, an example is given where when all of conditions 1 to 3 are satisfied, it is determined that the object is the same object. However, it is also possible to determine that an object that satisfies one or two of conditions 1 to 3 is the same object, and for that object, associate the coordinate values of the point information with the label information. In this case, at least one of conditions 1 to 3 can be set as a necessary condition. For example, setting condition 3 (integration of label information) as a necessary condition, even if the other two conditions are not satisfied, it can be determined that the object is the same object. Additionally, based on the number of conditions satisfied, the reliability of the identity of the object can be derived and used in vehicle control. For example, vehicle control corresponding to the object can be performed when all of conditions 1 to 3 are satisfied.

[0356] (Eighth Embodiment)

[0357] Next, while omitting or simplifying parts that overlap with the foregoing embodiments, the eighth embodiment related to the present embodiment will be described.

[0358] When the information processing device 10 that controls the autonomous driving of the vehicle 100 performs high-level arithmetic processing, heat generation becomes an issue. Therefore, the eighth embodiment provides a vehicle 100 having a cooling function for the information processing device 10.

[0359] Figure 8 is an explanatory diagram showing a schematic structure of the vehicle 100 related to the eighth embodiment. As Figure 8 shown, the vehicle 100 is equipped with an information processing device 10, a cooling execution device 110, and a cooling unit 120.

[0360] The information processing device 10 of the eighth embodiment is a device that controls the autonomous driving of the vehicle 100. As an example, the information processing device 10 has a structure similar to that of the first embodiment Figure 2The structure shown. The cooling execution device 110 obtains the detection result of an object based on the information processing device 10, and based on this detection result, causes the cooling unit 120 to perform cooling of the information processing device 10. The cooling unit 120 cools the information processing device 10 using at least one cooling mechanism such as an air cooling mechanism, a water cooling mechanism, and a liquid nitrogen cooling mechanism. Hereinafter, the object to be cooled in the information processing device 10 is set as the central brain 15 that controls the autonomous driving of the control vehicle 100 (specifically, the CPU 14 that constitutes the central brain 15) for explanation, but it is not limited thereto.

[0361] The information processing device 10 and the cooling execution device 110 are communicably connected via a network (not shown). This network can be any one of a vehicle network, the Internet, a local area network (LAN), and a mobile communication network. The mobile communication network can be based on any one of communication methods such as 5G (5th Generation), LTE (Long Term Evolution), 3G (3rd Generation), and 6G (6th Generation) and subsequent communication methods.

[0362] Fig. 9 It is a block diagram showing an example of the functional structure of the cooling execution device 110 according to the eighth embodiment. As Fig. 9 shown, the cooling execution device 110 has an acquisition unit 112, an execution unit 114, and a prediction unit 116 as functional structures.

[0363] The acquisition unit 112 obtains the detection result of an object based on the information processing device 10. For example, as this detection result, the acquisition unit 112 obtains the point information of the object output from the MoPU 12.

[0364] The execution unit 114 performs cooling of the central brain 15 based on the detection result of the object acquired by the acquisition unit 112. For example, when it is recognized from the point information of the object output from the MoPU 12 that the object is moving, the execution unit 114 causes the cooling unit 120 to start cooling the central brain 15.

[0365] It should be noted that the execution unit 114 is not limited to performing cooling of the central brain 15 based on the detection result of the object, and may also perform cooling of the central brain 15 based on the prediction result of the operating condition of the information processing device 10.

[0366] Here, the prediction unit 116 predicts the operating condition of the information processing device 10 based on the detection result of the object acquired by the acquisition unit 112. Specifically, it predicts the operating condition of the central brain 15. For example, the prediction unit 116 acquires a learning model stored in a specified storage area. And the prediction unit 116 predicts the operating condition of the central brain 15 by inputting the point information of the object output from the MoPU 12 acquired by the acquisition unit 112 into the learning model. Here, the learning model outputs the condition and change amount of the computing power of the central brain 15 as this operating condition. In addition, the prediction unit 116 can predict and output the temperature change of the information processing device 10 together with this operating condition. Specifically, it is the temperature change of the central brain 15. For example, the prediction unit 116 predicts the temperature change of the central brain 15 based on the number of point information of the object output from the MoPU 12 acquired by the acquisition unit 112. In this case, the prediction unit 116 predicts that the greater the number of point information, the greater the temperature change, and the smaller the number of point information, the smaller the temperature change.

[0367] In the above case, the execution unit 114 starts the cooling of the central brain 15 by the cooling unit 120 based on the prediction result of the operating condition of the central brain 15 by the prediction unit 116. For example, when the state and change amount of the computing power of the central brain 15 predicted as this operating condition exceed a predetermined threshold, the execution unit 114 starts the cooling by the cooling unit 120. In addition, when the temperature based on the temperature change of the central brain 15 predicted as this operating condition exceeds a specified threshold, the execution unit 114 starts the cooling by the cooling unit 120.

[0368] In addition, the execution unit 114 can also perform the cooling of the central brain 15 using a cooling mechanism corresponding to the prediction result of the temperature change of the central brain 15 by the prediction unit 116. For example, the execution unit 114 can use more cooling mechanisms to make the cooling unit 120 perform cooling when the predicted temperature of the central brain 15 is higher. As a specific example, when it is predicted that the temperature of the central brain 15 exceeds the first threshold, the execution unit 114 uses one cooling mechanism to make the cooling unit 120 perform cooling. On the other hand, when it is predicted that the temperature of the central brain 15 exceeds the second threshold higher than the first threshold, the execution unit 114 uses multiple cooling mechanisms to make the cooling unit 120 perform cooling.

[0369] In addition, the execution unit 114 can use a more powerful cooling mechanism to cool the central brain 15 when the predicted temperature of the central brain 15 is higher. For example, when it is predicted that the temperature of the central brain 15 exceeds the first threshold, the execution unit 114 uses the air cooling mechanism to make the cooling unit 120 perform cooling. In addition, when it is predicted that the temperature of the central brain 15 exceeds the second threshold higher than the first threshold, the execution unit 114 uses the water cooling mechanism to make the cooling unit 120 perform cooling. Further, when it is predicted that the temperature of the central brain 15 exceeds the third threshold higher than the second threshold, the execution unit 114 uses the liquid nitrogen cooling mechanism to make the cooling unit 120 perform cooling.

[0370] Further, the execution unit 114 can determine the cooling mechanism used for cooling based on the number of point information of the object output from the MoPU 12 acquired by the acquisition unit 112. In this case, the execution unit 114 can use a more powerful cooling mechanism as the number of point information is larger to cool the central brain 15. For example, when the number of point information exceeds the first threshold, the execution unit 114 uses the air cooling mechanism to make the cooling unit 120 perform cooling. In addition, when the number of point information exceeds the second threshold higher than the first threshold, the execution unit 114 uses the water cooling mechanism to make the cooling unit 120 perform cooling. Further, when the number of point information exceeds the third threshold higher than the second threshold, the execution unit 114 uses the liquid nitrogen cooling mechanism to make the cooling unit 120 perform cooling.

[0371] As a trigger for the central brain 15 to operate, a moving object existing on the lane is sometimes detected. For example, when a moving object existing on the lane is detected while the vehicle 100 is performing autonomous driving, the central brain 15 sometimes performs arithmetic processing for controlling the vehicle 100 on this object. As described above, the heat generation during the high-level arithmetic processing performed by the central brain 15 that controls the autonomous driving of the vehicle 100 becomes a problem. Therefore, the cooling execution device 110 according to the eighth embodiment predicts the heat dissipation of the central brain 15 based on the detection result of the object by the information processing device 10, and performs cooling of the central brain 15 before or at the same time as the start of heat dissipation. Thereby, it is possible to suppress the central brain 15 from becoming high temperature during the autonomous driving of the vehicle 100, and it is possible to perform high-level arithmetic processing of the central brain 15 during this autonomous driving.

[0372] (Ninth Embodiment)

[0373] Next, while omitting or simplifying the parts that overlap with the foregoing embodiments, the ninth embodiment according to this embodiment will be described.

[0374] The MoPU 12 included in the information processing apparatus according to the ninth embodiment derives the z coordinate value of an object as point information from an image of the object captured by a camera. Hereinafter, each mode of the information processing apparatus 10 according to the ninth embodiment will be described in sequence.

[0375] The information processing apparatus 10 according to the first aspect includes a structure similar to that of the second embodiment Figure 3 as shown.

[0376] In the above first mode, the MoPU 12 derives the z coordinate value of an object as point information from an image of the object captured by a plurality of cameras 302, specifically, from the images of the object captured by the camera 302L and the camera 302R. As described above, when one MoPU 12 is used, the x coordinate value and the y coordinate value of the object can be derived as point information. Here, when two MoPUs 12 are used, based on the principle of a stereo camera, the z coordinate value of the object can be derived as point information from the images of the object captured by the two cameras 302. Therefore, this first mode uses the principle of a stereo camera and derives the z coordinate value of the object as point information from the images of the object captured by the camera 302L of the MoPU 12L and the camera 302R of the MoPU 12R, respectively.

[0377] The information processing apparatus 10 according to the second mode includes a structure similar to that of the second embodiment Figure 3 as shown.

[0378] In the above second mode, the MoPU 12 derives the x coordinate value, the y coordinate value, and the z coordinate value of the object as point information from the image of the object captured by the camera 302 and the radar signal based on the reflected wave of the electromagnetic wave irradiated onto the object by the radar 32 and reflected from the object. As described above, the radar 32 can obtain the three-dimensional point cloud data of the object based on the radar signal. That is, the radar 32 can detect the coordinate of the z axis in the above three-dimensional orthogonal coordinate system. In this case, the MoPU 12 uses the principle of a stereo camera and combines the x coordinate value and the y coordinate value of the object captured by the camera 302 at the same timing as the timing when the three-dimensional point cloud data of the object is obtained by the radar 32 with the z coordinate value of the object represented by the three-dimensional point cloud data, and derives the coordinate values of the three axes of the object as point information.

[0379] The information processing apparatus 10 according to the third mode includes Fig.10 the structure shown. Fig.10 This is a first block diagram showing an example of the structure of the information processing apparatus 10 according to the ninth embodiment. It should be noted that Fig.10 only a part of the structure of the information processing apparatus 10 is shown.

[0380] In the above-described third method, the MoPU 12 derives the z coordinate value of the object as point information based on the image of the object captured by the camera 30 and the result of capturing the structured light irradiated onto the object by the irradiation device 130.

[0381] As Fig.17 shown, by the MoPU 12, the image of the object captured by the camera 30 and the distortion information indicating the distortion of the pattern of the structured light, which is the result of capturing the structured light irradiated onto the object by the camera 140, are input to the core 17 at a frame rate of 100 frames per second or more. And the core 17 outputs the point information to the central brain 15 based on the input object image and distortion information.

[0382] Here, as one of the methods for recognizing the three-dimensional position or shape of an object, there is the structured light method. The structured light method irradiates structured light formed into dots onto an object and obtains depth information based on the distortion of the pattern. The structured light method is disclosed, for example, in the reference (http: / / ex-press.jp / wp-content / uploads / 2018 / 10 / 018_teledyne_3rd.pdf).

[0383] Fig.10 The irradiation device 130 shown irradiates structured light onto the object. In addition, the camera 140 captures the structured light irradiated onto the object by the irradiation device 130. And the camera 140 outputs the distortion information based on the distortion of the pattern of the captured structured light to the core 17.

[0384] Here, the MoPU 12 synchronizes the timing of capturing an image by the camera 30 (when the camera 30 captures an image) with the timing of capturing the structured light by the camera 140. Specifically, the MoPU 12 outputs a control signal to the camera 30 and the camera 140 so that the camera 30 and the camera 140 capture images (capture the object) at the same timing. Thereby, the number of images per second captured by the camera 30 (the number of images per second obtained by capturing with the camera 30) is synchronized with the number of images per second captured by the camera 140 (the number of images per second obtained by capturing with the camera 140) (for example, 1920 frames per second). In this way, the number of images per second captured by the camera 30 (the number of images per second obtained by capturing with the camera 30) and the number of images per second captured by the camera 140 (the number of images per second obtained by capturing with the camera 140) are more than the frame rate of the ultra-high-resolution camera included in the IPU 11, that is, the number of images per second captured by the ultra-high-resolution camera.

[0385] Further, the core 17 combines the x - coordinate value and y - coordinate value of the object captured by the camera 30 at the same timing as the timing of capturing the structured light by the camera 140, and the distortion information based on the distortion of the pattern of the structured light, and derives the z - coordinate value of the object as point information.

[0386] The information processing apparatus 10 according to the fourth mode includes Fig.11 the structure shown. Fig.11 is a second block diagram showing an example of the structure of the information processing apparatus 10 according to the ninth embodiment. It should be noted that Fig.11 only shows a part of the structure of the information processing apparatus 10.

[0387] Fig.11 The block diagram shown is Figure 2 a block diagram obtained by adding the Lidar sensor 18 to the structure of the block diagram shown. The Lidar sensor 18 is a sensor that acquires point cloud data including an object existing in a three - dimensional space and the road surface on which the vehicle 100 travels. The information processing apparatus 10 can derive the position information in the depth direction of the object, that is, the z - coordinate value of the object, by using the point cloud data acquired by the Lidar sensor 18. It should be noted that it is assumed that the point cloud data obtained by the Lidar sensor 18 is acquired at an interval longer than the x - coordinate value and y - coordinate value of the object output from the MoPU 12. In addition, the MoPU 12 is provided with the camera 30 similarly to the third aspect of the ninth embodiment.

[0388] In the fourth mode, the MoPU 12 uses the principle of a stereo camera, combines the x - coordinate value and y - coordinate value of the object captured by the camera 30 at the same timing as the timing of acquiring the point cloud data of the object by the Lidar sensor 18, and the z - coordinate value of the object represented by the point cloud data, and derives the coordinate values of the three axes of the object as point information.

[0389] Here, in the above - mentioned fourth mode, the MoPU 12 derives the z - coordinate value of the object at time t + 1 as point information based on the x - coordinate value, y - coordinate value, and z - coordinate value of the object at time t and the x - coordinate value and y - coordinate value of the object at the next time point (for example, time t + 1) after time t. Time t is an example of the "first time point", and time t + 1 is an example of the "second time point". In the fourth mode, the z - coordinate value of the object at time t + 1 is derived using shape information, that is, geometric shape. The following will explain this in detail.

[0390] Fig. 12A is a diagram schematically showing the coordinate detection of an object in time series in the ninth embodiment. In Fig. 12A J represents the position of the object represented by a rectangle, and the position of the object moves in time series from J1 to J2. In Fig. 12A Among them, the coordinate values of the object at time t when the object is located at J1 are (x1, y1, z1), and the coordinate values of the object at time t + 1 when the object is located at J2 are (x2, y2, z2).

[0391] First, an explanation of time t is given.

[0392] The MoPU 12 derives the x - coordinate value and y - coordinate value of the object from the image of the object captured by the camera 30. Then, the MoPU 12 integrates the z - coordinate value of the object represented by the point cloud data obtained from the Lidar sensor 18 and the above - mentioned x - coordinate value and y - coordinate value, and derives the three - dimensional coordinate value (x1, y1, z1) of the object at time t.

[0393] Next, an explanation of time t + 1 is given.

[0394] The MoPU 12 derives the z - coordinate value of the object at time t + 1 based on the geometric shape of the space and the changes in the x - coordinate value and y - coordinate value of the object from time t to time t + 1. The geometric shape of the space includes the shape of the road surface and the shape of the vehicle 100 obtained from the image captured by the ultra - high - resolution camera provided in the IPU 11 (the image obtained by shooting with the ultra - high - resolution camera) and the point cloud data of the Lidar sensor.

[0395] The geometric shape representing the shape of the road surface is generated in advance at the time point of time t. The MoPU 12 combines and utilizes the geometric shape representing the shape of the vehicle 100 and the geometric shape representing the shape of the road surface, so as to be able to simulate the situation of the vehicle 100 driving on the road surface and estimate the movement amounts of the respective x - axis, y - axis, and z - axis.

[0396] Therefore, the MoPU 12 derives the x - coordinate value and y - coordinate value of the object at time t + 1 from the image of the object captured by the camera 30. The MoPU 12 simulates and calculates the movement amount of the z - axis when the x - coordinate value and y - coordinate value of the object change from (x1, y1) at time t to (x2, y2) at time t + 1, so as to be able to derive the z - coordinate value of the object at time t + 1. And the MoPU 12 integrates the above - mentioned x - coordinate value and y - coordinate value with the z - coordinate value, and derives the three - dimensional coordinate value (x2, y2, z2) of the object at time t + 1.

[0397] As Fig. 12AAs shown, since the object moves in the depth direction together with the movement of the planar coordinates (i.e., the x-axis and the y-axis), in order to precisely control the autonomous driving of the vehicle 100, it is also necessary to detect the movement in the z-axis direction. Here, sometimes MoPU 12 cannot obtain the z coordinate value of the object as fast as the x coordinate value and the y coordinate value of the object that can be derived from the point cloud data of the Lidar sensor 18. Therefore, in the above fourth method, it is assumed that MoPU 12 derives the z coordinate value of the object at time t + 1 based on the x coordinate value, y coordinate value, and z coordinate value of the object at time t and the x coordinate value and y coordinate value of the object at time t + 1. Thus, according to the information processing device 10 related to the above fourth method, through MoPU 12, two-dimensional motion detection and three-dimensional motion detection based on high-speed frame shooting can be achieved with high performance and low-usage data.

[0398] In addition, in the case where a plurality of point information is detected, MoPU 12 can derive the z coordinate value of the object at the next time point according to the priority determined for the label information associated with the point information. For example, in the central brain 15, the point information and the label information are associated, but for the object represented by the label information, the priority when processing the z coordinate value is determined in advance. Hereinafter, an example of the priority will be described. For example, in the case where the label information indicating the category of the object is an object that generates behavior such as a person, a vehicle ahead, an animal, or an obstacle, and the behavior of the object is an object that affects driving, a category with a high priority (rank: A) is set. On the other hand, in the case where the label information is a static object such as an identifier or a roadside installation, that is, an object that does not generate behavior, a low priority (rank: B) is set. In this way, through MoPU 12, the priority can be set according to the presence or absence of the behavior of the object in the label information. In addition, the magnitude of the behavior of the object can be obtained from the central brain 15, the behavior of the object can be weighted, and the priority of the object can be updated. For example, the priority is updated so that the weight is increased for an object with a large behavior and the weight is decreased for an object with a small behavior.

[0399] In addition, the levels of priority are not limited to A and B, and can be from A to C or from A to D, etc., which are determined by appropriately setting stages according to the categories of objects. Additionally, within the aforementioned high-priority levels, sub-priorities can be set. For example, a person and a vehicle ahead can also be understood as objects with a high risk of causing major accidents. Therefore, for the high-priority level: A, it is divided into level: A1 and level: A2. A person and a vehicle ahead are set to level: A1, and animals or obstacles, etc. are set to level: A2. In this way, priorities corresponding to the risk factors of the objects can be set. Additionally, among the vehicles ahead, large trucks or trucks loaded with goods, etc. can be said to have a higher risk than ordinary vehicles. Therefore, they can be subdivided and identified using the tag information, and in the MoPU 12, a higher priority than that of ordinary vehicles can be set.

[0400] In addition, it is possible to determine whether to set the processing order using priorities based on the number of objects (i.e., the number of tag information associated with the point information). For example, when the number of objects is equal to or more than a specified number (such as three, etc.), the z coordinate values of each object are derived according to the priority order. When the number of objects is less than the specified number, the z coordinate values of each object are processed without determining the processing order. Additionally, it is possible to determine whether to set the processing order using priorities based on the distance between objects and the density of the objects.

[0401] In addition, the priority is not limited to the case of deriving the z coordinate value of the object as the point information in the ninth embodiment, and can also be applied to the cases of the aforementioned first to eighth embodiments. When there are multiple pieces of point information, the MoPU 12 can derive the point information at the next time point according to the priority determined for the aforementioned tag information.

[0402] In addition, the MoPU 12 can determine the update frequency of the z coordinate value based on the change amounts of the x coordinate value and the y coordinate value, and derive the z coordinate value according to this frequency. For example, when the change amounts of the x coordinate value and the y coordinate value are small, it is assumed that the driving environment is a flat road or a road with good visibility, and the speed of autonomous driving is slow and the changes in autonomous driving behaviors are few. In such a case, it is considered that obtaining only the x coordinate value and the y coordinate value is sufficient to capture an object. Therefore, when the change amounts of the x coordinate value and the y coordinate value are below the threshold, the derivation frequency of the z coordinate value is set to a low frequency, and when the x coordinate value and the y coordinate value are above the threshold, the derivation frequency of the z coordinate value is set to a high frequency. In this way, the update frequency is set for the derivation of the z coordinate value. The change amounts of the x coordinate value and the y coordinate value are an example of "prescribed conditions for an image", and the update frequency determined according to the change amounts is an example of "each time point satisfying the prescribed conditions". It should be noted that the point information per second is set to include the x coordinate value and the y coordinate value of 100 frames or more, but the average value of them can be used at each time point. In addition, the interval between time t and time t + 1 is a matter that can be appropriately designed.

[0403] Fig. 12B It is a flowchart in the case of determining the update frequency of the z coordinate value according to the change amount. It should be noted that it is set to derive at least the three-dimensional coordinate values (x1, y1, z1) at time t as the first time point and the x coordinate value and the y coordinate value at time t + 1.

[0404] In step S300, the MoPU 12 acquires the x coordinate value and the y coordinate value at time t, and the x coordinate value and the y coordinate value at time t + 1.

[0405] In step S302, the MoPU 12 determines whether the change amounts of the x coordinate value and the y coordinate value from time t to time t + 1 are above the threshold. If they are above the threshold, it transfers to step S304, and if they are less than the threshold, it transfers to step S306.

[0406] In step S304, the MoPU 12 sets the update frequency of the z coordinate value to a high frequency.

[0407] In step S306, the MoPU 12 sets the update frequency of the z coordinate value to a low frequency.

[0408] In step S308, when it is the update timing, the MoPU 12 derives the z coordinate value at this time according to the set frequency. It should be noted that regardless of the execution of the foregoing steps S300 to S306, when it becomes the update timing of the set frequency, the process of S308 is executed to derive the z coordinate value.

[0409] Thus, by reducing the processing load and deriving the z - coordinate value according to the driving condition, it is possible to accurately determine the position of an object when necessary. It should be noted that the setting of the update frequency can be applied to the case of deriving the z - coordinate value from the image in the foregoing first to fourth aspects.

[0410] In addition, when there is a difference between the estimated movement amount of the z - axis and the z - coordinate value detected by the lidar sensor 18, the MoPU 12 can learn correction information. For example, collect sensor information when the difference between the estimated z - coordinate value and the detected z - coordinate value is above a threshold, and learn the correction information for the depth estimation of the object from the data. From the sensor information, determine the driving condition of the correction target of the vehicle 100, and learn the correction information for the determined driving condition of the correction target. A method for determining the driving condition of the correction target is, for example, to quantify the characteristics of the sensor information and set the item of the characteristic quantity that differs from the average characteristic quantity as the learning condition. For the learning method, any of the following methods can be used: taking the sensor information of the driving condition of the correction target such as deep learning as input and being able to output correction information. When the input driving condition is suitable for the correction target, the correction information can be learned as an estimation model that outputs a correction candidate z - coordinate value. In addition, it can be configured to verify the accuracy of the corrected z - coordinate value and re - learn the correction information.

[0411] Fig. 12C It is a flowchart in the case of learning correction information. It should be noted that it is assumed that the z - coordinate value, x - coordinate value, and y - coordinate value at time t are integrated at each time point to derive the three - dimensional coordinate value of the object at time t.

[0412] In step S400, the MoPU 12 acquires the estimated z - coordinate value for each time and the z - coordinate value detected by the lidar sensor 18.

[0413] In step S402, the MoPU 12 collects sensor information when the difference between the estimated z - coordinate value and the detected z - coordinate value is above a threshold.

[0414] In step S404, the MoPU 12 determines the driving condition of the correction target from the collected sensor information.

[0415] In step S406, the MoPU 12 uses the sensor information related to the driving condition of the correction target as learning data to learn correction information.

[0416] When estimating the z - coordinate value, the MoPU 12 applies the learned correction information, outputs a correction candidate z - coordinate value, and corrects the z - coordinate value.

[0417] In this way, even in a driving situation where the accuracy is unstable, calibration information can be used to determine the position of an object. It should be noted that the learning of calibration information can be applied to the case of deriving the z coordinate value from an image in the foregoing first to fourth aspects.

[0418] In addition, in the above description, an example is given where MoPU 12 derives the z coordinate value of an object as point information from an image of the object captured by camera 30, but the disclosed technology is not limited to this method. For example, the central brain 15 can derive the z coordinate value of an object as point information instead of MoPU 12. In this case, the central brain 15 derives the z coordinate value of an object as point information by performing the processing executed by MoPU 12 in the above description on the image of the object captured by camera 30. As an example, the central brain 15 derives the z coordinate value of an object as point information from images of the object captured by multiple cameras 302, specifically, by camera 302L and camera 302R. In this case, the central brain 15 uses the principle of a stereo camera and derives the z coordinate value of an object as point information based on the images of the object captured by camera 302L of MoPU 12L and camera 302R of MoPU 12R respectively.

[0419] (Tenth Embodiment)

[0420] Next, while omitting or simplifying parts that overlap with the foregoing embodiments, the tenth embodiment related to this embodiment will be described.

[0421] Fig.13 is a block diagram showing an example of the structure of the information processing apparatus 10 related to the tenth embodiment. It should be noted that Fig.13 only a part of the structure of the information processing apparatus 10 is shown.

[0422] As Fig.13 shown, through MoPU 12, an image of an object captured by event camera 30C (hereinafter sometimes also referred to as an "event image") is input to core 17. Moreover, core 17 outputs point information to central brain 15 based on the input event image. It should be noted that event cameras are disclosed, for example, in the reference (https: / / dendenblog.xyz / event-based-camera / ).

[0423] FIG. 14 is an explanatory diagram for explaining an image (event image) of an object captured by event camera 30C. Fig.14A is a diagram showing an object that is the subject of photography by event camera 30C. Fig. 14B is a diagram showing an example of an event image. Fig. 14C This is a diagram showing an example in which the center of gravity of the difference portion between an image captured at the current moment (an image obtained by capturing at the current moment) and an image captured at the previous moment (an image obtained by capturing at the previous moment) represented by an event image is calculated as point information. In the event image, the difference portion between the image captured at the current moment (an image obtained by capturing at the current moment) and the image captured at the previous moment (an image obtained by capturing at the previous moment) is extracted as points. Therefore, in the case of using the event camera 30C, for example, as Fig. 14B shown, Fig.14A the points of the moving parts in the person area as shown are extracted.

[0424] In contrast, as Fig. 14C shown, after the core 17 extracts a person as an object, the coordinates of the feature points representing the person area (for example, only one point) are extracted. Thereby, the amount of data transmitted to the central brain 15 and the memory 16 can be suppressed. The event image can extract a person as an object at an arbitrary frame rate. Therefore, in the case of the event camera 30C, in the above-described embodiment, it is also possible to extract at a frame rate higher than the maximum frame rate (for example: 1920 frames / second) of the camera 30 mounted on the MoPU 12, and the point information of the object can be captured with high precision.

[0425] It should be noted that, similar to the foregoing embodiment, the information processing apparatus 10 according to the tenth embodiment may further include a visible light camera 305A in addition to the event camera 30C. In this case, through the MoPU 12, the visible light image and the event image of the object captured by the visible light camera 305A are respectively input to the core 17. Then, the core 17 outputs the point information to the central brain 15 based on at least one of the input visible light image and the event image.

[0426] For example, in a case where an object can be recognized from a visible light image of the object captured by the visible light camera 305A, the core 17 outputs point information based on the visible light image. On the other hand, in a case where the object cannot be captured from the visible light image due to a specified factor, the core 17 outputs point information based on the event image. The specified factor includes at least one of a case where the moving speed of the object is equal to or higher than a specified value and a case where the change in the amount of light per unit time of the ambient light is equal to or higher than a specified value. For example, in a case where the object moves at a high speed and the object is not captured from the visible light image, the core 17 recognizes the object based on the event image and outputs the x coordinate value and the y coordinate value of the object as point information. In addition, in a case where the object is not captured from the visible light image due to a sudden change in the amount of ambient light such as backlight, the core 17 recognizes the object based on the event image and outputs the x coordinate value and the y coordinate value of the object as point information. With this configuration, according to the information processing device 10, the camera 30 for photographing the object can be used separately according to the specified factor.

[0427] (The Eleventh Embodiment)

[0428] Next, while omitting or simplifying parts that overlap with the foregoing embodiments, the eleventh embodiment related to the present embodiment will be described.

[0429] Fig.15 is a first block diagram showing an example of the configuration of the information processing device 10 related to the eleventh embodiment, Fig.16 is a second block diagram showing an example of the configuration of the information processing device 10 related to the eleventh embodiment.

[0430] As Fig.15 shown, the images from the camera installed on the left side of the vehicle 100 and the images from the camera installed on the right side of the vehicle 100 can be input to the core 12A of the MoPU 12. Each of these images is, for example, an image including color information of 1000 pixels × 1000 pixels and can be input to the core 12A at a frame rate of 1000 frames per second. The core 12A of the MoPU 12 can transmit vector information of the motion along each of the three coordinate axes (x-axis, y-axis, z-axis) in the three-dimensional orthogonal coordinate system to the central brain 15 at a frame rate of 1000 frames per second based on these images. In addition, as Fig.16 shown, separate cores 12A 1 and 12A 2 can also be used to process the images from the camera installed on the left side of the vehicle 100 and the images from the camera installed on the right side of the vehicle 100.

[0431] (The Twelfth Embodiment)

[0432] Next, while omitting or simplifying parts that overlap with the foregoing embodiments, the twelfth embodiment related to the present embodiment will be described.

[0433] Fig.17 FIG. 1 is a first block diagram showing an example of the configuration of the information processing apparatus 10 according to the twelfth embodiment. Fig.18 FIG. 2 is a second block diagram showing an example of the configuration of the information processing apparatus 10 according to the twelfth embodiment.

[0434] As Fig.17 shown, the MoPU 12 can output motion information based on at least one of a visible light image and an infrared image. The visible light image is an image captured by a visible light camera, and the infrared image is an image captured by an infrared camera. The visible light image and the infrared image are each input to the core 12A at a frame rate of 1000 frames per second or more. Preferably, the visible light image and the infrared image are synchronized with each other. By using the infrared image in object detection performed by the MoPU 12, object detection can be performed even in a situation where it is difficult to perform object detection using the visible light image, such as at night. The MoPU 12 can output motion information based only on the infrared image among the visible light image and the infrared image, or can output motion information based on both the visible light image and the infrared image.

[0435] In addition, as Fig.18 shown, the MoPU 12 can output motion information based on an image and a radar signal. The radar signal is a signal based on a reflected wave of an electromagnetic wave irradiated onto the object and reflected from the object. The MoPU 12 can derive the distance to the object based on the image and the radar signal, and output vector information representing the motion of a point indicating the presence position of the object along each of the three axes in a three-dimensional orthogonal coordinate system as the motion information. The image can include at least one of a visible light image and an infrared image. The image and the radar signal are input to the core 12A at a frame rate of 1000 frames per second or more.

[0436] In addition, in the above description, a case is exemplified in which the central brain 15 performs driving control of the vehicle 100 based on the image output from the IPU 11 and the motion information output from the MoPU 12. However, the disclosed technology is not limited to this aspect. The central brain 15 can also perform motion control of a robot based on the image output from the IPU 11 and the motion information output from the MoPU 12 as response control for the object. The robot can be, for example, a humanoid intelligent robot that performs operations instead of a human. For example, the central brain 15 performs motion control of the robot's arm, palm, finger, and foot based on the image output from the IPU 11 and the motion information output from the MoPU 12, so as to perform motion control such as grasping, holding, hugging, carrying on the back, moving, transporting, throwing, kicking, and avoiding the object. The IPU 11 and the MoPU 12 can be mounted, for example, at the positions of the right eye and the left eye of the robot.

[0437] (The Thirteenth Embodiment)

[0438] Next, while omitting or simplifying the parts that overlap with the foregoing embodiments, the thirteenth embodiment related to the present embodiment will be described. The feature of the thirteenth embodiment is that the frame rate at the time of shooting an image is variable and the like.

[0439] (Mounting the information processing device on the vehicle 100: Smart Car)

[0440] Fig.19 is a block diagram of the information processing device 213 related to the thirteenth embodiment mounted on the vehicle 100. As Fig.19 shown, the information processing device 213 mounted on the vehicle 100 includes a MoPU 12L corresponding to the left eye, a MoPU 12R corresponding to the right eye, an IPU 11, a core 12X, and a central brain 15.

[0441] The MoPU 12L includes a camera 313L, a radar 32L, an infrared camera 34L, and a dedicated core 12AL. In addition, the MoPU 12R includes a camera 313R, a radar 32R, an infrared camera 34R, and a dedicated core 12AR. The radars 32L and 32R detect the aforementioned radar signals. The infrared cameras 34L and 34R acquire the aforementioned infrared images.

[0442] The IPU 11 includes a high-resolution camera as described above (not shown), detects an object from the high-resolution image captured by the high-resolution camera, and outputs information indicating the category of the object (hereinafter, simply referred to as "label information").

[0443] It should be noted that hereinafter, only the processing of the MoPU 12L corresponding to the left eye will be described.

[0444] The camera 313L of MoPU 12L captures images at a frame rate (120, 240, 480, 960, or 1920 frames per second) higher than that of the high-resolution camera (e.g., capturing at 10 frames per second) of IPU 11. The camera 313L is a camera capable of changing the frame rate.

[0445] The core 12AL of MoPU 12L (e.g., composed of one or more CPUs) extracts feature points for each frame image captured by the camera 313L and outputs their coordinate values (X, Y). MoPU 12L outputs, for example, the center point (center of gravity point) of the object extracted from the image as a feature point. It should be noted that the feature points can be two diagonal vertices in a rectangle approximately enclosing the object. Additionally, when capturing an object as an object with a certain size, it is not necessary to extract only at least two diagonal coordinate points among the vertices of the quadrilateral enclosing the contour of the object recognized from the image captured by the camera. Multiple coordinate points including the contour can also be extracted.

[0446] Specifically, MoPU 12L outputs the coordinate values (X, Y) of the feature points extracted from one object. It should be noted that, for example, when multiple objects (e.g., object A, object B, and object C, etc.) are reflected in one image, MoPU 12L can output the coordinate values (Xn, Yn) of the feature points extracted from each of these multiple objects. The sequence of feature points in the image captured at each moment corresponds to the motion information of the object.

[0447] Additionally, for example, consider the case where MoPU 12L cannot recognize an object due to darkness. In this case, MoPU 12L can use the infrared camera 34L to detect the heat of the object and, based on the infrared image as the detection result and the image captured by the camera 313L, output the coordinates (Xn, Yn) of the object. Additionally, the image capture using the camera 313L can be synchronized with the infrared image capture using the infrared camera 34L. In this case, for example, the number of images per second captured by the camera 313L is synchronized with the number of infrared images per second captured by the infrared camera 34L (e.g., 1920 frames per second).

[0448] Additionally, MoPU 12L can obtain the coordinate value of the Z-axis of the object based on the three-dimensional point cloud data acquired by the radar 32L. It should be noted that in this case, the image capture using the camera 313L can be synchronized with the acquisition of the three-dimensional point cloud data using the radar 32L. For example, the number of three-dimensional point cloud data per second acquired by the radar 32L is synchronized with the number of images per second captured by the camera 313L (e.g., 1920 frames per second).

[0449] In addition, the number of images per second captured by the camera 313L, the number of images per second captured by the infrared camera 34L, and the number of pieces of three-dimensional point cloud data acquired by the radar 32L can be made the same to synchronize the timing of data acquisition.

[0450] The core 12X acquires the coordinates of the feature points output from the MoPU 12L and the label information of the object (information indicating whether the object is a dog, a cat, or a bear) output from the IPU 11. Then, the core 12X associates the label information with the coordinates corresponding to the feature points and outputs the result. Thereby, the information on what the object represented by the feature points is and the motion information of the object represented by the feature points can be associated.

[0451] The above is the processing of the MoPU 12L corresponding to the left eye. The MoPU 12R corresponding to the right eye performs the same processing as the MoPU 12L corresponding to the left eye.

[0452] It should be noted that based on the images captured by the camera 313L of the MoPU 12L and the images captured by the camera 313R of the MoPU 12R, the coordinate value Zn in the depth direction of the feature points can be further calculated using the principle of a stereo camera.

[0453] (Information processing device mounted on a robot: intelligent robot)

[0454] Fig. 20 is a block diagram of the information processing device 310 according to the thirteenth embodiment mounted on a robot. As Fig. 20 shown, the information processing device 310 mounted on a robot includes a MoPU 12L corresponding to the left eye, a MoPU 12R corresponding to the right eye, an infrared camera 34, a structured light 3613, a core 12X, and a central brain 15. The information processing device 310 mounted on a robot has the same functions as the information processing device 213 mounted on the vehicle 100.

[0455] For example, when the MoPU 12L cannot recognize an object due to the influence of darkness, the MoPU 12L uses the infrared camera 34 to detect the heat of the object, and based on the infrared image as the detection result and the image captured by the camera 313L, outputs the coordinates (Xn, Yn) of the object. In addition, the image capture using the camera 313L and the infrared image capture using the infrared camera 34 can be synchronized. In this case, for example, the number of images per second captured by the camera 313L is synchronized with the number of images per second captured by the infrared camera 34 (e.g., 1920 frames / second).

[0456] In addition, the core 12X can use structured light 3613 to obtain the coordinates Zn in the depth direction of an object. The structured light 3613 is disclosed, for example, in the reference (http: / / ex-press.jp / wp-content / uploads / 2018 / 10 / 018_teledyne_3rd.pdf). In this case, the imaging using the images of the cameras 313L and 313R can be synchronized with the measurement of the three-dimensional data using the structured light 3613. For example, the number of images per second captured by the cameras 313L and 313R can be synchronized with the three-dimensional data per second measured by the structured light 3613 (e.g., 1920 frames / second).

[0457] Furthermore, both the infrared image captured by the infrared camera 34 and the three-dimensional data measured by the structured light 3613 can be used in combination.

[0458] (Change of frame rate corresponding to the external environment)

[0459] The information processing devices 213 and 310 can change the frame rate of the camera according to the external environment. For example, the information processing devices 213 and 310 calculate a score related to the external environment and determine the frame rate of the cameras 313L and 313R based on the score. Then, the information processing devices 213 and 310 output a control signal to the cameras 313L and 313R, and the control signal instructs to capture images at the determined frame rate. The cameras 313L and 313R capture images at the determined frame rate. Then, the information processing devices 213 and 310 extract the points representing the presence position of the object from the images captured by the cameras 313L and 313R and output the points representing the presence position of the object.

[0460] It should be noted that the information processing device 213 mounted on the vehicle 100 is equipped with a variety of sensors (not shown). One or more processors of the information processing device 213 mounted on the vehicle 100 calculate, based on the sensor information obtained from a variety of sensors (omitted from illustration) (e.g., the movement of the center of gravity of the weight, the detection of the road material, the detection of the external air temperature, the detection of the external air humidity, the detection of the inclination angles of the slope in the up, down, horizontal, and diagonal directions, the freezing mode of the road, the detection of the moisture content, the material, wear condition, and air pressure of each tire, the road width, the presence or absence of overtaking prohibition, the oncoming vehicle, the vehicle types of the front and rear vehicles, the cruising states of these vehicles, or the surrounding conditions (birds, animals, footballs, accident vehicles, earthquakes, fires, winds, typhoons, heavy rains, light rains, snowstorms, fog, etc.)), the degree to which the vehicle 100 will travel in a dangerous place in the future as a score related to the external environment.

[0461] Then, one or more processors provided in the information processing device 213 switch the number of images captured per second (frame rate) based on the calculated risk level and the threshold value.

[0462] For example, when the risk level is lower than the first threshold value, one or more processors provided in the information processing device 213 output a control signal to the cameras 313L, 313R, the radars 32L, 32R, and the infrared cameras 34L, 34R to select 120 frames per second, and capture images, acquire radar signals, or capture infrared images at this frame rate.

[0463] In addition, for example, when the risk level is equal to or higher than the first threshold value and lower than the fourth threshold value, one or more processors provided in the information processing device 213 output a control signal to each device to select any one of 240, 480, and 960 frames per second, and acquire various data at this frame rate.

[0464] In addition, for example, when the risk level is equal to or higher than the fourth threshold value, one or more processors provided in the information processing device 213 output a control signal to each device to select 1920 frames per second, and acquire various data at this frame rate.

[0465] In addition, one or more processors provided in the information processing device 213 can use big data related to driving that is known before the vehicle 100 travels, such as long-tail event AI (Artificial Intelligence) data (for example, the travel data of a vehicle equipped with a level 5 autonomous driving control method) or map information, to predict the risk level.

[0466] In addition, one or more processors included in the information processing device 310 mounted on the robot can calculate a score related to the external environment based on, for example, the speed of an object reflected in the cameras 313L, 313R, and change the frame rate according to this score. For example, the score related to the external environment is calculated such that the higher the speed of the object, the greater the score related to the external environment. Therefore, when the score related to the external environment is high, one or more processors included in the information processing device 310 mounted on the robot output a control signal to each device to select 1920 frames per second, and acquire various data at this frame rate. In addition, when the score related to the external environment is low, one or more processors included in the information processing device 310 mounted on the robot output a control signal to each device to select 120 frames per second, and acquire various data at this frame rate. Other controls are the same as those of the information processing device 213 mounted on the aforementioned vehicle 100.

[0467] (Output of feature points corresponding to the area where the object is detected)

[0468] When the position where an object appears in an image is in a specified area, the information processing devices 213 and 310 can output a point indicating the position where the object exists. In this case, the information processing devices 213 and 310 determine whether to output the feature points of the object according to the area where the object is detected. For example, the cores 12AL and 12AR of the information processing device 213 mounted on the vehicle 100 do not extract feature points from an object different from the object detected in the road area where the vehicle 100 travels (for example, an object existing on the sidewalk). In FIG. 21, a diagram is shown for explaining the processing in the case where, for example, feature points are not extracted from an object existing on the sidewalk.

[0469] In Fig.21A objects B1 to B4 are extracted. Usually, for each of the objects B1 to B4, the coordinates representing the feature points are extracted.

[0470] In this case, for example, as Fig.21B shown, the cores 12AL and 12AR included in the information processing device 213 mounted on the vehicle 100 use known techniques to sequentially detect the road boundary L from the image in front of the vehicle 100. Then, the cores 12AL and 12AR provided in the information processing device 213 extract the coordinates representing the feature points only from the objects B1 to B3 located on the road determined by the road boundary L.

[0471] In addition, for example, the cores 12AL and 12AR provided in the information processing device 213 may not extract the object area itself of an object B2 different from the objects B1, B3, and B4 located on the road, but may extract only the coordinates representing the feature points from the objects B1, B3, and B4.

[0472] (Output of Feature Points Corresponding to the Movement of the Object)

[0473] The information processing devices 213 and 310 can calculate the scores of each object reflected in the image and extract points representing the existence positions of the objects with scores above a specified threshold. In this case, for example, the information processing devices 213 and 310 can determine whether to output the feature points of the object according to the movement of the object. For example, the cores 12AL and 12AR of the information processing device 213 may not extract feature points from objects that do not affect the driving of the vehicle 100. Specifically, the cores 12AL and 12AR of the information processing device 213 calculate the moving direction or speed of the objects reflected in the image by using AI or the like. Moreover, for example, the cores 12AL and 12AR of the information processing device 213 do not extract feature points from pedestrians who are gradually moving away from the road. On the other hand, the cores 12AL and 12AR of the information processing device 213 extract feature points from objects approaching the road (for example, children who may jump onto the road).

[0474] In addition, one or more processors provided in the information processing device 213 can also extract feature points from, for example, images captured by an event camera (https: / / dendenblog.xyz / event-based-camera / ). In FIG. 14, a diagram for explaining an image captured by an event camera is shown.

[0475] As shown in FIG. 14, in the image captured by the event camera, the different parts between the image captured at the current moment and the image captured at the previous moment are extracted as points. Therefore, in the case of using an event camera, for example, as Fig. 14B shown, Fig.14A the points of the moving parts in the person area shown are extracted.

[0476] In contrast, as Fig. 14C shown, after extracting a person as an object, one or more processors provided in the information processing device 213 extract the coordinates (for example, only one point) representing the feature points of the person area. Thereby, the amount of information transmitted to the central brain 15 and the memory 16 can be suppressed. Since the images captured by the event camera can extract a person as an object at an arbitrary frame rate, the cameras 313L and 313R mounted on the MoPU 12L and 12R capture images at a frame rate of up to 1920 frames per second. However, in the case of an event camera, it is not limited to 1920 frames per second, and it can also be extracted at a frame rate of 1920 frames per second or more, and the movement information of the object can be captured with higher accuracy.

[0477] As described above, the information processing devices 213 and 310 of the thirteenth embodiment extract points indicating the existence position of an object from an image showing the object, and output the points indicating the existence position of the object. Thereby, the amount of information transmitted to the cores 12X, the central brain 15, and the memory 16 can be suppressed. In addition, by associating the points indicating the existence position of the object with the tag information output from the IPU 11, information on how a certain object is moving can be grasped. In particular, the cameras 313L and 313R mounted on the MoPUs 12L and 12R can capture images at a frame rate of up to 1920 frames per second, so that the motion information of the object can be captured with high precision.

[0478] In addition, the information processing devices 213 and 310 include cameras 313L and 313R capable of changing the frame rate, calculate a score related to the external environment, and determine the frame rate of the cameras based on this score. Then, the information processing devices 213 and 310 output a control signal instructing the cameras 313L and 313R to capture images at the determined frame rate, extract points indicating the existence position of the object from the images captured by the cameras 313L and 313R, and output the points indicating the existence position of the object. Thereby, images can be captured at a frame rate suitable for the external environment.

[0479] In addition, the information processing device 213 calculates the risk related to the travel of the vehicle 100 as the score related to the external environment, determines the frame rate of the cameras 313L and 313R based on the risk, outputs a control signal instructing the cameras 313L and 313R to capture images at the determined frame rate, and extracts points indicating the existence position of the object from the images captured by the cameras 313L and 313R. Thereby, the frame rate can be changed according to the risk related to the travel of the vehicle 100.

[0480] In addition, the information processing devices 213 and 310 extract an object from the image, and when the existence position of the object is in a specified area, extract points indicating the existence position of the object, and output the points indicating the existence position of the object. Thereby, the cores 12AL and 12AR of the information processing devices 213 and 310 do not need to acquire points in areas with relatively low importance for control processing.

[0481] In addition, the information processing devices 213 and 310 extract an object from the image, calculate a score for each object, extract points indicating the existence position of the object with a score equal to or higher than a specified threshold, and output the points indicating the existence position of the object. Thereby, the cores 12AL and 12AR of the information processing devices 213 and 310 do not need to acquire points of objects with relatively low importance for control processing.

[0482] (Fourteenth Embodiment)

[0483] Next, while omitting or simplifying parts that overlap with the foregoing embodiments, the fourteenth embodiment of the present embodiment will be described. The fourteenth embodiment is characterized in that the MoPU 12 includes a plurality of cameras with different perspectives, among other things.

[0484] (Information processing device mounted on a vehicle: intelligent vehicle)

[0485] Fig. 22 is a block diagram of the information processing device 410 according to the fourteenth embodiment, mounted on a vehicle 50 (refer to Fig.23 ). Fig.23 is a diagram showing the shooting directions and shooting angle ranges of the high-resolution camera 30L and the omnidirectional camera 30R mounted on the vehicle 50. It should be noted that, for ease of explanation, the Fig.23 direction along the arrow H is set as the up-down direction of the vehicle 50, the direction along the arrow W is set as the left-right direction of the vehicle 50, and the direction along the arrow D is set as the front-rear direction of the vehicle 50.

[0486] As Fig. 22 shown, the information processing device 410 mounted on the vehicle 50 (refer to Fig.23 ) includes an IPU 11, a MoPU 12, and a central brain 15.

[0487] The IPU 11 is equipped with the high-resolution camera 30A and the core 11A (e.g., composed of one or more CPUs) as described above. From the high-resolution images captured by the camera 30A, it detects objects and outputs information indicating the categories of the objects.

[0488] The MoPU 12 is equipped with a high-resolution camera 30L, an omnidirectional camera 30R, an adjustment unit 40, and a core 12A (e.g., composed of one or more CPUs). From the images captured by the high-resolution camera 30L and the omnidirectional camera 30R, it detects objects and outputs motion information indicating the motion of the objects.

[0489] The high-resolution camera 30L is a camera with a specified horizontal viewing angle and has a higher resolution per unit viewing angle than the omnidirectional camera 30R. In the present embodiment, as an example, the horizontal viewing angle of the high-resolution camera 30L is set to 45°, but it is not limited thereto and can be set to any viewing angle. In addition, the horizontal viewing angle of the high-resolution camera 30L can be changed, for example, through a zoom function, etc. The high-resolution camera 30L is an example of the "first camera" in the disclosed technology.

[0490] In addition, the omnidirectional camera 30R is a camera with a horizontal viewing angle of 360°. The omnidirectional camera 30R is an example of the "second camera" in the publicly known technology. It should be noted that the "second camera" is not limited to the omnidirectional camera 30R, and any camera with a horizontal viewing angle greater than that of the "first camera" can be used, such as a camera with a horizontal viewing angle of 270°.

[0491] As Fig.23 shown, the high-resolution camera 30L and the omnidirectional camera 30R are provided on the roof of the vehicle 50. The high-resolution camera 30L is provided so as to be rotatable about a rotation axis parallel to the height direction H of the vehicle 50, and is configured to be able to adjust the shooting direction in the horizontal direction by an adjustment unit 40 described later. The shooting direction of the high-resolution camera 30L generally faces the front of the vehicle 50.

[0492] In addition, the high-resolution camera 30L and the omnidirectional camera 30R are provided at different positions in the left-right direction of the vehicle 50. The high-resolution camera 30L is relatively provided on the left side, and the omnidirectional camera 30R is relatively provided on the right side. As described in the foregoing embodiment, the MoPU 12 of the present embodiment can calculate the coordinate value Z in the depth direction of the feature point based on the image obtained by the high-resolution camera 30L and the image obtained by the omnidirectional camera 30R using the principle of a stereo camera.

[0493] It should be noted that in the case of calculating the coordinate value Z in the depth direction of the feature point using the principle of a stereo camera based on the images obtained by two cameras with different viewing angles, for example, the image obtained by the omnidirectional camera 30R with a wider viewing angle may be set as follows: a corrected image trimmed so as to be consistent with the shooting direction and shooting range of the image obtained by the high-resolution camera 30L with a narrower viewing angle.

[0494] The adjustment unit 40 is composed of, for example, a motor, and is configured to be able to rotate the high-resolution camera 30L under the control of the core 12A to adjust the shooting direction in the horizontal direction. The adjustment unit 40 uses the front of the vehicle 50 as the reference direction (0°) of the shooting direction of the high-resolution camera 30L, and is configured to be able to adjust the shooting direction of the high-resolution camera 30L in the positive direction (right rotation direction) and the negative direction (left rotation direction).

[0495] When the movement of an object located in the blind spot of the high-resolution camera 30L is detected in the image obtained by the omnidirectional camera 30R, the core 12A controls the adjustment unit 40 so that the shooting direction of the high-resolution camera 30L faces the direction of the detected object.

[0496] Here, the control of the shooting direction of the high-resolution camera 30L in the core 12A will be described in detail. Fig.24 It is a diagram showing the relationship between the shooting ranges of the image obtained by the omnidirectional camera 30R and the image obtained by the high-resolution camera 30L.

[0497] As Fig.24 shown, the image GR obtained by the omnidirectional camera 30R is an image with a horizontal viewing angle of 360°. That is, the image GR is an image within the range of ±180° with the front of the vehicle 50 as the reference direction (0°) of the shooting direction of the high-resolution camera 30L.

[0498] The image GL obtained by the high-resolution camera 30L is an image with a horizontal viewing angle of 45°. The shooting direction of the high-resolution camera 30L is usually set to the front of the vehicle 50. That is, the normal image GL is an image within the range of ±22.5° with the front of the vehicle 50 as the reference direction (0°) of the shooting direction of the high-resolution camera 30L.

[0499] As an example, as Fig.23 shown, the case where the moving object B is in the -100° direction with the front of the vehicle 50 as the reference direction (0°) will be described. In this case, as Fig.24 shown, in the image GR obtained by the omnidirectional camera 30R, with the front of the vehicle 50 as the reference direction (0°), the moving object B is reflected at the position corresponding to -100°.

[0500] When the moving object B is detected in the image GR, the core 12A determines whether the moving object B is included in the shooting range of the high-resolution camera 30L. Fig.23 And Fig.24 The moving object B shown is an object outside the shooting range of the high-resolution camera 30L and in the blind spot of the high-resolution camera 30L.

[0501] In this case, as Fig.25 And Fig.26 shown, the core 12A controls the adjustment unit 40 so that the shooting direction of the high-resolution camera 30L in the horizontal direction faces the direction (-100° direction) of the center position of the moving object B.

[0502] It should be noted that regarding the adjustment of the shooting direction of the high-resolution camera 30L, as long as the adjustment is made so that the moving object B is included in the image GL obtained after the shooting direction adjustment, it is not limited to the aspect of adjusting the shooting direction to be consistent with the center position of the moving object B. For example, if the adjustment is made so that the end on the rotation direction side of the moving object B is reflected in the image GL, compared with the case where the adjustment is made so that the moving object B is reflected in the center of the image GL, the rotation amount of the high-resolution camera 30L can be reduced.

[0503] In addition, the moving direction of the moving object B in the image GR obtained by the omnidirectional camera 30R can be predicted and reflected in the adjustment control of the shooting direction of the high-resolution camera 30L.

[0504] For example, as in the aforementioned example, when the shooting direction of the high-resolution camera 30L is adjusted from the reference direction (0°) to the direction of the center position of the moving object B (the direction of -100°), when the moving object B is moving toward the reference direction side, the adjustment target of the shooting direction of the high-resolution camera 30L can be corrected to an angle with a smaller absolute value than -100°. Thereby, the rotation amount of the high-resolution camera 30L can be reduced.

[0505] In addition, when the shooting direction of the high-resolution camera 30L is adjusted from the reference direction (0°) to the direction of the center position of the moving object B (the direction of -100°), when the moving object B is moving toward the side opposite to the reference direction, the adjustment target of the shooting direction of the high-resolution camera 30L can be corrected to an angle with a larger absolute value than -100°. Thereby, it is possible to prevent the moving object B from being excluded from the shooting range of the high-resolution camera 30L after the shooting direction adjustment of the high-resolution camera 30L.

[0506] In the foregoing case, the correction amount from the adjustment target can be changed according to the moving speed of the moving object B in the image GR obtained by the omnidirectional camera 30R. For example, it can be that the faster the moving speed of the moving object B, the larger the correction amount from the adjustment target is set, and the slower the moving speed of the moving object B, the smaller the correction amount from the adjustment target is set. Thereby, the adjustment of the shooting direction of the high-resolution camera 30L can be performed more appropriately.

[0507] As described above, in the information processing apparatus 410 of the fourteenth embodiment, when the movement of an object located in the blind spot of the first camera (in this example, the high-resolution camera 30L) is detected in the image obtained by the second camera (in this example, the omnidirectional camera 30R), the MoPU 12 controls the adjustment unit 40 so that the shooting direction of the first camera faces the direction of the detected object.

[0508] Accordingly, when detecting a moving object while obtaining wide-range information using the image acquired by the second camera, it is possible to detect the moving object by combining the images acquired by the first camera, and thus the detection accuracy of the moving object can be improved.

[0509] In addition, the horizontal viewing angle of the second camera is set to 360°. Accordingly, it is possible to obtain information on the entire periphery of the moving body on which the information processing device 410 is mounted.

[0510] In addition, the resolution per unit viewing angle of the first camera is set to be higher than that of the second camera. Accordingly, the detection accuracy of the moving object can be further improved.

[0511] It should be noted that, in the information processing device 410 of the fourteenth embodiment, the adjustment range of the shooting direction of the adjustment unit 40 in the horizontal direction is set to be within ±135° with respect to the reference direction, preferably within the range of ±90°. In the case where the moving object is located at a position deviating from this adjustment range, the adjustment of the shooting direction may not be performed.

[0512] In this way, by limiting the adjustment range of the shooting direction to the vicinity of the front, the first camera can focus on the detection in the vicinity of the front, which is of relatively high importance in the movement control of the moving body on which the information processing device 410 is mounted. Therefore, the followability of the moving object in the vicinity of the front can be improved.

[0513] In addition, the MoPU 12 can use the image acquired by the first camera and the image acquired by the second camera to obtain information on the distance to the object in the image. As described above, even for images acquired by two cameras with different viewing angles, it is possible to obtain information on the distance to the object in the image using the principle of a stereo camera. By setting it in this way, the amount of information acquired in the MoPU 12 can be increased.

[0514] (Fifteenth Embodiment)

[0515] Next, while omitting or simplifying the parts that overlap with the foregoing embodiments, the fifteenth embodiment related to this embodiment will be described. The fifteenth embodiment is characterized in that the IPU 11 associates the category information indicating the category of the object extracted from the image acquired by the external camera with the position information of the external camera and outputs the same, etc.

[0516] (Information Processing Device Mounted on a Vehicle: Intelligent Vehicle)

[0517] Fig. 27 is mounted on a vehicle 50 (refer to Fig.29)Block diagram of the information processing apparatus 410 according to the fifteenth embodiment.

[0518] As Fig. 27 shown, the information processing apparatus 410 mounted on the vehicle 50 (see Fig.29 ) includes an IPU 11, a MoPU 12, a central brain 15, and a navigation system 45.

[0519] The IPU 11 includes a camera 30A, a core 11A (e.g., composed of one or more CPUs), and an acquisition unit 415, which detects an object (hereinafter referred to as "object") from the images captured by the camera 30A and the images captured by an external camera 60 provided outside the vehicle 50, and outputs information indicating the category of the object.

[0520] The acquisition unit 415 acquires the images acquired by the external camera 60 provided outside the vehicle 50 and the position information of the external camera 60, and is composed of, for example, one or more CPUs and a wireless communication unit for wirelessly connecting to an external device.

[0521] Note that, outside the vehicle 50, a plurality of external cameras 60A, 60B, 60C,... are provided. These external cameras 60A, 60B, 60C are connected to the management server 70 via a network. Here, the network can be any network such as the Internet, a LAN (Local Area Network), a WAN (Wide Area Network), etc.

[0522] The MoPU 12 includes a camera 30B and a core 12A (e.g., composed of one or more CPUs), and detects an object from the images captured by the camera 30B and outputs motion information indicating the motion of the object.

[0523] The navigation system 45 is a system for guiding the movement of the vehicle 50, and can use, for example, a system equivalent to a known car navigation system.

[0524] (Detailed description of the processing of the IPU 11)

[0525] Next, the processing of the IPU 11 in the present embodiment will be described in detail.

[0526] First, the management server 70 that manages the plurality of external cameras 60A, 60B, 60C provided outside the vehicle 50 will be described. The management server 70 includes a camera management database.

[0527] Fig.28 is a diagram showing an example of the camera management database managed in the management server 70. As Fig.28As shown, the camera management database associates and stores the camera ID, address, and installation location. The camera ID is an inherent ID assigned to each of the external cameras 60. The address is information indicating the address when connecting to each of the external cameras 60 via a network, such as an IP (Internet Protocol) address or the like. It should be noted that there is no limitation on the version of the IP address. For example, it can be set to various versions such as IPv4 (Internet Protocol version 4) or IPv6 (Internet Protocol version 6). The installation location is information indicating the location where each of the external cameras 60 is installed, such as GPS (Global Positioning System) coordinates (e.g., latitude, longitude, etc.).

[0528] The external camera 60 is, for example, a camera installed in a building along a road, a traffic signal, or a median strip.

[0529] The management server 70 can connect to any external camera 60 by referring to the camera management database. In addition, the camera management database can also be referred to by a device external to the management server 70. The acquisition unit 415 of the IPU 11 is configured to be able to connect to the management server 70 via wireless connection.

[0530] When the vehicle 50 moves, the acquisition unit 415 connects to the management server 70, and based on the camera management database stored in the management server 70, the position information of the vehicle 50 obtained from the navigation system 45, and the information on the traveling direction, determines the external camera 60 closest to the vehicle 50 among the external cameras 60 located in front of the traveling route of the vehicle 50.

[0531] In the present embodiment, as an example, the external camera 60 located in front of the traveling route of the vehicle 50 is set as the external camera 60 within a specified width range in front of the vehicle body of the vehicle 50 with the central axis in the longitudinal direction of the vehicle body as a reference.

[0532] It should be noted that the external camera 60 located in front of the traveling route of the vehicle 50 is not limited to the above aspect. As long as it is located in front of the traveling route of the vehicle 50, the external camera 60 can be any external camera 60. For example, in a situation where the vehicle 50 is guided by the navigation system 45, it can be set as the external camera 60 within a specified range from the route set by the navigation system 45.

[0533] Then, the acquisition unit 415 acquires the current image acquired by the determined external camera 60 and the position information of the external camera 60.

[0534] In the present embodiment, while the vehicle 50 is passing near the determined external camera 60, the acquisition unit 415 acquires an image from the determined external camera 60 every second. Here, the time interval for acquiring the image is not limited to every second, and can be set to any aspect such as every 10 seconds, every 5 seconds, or every 0.1 second.

[0535] It should be noted that the acquisition of the image in the acquisition unit 415 is not limited to the above aspects. For example, while the vehicle 50 is passing near the determined external camera 60, an image can be acquired every certain distance of the traveling distance of the vehicle 50, or it can be set to any aspect such as acquiring an image only once from the determined external camera 60.

[0536] For example, as Fig.29 shown, when there are a plurality of external cameras 60A to 60N around the vehicle 50, the acquisition unit 415 determines the external camera 60G closest to the vehicle 50 among the external cameras 60 located in front of the traveling route of the vehicle 50, and acquires the current image acquired by the determined external camera 60G and the position information of the external camera 60G.

[0537] It should be noted that the acquisition of the current image acquired by the determined external camera 60G can be acquired via the management server 70, or can be directly connected to the external camera 60G and acquired.

[0538] The core 11A of the IPU 11 associates and outputs the category information indicating the category of the object extracted from the image acquired by the external camera 60 with the position information of the external camera 60.

[0539] In Fig.29 the example shown, the acquisition unit 415 acquires an image from the external camera 60G. Fig.30 is a diagram showing an example of the image acquired by the external camera 60G. As Fig.30 shown, in the image GEx acquired by the external camera 60G, two cars and a person are reflected.

[0540] The core 11A of the IPU 11 extracts two cars and a person as objects B1 to B3 from the image GEx. Next, the core 11A determines the category of each of the objects B1 to B3. Then, for each of the objects B1 to B3, the core 11A associates the category information indicating the category of the object with the position information of the external camera 60G and outputs it, and registers it in the object management database DB. The object management database DB is stored, for example, in the memory 16 (refer to Figure 2 ).

[0541] At this time, when the extracted object is an object that may affect the travel of the vehicle 50, the core 11A attaches identification information indicating that it is an object that may affect the travel (hereinafter, referred to as "attention required flag"), and outputs the category information of the object.

[0542] It should be noted that examples of objects that may affect the travel of the vehicle 50 include people or animals. In the present embodiment, as an example, when the extracted object is a person or an animal, the core 11A attaches an attention required flag and outputs the category information of the object.

[0543] Fig.31 FIG. is a diagram showing an example of the object management database DB managed in the information processing apparatus 410. As Fig.31 shown, the object management database DB associates and stores an object ID, a registration date and time, an object position, an object category, and the presence or absence of an attention required flag.

[0544] The object ID is an inherent ID assigned to each of the objects detected by the core 11A. The registration date and time is information on the date and time when the object is registered in the object management database DB. The object position is information indicating the position of the external camera 60 that captured the object, and is the position information of the external camera 60 acquired by the acquisition unit 415. The object category is information indicating the category of the object, such as information on people, animals, cars, motorcycles, bicycles, signals, etc. The presence or absence of the attention required flag is information indicating whether the aforementioned attention required flag is assigned to the object.

[0545] As described in the above embodiment, the central brain 15 performs driving control of the vehicle 50 based on the image and the recognition information and the motion information, where the image and the recognition information are information output from the IPU 11 based on the image acquired by the camera 30A provided in the IPU 11, and the motion information is information output from the MoPU 12 based on the image acquired by the camera 30B provided in the MoPU 12.

[0546] In addition, the central brain 15 of the present embodiment can refer to the object management database DB to grasp in advance an object located in front of the vehicle 50 and likely to affect the travel of the vehicle 50. Therefore, the driving control of the vehicle 50 can be performed more appropriately.

[0547] For example, as Fig.29 shown, at the stage when the vehicle 50 is in front of the external camera 60G, it is possible to grasp in advance an object located around the external camera 60G and likely to affect the travel of the vehicle 50. In this example, as Fig.30 shown, it is possible to grasp that a person (in the figure, object B3) exists around the external camera 60G.

[0548] Thus, considering the possibility that a person jumps out around the external camera 60G, the central brain 15 can perform pre-control, such as decelerating before the vehicle 50 reaches the vicinity of the external camera 60G, or making preparations to be able to brake immediately when a person jumps out, etc.

[0549] As a result, as Fig.32 shown, when the vehicle 50 reaches the vicinity of the external camera 60G, even if a person jumps out, due to the pre-control, it is possible to respond calmly.

[0550] As described above, in the information processing device 410 of the fifteenth embodiment, the IPU 11 associates the category information with the position information of the external camera 60 and outputs it, and the category information represents the category of an object extracted from the image obtained by the external camera 60. Thus, the central brain 15 can execute the driving control of the vehicle 50 even considering the image obtained by the external camera 60 provided outside the vehicle 50 on which the information processing device 410 is mounted.

[0551] In addition, when an object extracted from the image obtained by the external camera 60 is an object such as a person or an animal that may affect the travel of the vehicle 50, the IPU 11 attaches identification information indicating that it is an object that may affect the travel and outputs the category information of the object. Thus, the central brain 15 can execute the driving control of the vehicle 50 considering the object that may affect the travel.

[0552] In addition, the acquisition unit 415 acquires an image and position information from an external camera 60 located in front of the travel route of the vehicle 50. Thus, the central brain 15 can perform pre-control for an object that may affect the travel before reaching the installation position of the external camera 60.

[0553] In addition, the acquisition unit 415 acquires images and position information only from the external camera closest to the present device in front of the traveling route of the vehicle 50. Thereby, the communication data volume in the acquisition unit 415 can be suppressed.

[0554] It should be noted that the method for determining the external camera 60 from which the acquisition unit 415 acquires images and position information is not limited to the foregoing aspect and can be set to any aspect. For example, the acquisition unit 415 can determine the external camera 60 located in front of the traveling route of the vehicle 50 based on the route information of the navigation system 45 that guides the traveling route of the vehicle 50, and acquire images and position information from the determined external camera 60.

[0555] For example, as Fig.33 shown, it is assumed that the traveling route R of the vehicle 50 is set by the navigation system 45. In this case, the acquisition unit 415 can determine the external cameras 60A, 60B, 60C, 60E, 60F, 60G arranged along the traveling route R as the external cameras 60 located in front of the traveling route of the vehicle 50, and acquire images and position information from the determined external cameras 60.

[0556] In this case, for the acquisition of images and position information, the acquisition unit 415 does not necessarily need to acquire images and position information from all the external cameras 60 arranged along the traveling route R. For example, it can acquire images and position information only from the external cameras 60 located within a specified distance range from the vehicle 50.

[0557] In addition, in the case where the camera management database manages information on the shooting direction and / or shooting range of the camera in addition to managing the camera ID, address, and installation position, the acquisition unit 415 can determine, among the external cameras 60 arranged along the traveling route R, the external cameras 60 that can shoot the vehicle 50 when the vehicle 50 travels along the traveling route R, and acquire images and position information from the determined external cameras 60. By setting it in this way, even for the external cameras 60 arranged along the traveling route R, images and position information are not acquired from the external cameras 60 that cannot shoot the vehicle 50, and acquisition of data that cannot be used for driving control is not performed, so it helps to reduce the communication data volume and the processing volume in the acquisition unit 415.

[0558] (Sixteenth Embodiment)

[0559] Next, while omitting or simplifying the parts that overlap with the foregoing embodiments, the sixteenth embodiment related to the present embodiment will be described.

[0560] Fig.34 is a block diagram showing an example of the structure of the information processing apparatus 10 related to the sixteenth embodiment.

[0561] As shown Fig.34 in FIG. 1, the information processing apparatus 10 according to the sixteenth embodiment is provided with only one MoPU 12. The MoPU 12 includes a photographing unit 3616, and the photographing unit 3616 includes a single camera 30 and a moving unit 3816 that moves the camera 30. As shown Fig.35 in FIG. 2, the moving unit 3816 includes a disk-shaped rotating member 4016, and the rotating member 4016 is arranged such that its axis is substantially along the horizontal direction. In addition, the moving unit 3816 includes an actuator 42 constituted by a motor or the like, and the rotating member 4016 rotates around the axis of the rotating member 4016 by the actuator 42.

[0562] The camera 30 is mounted on the peripheral portion of the rotating member 4016 such that the photographing optical axis is parallel to the axis of the rotating member 4016. Thus, when the rotating member 4016 is rotated by the actuator 42, the camera 30 sequentially moves to the first position P1 and the second position P2 whose position in the horizontal direction is different from that of the first position P1. More specifically, the camera 30 moves cyclically along an annular path that passes through each of the first position P1 and the second position P2.

[0563] Here, the MoPU 12 determines the frame rate FR of the camera 30 according to a score related to the external environment, and the photographing unit 3616 of the MoPU 12 rotates the rotating member 4016 at a rotation period (= 1 / FR) corresponding to the determined frame rate FR by the actuator 42. In addition, the moving unit 3816 of the photographing unit 3616 captures images at the timing when the camera 30 is located at the first position P1 and at the timing when it is located at the second position P2, respectively. Thus, the photographing period of the camera 30 is the reciprocal of twice the frame rate (= 1 / (2×FR)).

[0564] In addition, the MoPU 12 includes a core 17. In the present embodiment, since there is a difference in the photographing time between the first image captured at the first position P1 and the second image captured at the second position P2, a deviation occurs in the two-dimensional position of the object in the first image and the second image as compared with the case where the first image and the second image are captured at the same time. Therefore, the core 17 corrects the deviation in the two-dimensional position of the object in the first image and the second image caused by the difference in the photographing time between the first image and the second image.

[0565] Specifically, for example, as shown Fig.36AAs shown, consider the following situation: Based on the first image captured at the first position P1 at the first moment t1, calculate the two-dimensional position (XlA(t1), YlA(t1)) of object A. Based on the second image captured at the second position P2 at the third moment t3, calculate the two-dimensional position (XrA(t3), YrA(t3)) of object A. Based on the first image captured at the first position P1 at the second moment t2, calculate the two-dimensional position (XlA(t2), YlA(t2)) of object A (t1 < t2 < t3).

[0566] In this case, the core 17 performs integration between the first image captured at the first position P1 at the first moment t1 and the first image captured at the first position P1 at the second moment t2. Additionally, based on the result of the integration, the core 17 estimates the two-dimensional position of object A in the virtual first image obtained when shooting is performed at the first position P1 at the third moment t3, which is the shooting moment at the second position P2 (also refer to the two-dimensional position (XlA(t3), YlA(t3)) shown as "Estimation of the two-dimensional position of object A at moment t3") Fig.36B in.

[0567] Thus, the estimated two-dimensional position (XlA(t3), YlA(t3)) of object A and the two-dimensional position (XrA(t3), YrA(t3)) of object A are output from the MoPU 12 including the core 17 to the central brain 15 at a period corresponding to the frame rate FR. Then, the central brain 15 calculates the three-dimensional position (XA(t3), YA(t3), ZA(t3)) of object A at moment t3 based on the estimated two-dimensional position (XlA(t3), YlA(t3)) of object A and the two-dimensional position (XrA(t3), YrA(t3)) of object A.

[0568] Additionally, for example, as Fig.36A shown, consider the following situation: Based on the first image captured at the first position P1 at the first moment t1, calculate the two-dimensional position (XlB(t1), YlB(t1)) of object B. Based on the second image captured at the second position P2 at the third moment t3, calculate the two-dimensional position (XrB(t3), YrB(t3)) of object B. Based on the first image captured at the first position P1 at the second moment t2, calculate the two-dimensional position (XlB(t2), YlB(t2)) of object A (t1 < t2 < t3).

[0569] In this case, the core 17 performs integration between a first image captured at a first position P1 at a first time t1 and a first image captured at the first position P1 at a second time t2. Further, based on the result of the integration, the core 17 estimates the imaging time at the second position P2, that is, the two-dimensional position of the object B in a virtual first image obtained when imaging is performed at the first position P1 at a third time t3 (also refer to the two-dimensional position (XlB(t3), YlB(t3)) shown as “estimation of two-dimensional position of object B at time t3” in Fig.36B ).

[0570] Thereby, the estimated two-dimensional position (XlB(t3), YlB(t3)) of the object B and the two-dimensional position (XrB(t3), YrB(t3)) of the object B are output from the MoPU 12 including the core 17 to the central brain 15 at a period corresponding to the frame rate FR. Then, the central brain 15 calculates the three-dimensional position (XB(t3), YB(t3), ZB(t3)) of the object B at time t3 based on the estimated two-dimensional position (XlB(t3), YlB(t3)) and the two-dimensional position (XrB(t3), YrB(t3)) of the object B. In this way, the MoPU 12 including the core 17 and the central brain 15 that calculates the three-dimensional position of the object are an example of the processing unit according to the present disclosure.

[0571] As described above, in the information processing apparatus 10 according to the sixteenth embodiment, the imaging unit 3616 causes the camera 30 to sequentially move to a first position and a second position P2 whose position in at least the horizontal direction is different from the first position P1, and causes the camera 30 to capture images at the first position P1 and the second position P2, respectively. Then, the processing unit (MoPU 12 and central brain 15) calculates the three-dimensional positions of the objects respectively reflected in the first image and the second image based on the first image captured at the first position P1 and the second image captured at the second position P2. Thereby, when calculating the three-dimensional positions of the objects, it is not necessary to provide a plurality of cameras 30, and the structure of the information processing apparatus 10 can be simplified.

[0572] Further, in the sixteenth embodiment, the imaging unit 3616 includes a moving unit 3816 that rotates a rotating member 4016 on which the camera 30 is mounted so that the camera 30 moves in a circular path passing through each of the first position P1 and the second position P2. Thereby, it is easier to implement compared to, for example, a method of linearly reciprocating the camera 30 between the first position P1 and the second position P2.

[0573] In addition, in the sixteenth embodiment, the processing unit (MoPU 12) calculates a score related to the external environment, and determines the frame rate FR of the camera 30 based on the calculated score. The moving unit 3816 changes the rotation speed of the rotating member 4016 according to the frame rate FR determined by the processing unit (MoPU 12). Thereby, the frame rate FR of the camera 30 can be continuously changed.

[0574] In addition, in the sixteenth embodiment, the processing units (MoPU 12 and Central Brain 15) correct the deviation of the two-dimensional positions of the objects in the first image and the second image caused by the shooting time difference between the first image and the second image, so as to calculate the three-dimensional positions of the objects. Thereby, the calculation accuracy of the three-dimensional positions of the objects can be improved.

[0575] In addition, in the sixteenth embodiment, the core 17 of the processing unit (MoPU 12) performs integration between the first image captured at the first position P1 at the first time and the first image captured at the first position P1 at the second time, and estimates the shooting time at the second position, that is, the two-dimensional position of the object in the virtual first image obtained when shooting is performed at the first position P1 at the third time based on the integration result, thereby correcting the deviation of the two-dimensional position of the object. Thereby, it is possible to improve the calculation accuracy of the three-dimensional position of the object by using relatively simple processing.

[0576] It should be noted that in the sixteenth embodiment, a method is described in which the rotating member 4016 equipped with the camera 30 is rotated so that the camera 30 moves cyclically along an annular path passing through each of the first position P1 and the second position P2, thereby realizing the sequential movement of the camera 30 to the first position P1 and the second position P2. However, the present disclosure is not limited thereto, and for example, the camera 30 may be linearly reciprocated between the first position P1 and the second position P2, thereby realizing the sequential movement of the camera 30 to the first position P1 and the second position P2.

[0577] In addition, in the sixteenth embodiment, it is described that the frame rate FR of the camera 30 is determined according to a score related to the external environment, and the rotation speed of the rotating member 4016 is changed according to the determined frame rate FR. However, the present disclosure is not limited thereto. For example, the rotation speed of the rotating member 4016 may be set to be constant, and the frame rate FR of the camera 30 may be set to the reciprocal of an integer multiple of the time obtained by dividing the rotation period of the rotating member 4016 by 2.

[0578] (Seventeenth embodiment)

[0579] Next, while omitting or simplifying parts that overlap with the previous embodiments, the seventeenth embodiment related to the present embodiment will be described.

[0580] As an example, the information processing apparatus 10 related to the seventeenth embodiment has a structure similar to that of the first embodiment Figure 2 as shown.

[0581] The MoPU 12 related to the seventeenth embodiment outputs point information obtained by capturing the photographed object as points based on the images of the objects photographed by other cameras, and calculates and outputs motion information indicating the motion of the points based on the time series of the point information. Specifically, the MoPU 12 uses the Hough transform based on the time series of the point information to calculate the direction of the motion of the points, and calculates the speed of the motion based on the change in the point information in the calculated direction of the motion of the points.

[0582] For example, based on the time series of the point information as Fig.38A shown, as Fig.38B shown, the Hough transform is used to calculate the direction of the motion of the points.

[0583] The IPU 11 outputs recognition information obtained by recognizing the object photographed based on the image of the object photographed by the ultra-high-resolution camera in the photographing direction corresponding to the other cameras.

[0584] The central brain 15 associates the motion information output from the MoPU 12 and the recognition information output from the IPU 11. Functionally, as Fig.37 shown, the central brain 15 includes an acquisition unit 170, a motion detection unit 172, a removal unit 174, and an association unit 176.

[0585] The acquisition unit 170 acquires the motion information output from the MoPU 12 and the recognition information output from the IPU 11.

[0586] The motion detection unit 172 detects the motion information of multiple objects corresponding to the direction of motion based on the motion information output from the MoPU 12. Specifically, when the formed angle of the direction of motion between the motion information is below the threshold and the number of the motion information of multiple objects corresponding to the direction of motion is above the threshold, it is determined that the motion information of multiple objects corresponding to the direction of motion is detected.

[0587] For example, as Fig.39 shown, based on the motion information M1 to M8 of multiple objects, the motion information M1 to M6 of multiple objects corresponding to the direction of motion is detected.

[0588] When the motion information of multiple objects corresponding to the direction of motion is detected, the removal unit 174 removes the motion information of the multiple objects.

[0589] For example, as Fig.40 shown, by removing the motion information M1 to M6 of the plurality of objects corresponding to the direction of motion from the motion information M1 to M8 of the plurality of objects, the motion information M7 and M8 are obtained.

[0590] The association unit 176 associates the removed motion information with the recognition information output from the IPU 11, and controls the autonomous driving of the moving body based on the associated motion information and recognition information.

[0591] Next, the association process performed by the central brain 15 will be described using Fig.41 . At this time, it is assumed that motion information is input from the MoPU 12 to the central brain 15, and recognition information is input from the IPU 11 to the central brain 15.

[0592] First, in step S100, the acquisition unit 170 acquires the motion information output from the MoPU 12 and the recognition information output from the IPU 11.

[0593] In step S102, the motion detection unit 172 detects the motion information of the plurality of objects corresponding to the direction of motion based on the motion information output from the MoPU 12.

[0594] In step S104, when the motion information of the plurality of objects corresponding to the direction of motion is detected, the removal unit 174 removes the motion information of the plurality of objects.

[0595] In step S106, the association unit 176 associates the removed motion information with the recognition information output from the IPU 11.

[0596] As described above, according to the information processing apparatus 10 according to the seventeenth embodiment, when the motion information of the plurality of objects corresponding to the direction of motion is detected, the central brain 15 removes the motion information of the plurality of objects and associates the remaining motion information with the recognition information. Thus, when a plurality of small objects corresponding to the direction of motion are detected based on the motion information output from the MoPU 12, the small objects can be removed and associated with the recognition information. For example, noise during rainfall and snowfall can be removed from the motion information output by the MoPU 12.

[0597] (Eighteenth Embodiment)

[0598] Next, while omitting or simplifying parts that are repeated with the foregoing embodiments, the eighteenth embodiment according to the present embodiment will be described.

[0599] As an example, the information processing apparatus 10 according to the eighteenth embodiment has a structure similar to that of the first embodiment. Figure 2 as shown.

[0600] Similar to the first embodiment, the MoPU 12 according to the eighteenth embodiment outputs point information obtained by capturing the object as a point based on an image of the object captured by another camera.

[0601] Then, as Fig.42 shown, when the MoPU 12 detects a certain number or more of points whose moving direction is either the up or down direction and is the same direction, and the moving amount in the up and down direction is within a certain range, based on the point information output from multiple frames of images, the MoPU 12 outputs only the points other than the detected certain number or more of points among the points included in the point information. It should be noted that the up and down direction mentioned here is the up and down direction of the image, which corresponds to the height direction of the vehicle 100. In addition, the left and right direction mentioned here is the left and right direction of the image, which corresponds to the vehicle width direction of the vehicle 100. The certain range in this case can be preset as a fixed value or set according to the state of the road surface. In addition, the certain number in this case can be preset as a fixed value or set according to the number of points included in the point information. For example, it can be set according to 10% of the number of points included in the point information, etc.

[0602] In Fig.42 , an example is shown in which in each of the images of three consecutive frames, three points Q1, Q2, and Q3 are included in the point information. In addition, in Fig.42 , an example is shown in which the points Q1 and Q2 move in the same direction in the up and down direction, and the moving amount in the up and down direction is within a certain range. In addition, in Fig.42 , an example is shown in which the point Q3 moves in the upper left direction and then in the lower left direction.

[0603] In Fig.42 's example, the MoPU 12 outputs only the point Q3 from among the three points Q1, Q2, and Q3 included in the point information based on the point information output from the images of three consecutive frames. This is because the points Q1 and Q2 are estimated to be actually stationary, but due to the influence of the road surface state and the like on the vehicle 100, the points Q1 and Q2 appear to be moving within the image.

[0604] Similar to the first embodiment, the point information representing the points output by the MoPU 12 passes through the central brain 15 and is thus associated with the label information output from the IPU 11.

[0605] As described above, according to the present embodiment, information indicating a point that is estimated to be actually stationary but appears to move in the image among the point information to be output is set as non-output object. Therefore, according to the present embodiment, when outputting the point information, which is the shooting information of an object shot by a camera, to a predetermined output destination, the amount of data output to the output destination can be reduced.

[0606] It should be noted that, in the present embodiment, when the vehicle 100 is going straight, the MoPU 12 may also output points with a lateral movement amount equal to or greater than a threshold value from among a certain number or more of detected points. The MoPU 12 can use a well-known technique to determine whether the vehicle 100 is going straight, and the well-known technique is a technique for determining using the steering angle of the vehicle 100, or a technique for determining using the lateral wheel speed ratio of each of the front wheels and the rear wheels.

[0607] In addition, in the present embodiment, the central brain 15 may execute the process of outputting only points other than a certain number or more of detected points from among the points included in the point information, which is executed by the MoPU 12.

[0608] (Nineteenth Embodiment)

[0609] Next, while omitting or simplifying parts that overlap with the foregoing embodiments, the nineteenth embodiment related to the present embodiment will be described.

[0610] Fig.43 It is a block diagram showing an example of the structure of the information processing apparatus 10 related to the nineteenth embodiment. It should be noted that Fig.43 only a part of the structure of the information processing apparatus 10 is shown. The information processing apparatus 10 is an example of the "information processing apparatus" and "computer" of the present disclosure.

[0611] The information processing apparatus 10 is mounted on a plurality of vehicles 200. In Fig.43 the example shown, the first vehicle 200A and the second vehicle 200B are shown as examples of the plurality of vehicles 200. The first vehicle 200A is an example of the "first moving body" of the present disclosure, and the second vehicle 200B is an example of the "second moving body" of the present disclosure. Hereinafter, for the sake of convenience of explanation, when it is not necessary to distinguish between the first vehicle 200A and the second vehicle 200B, it is referred to as "vehicle 200".

[0612] The information processing apparatus 10 includes a central brain 15, a communication I / F 202, a first memory 204, and a second memory 206. Here, I / F is an abbreviation for "Interface".

[0613] The central brain 15 of the information processing device 10 mounted on the first vehicle 200A is used for the first vehicle 200A. In addition, the central brain 15 of the information processing device 10 mounted on the second vehicle 200B is used for the second vehicle 200B. It should be noted that the central brain 15 of the information processing device 10 mounted on the first vehicle 200A is an example of the "first mobile body processor" of the present disclosure, and the central brain 15 of the information processing device 10 mounted on the second vehicle 200B is an example of the "second mobile body processor" of the present disclosure.

[0614] Hereinafter, for the sake of convenience of explanation, the information processing device 10 mounted on the first vehicle 200A will be referred to as the "information processing device 10 of the first vehicle 200A", and the information processing device 10 mounted on the second vehicle 200B will be referred to as the "information processing device 10 of the second vehicle 200B". In addition, hereinafter, for the sake of convenience of explanation, the central brain 15 of the information processing device 10 of the first vehicle 200A will be referred to as the "central brain 15 of the first vehicle 200A", and the central brain 15 of the information processing device 10 of the second vehicle 200B will be referred to as the "central brain 15 of the second vehicle 200B".

[0615] The communication I / F 202, the first memory 204, and the second memory 206 are connected to the central brain 15. The communication I / F 202 is a communication interface including a communication processor and an antenna, etc., and is included in the Figure 1 gateway shown. The communication I / F 202 is responsible for communication between different vehicles 200 using the central brain 15. For example, the central brain 15 of the first vehicle 200A and the central brain 15 of the second vehicle 200B send and receive various signals via the communication I / F 202 of the first vehicle 200A and the communication I / F 202 of the second vehicle 200B. As an example of the communication standard applied to the communication I / F 202, wireless communication standards such as Wi-Fi (registered trademark) or 5G (5th Generation Mobile Communication System) can be cited.

[0616] The first memory 204 is, for example, a non-volatile memory (e.g., flash memory, etc.). The first memory 204 stores various programs including the program 208 and various parameters, etc. The program 208 is an example of the "information processing program" of the present disclosure.

[0617] The second memory 206 is, for example, a volatile memory (e.g., RAM (Random Access Memory), etc.). The second memory 206 is a memory for temporarily storing information and is used as a working memory by a processor such as the central brain 15.

[0618] In the first vehicle 200A and the second vehicle 200B, as described in the foregoing first embodiment and the like, autonomous driving is performed under the control of the information processing device 10. In the autonomous driving of the first vehicle 200A, the driving of the first vehicle 200A is controlled based on the information of the second vehicle 200B as other surrounding vehicles (for example, information on the current position and speed). On the other hand, in the second vehicle 200B, autonomous driving is also performed in the same manner. In this case, after the second vehicle 200B recognizes the first vehicle 200A, it predicts the movement of the first vehicle 200A and performs autonomous driving.

[0619] In this way, when the first vehicle 200A and the second vehicle 200B predict each other's movements and perform autonomous driving after recognizing each other, for example, it can be considered that both the first vehicle 200A and the second vehicle 200B change their forward routes to avoid a head-on collision between the first vehicle 200A and the second vehicle 200B. However, originally, only the first vehicle 200A or the second vehicle 200B needs to change its forward route. However, since both the first vehicle 200A and the second vehicle 200B change their forward routes, there is a risk that the driving of surrounding vehicles other than the first vehicle 200A and the second vehicle 200B (for example, the vehicles behind the first vehicle 200A and the second vehicle 200B, and the parallel vehicles of the first vehicle 200A and the second vehicle 200B, etc.) will be disturbed.

[0620] In addition, in order to avoid a head-on collision, originally, only the second vehicle 200B needs to change its forward route. However, due to the control of the autonomous driving of the first vehicle 200A and the second vehicle 200B, the first vehicle 200A also changes its forward route. In this case, for example, there is a risk of discomfort to the occupants due to the sudden change of the forward route. It should be noted that not only in the case of avoiding a head-on collision, but also in the case of avoiding contact during lane change and / or contact during queuing driving, the same problem can be said to exist.

[0621] In addition, in the existing known technologies (for example, the technology described in Patent Document 1), it is difficult for the driver of the first vehicle 200A or the like to grasp whether the second vehicle 200B has recognized the first vehicle 200A. Therefore, it is difficult to change the driving method of the first vehicle 200A or to change the awareness of the driver of the first vehicle 200A regarding the second vehicle 200B according to whether the second vehicle 200B has recognized the first vehicle 200A.

[0622] Therefore, in view of such a situation, in this nineteenth embodiment, the central brain 15 performs autonomous driving control processing.

[0623] The central brain 15 performs an autonomous driving control process by reading a program 208 from the first memory 204 and executing the read program 208 on the second memory 206. In accordance with the program 208 executed on the second memory 206, the central brain 15 operates as a first determination unit 15A, a second determination unit 15B, a third determination unit 15C, a setting unit 15D, an inference unit 15E, an adjustment unit 15F, and a driving control unit 15G, thereby implementing the autonomous driving control process.

[0624] Priority information 209 is assigned to each of the plurality of vehicles 200 such that the priority information 209 for each vehicle is different from that of the other vehicles. The priority information 209 is information indicating the priority of the vehicle 200 among the plurality of vehicles 200 that is to be preferentially processed by the setting unit 15D. The priority indicated by the priority information 209 assigned to the first vehicle 200A is an example of the "first priority" of the present disclosure, and the priority indicated by the priority information 209 assigned to the second vehicle 200B is an example of the "second priority" of the present disclosure. The priority information 209 is stored in the first memory 204 of the information processing device 10 of each vehicle 200.

[0625] Fig.44 It is a schematic diagram showing an example of the processing contents of the first determination unit 15A, the second determination unit 15B, the third determination unit 15C, and the setting unit 15D of the first vehicle 200A.

[0626] Tag information 210 is input from the IPU 11 to the central brain 15 of the first vehicle 200A. The tag information 210 is the same information as the tag information described in the foregoing first embodiment.

[0627] It should be noted that the tag information 210 input to the central brain 15 of the first vehicle 200A is an example of the "second moving body information capable of identifying the second moving body" of the present disclosure. In addition, the tag information 210 input to the central brain 15 of the second vehicle 200B is an example of the "first moving body information capable of identifying the first moving body" of the present disclosure.

[0628] The first determination unit 15A acquires the tag information 210 and refers to the tag information 210 to identify the type of an object (an example of the "first object" of the present disclosure) reflected in a first image, which is an image obtained by the ultra-high-resolution camera of the first vehicle 200A photographing the surroundings of the first vehicle 200A (hereinafter, also simply referred to as the "first image"). Then, the first determination unit 15A determines whether the type identified as the type of the object reflected in the first image is the second vehicle 200B.

[0629] Note that in the information processing device 10 of the second vehicle 200B, in the same manner, the type of an object (an example of the "second object" in the present disclosure) appearing in the second image is identified. The second image is an image obtained by the ultra-high resolution camera of the second vehicle 200B photographing the surroundings of the second vehicle 200B (hereinafter, also simply referred to as the "second image"). Then, it is determined whether the identified type is the first vehicle 200A.

[0630] When the first determination unit 15A determines that the second vehicle 200B appears in the first image, the central brain 15 of the first vehicle 200A sends a request signal 212 to the information processing device 10 of the second vehicle 200B via the communication I / F 202. The request signal 212 is a signal for requesting the information processing device 10 of the second vehicle 200B to send a response signal 214.

[0631] The communication I / F 202 of the second vehicle 200B receives the request signal 212 sent by the central brain 15 of the first vehicle 200A. The central brain 15 of the second vehicle 200B sends a response signal 214 corresponding to the request signal 212 to the information processing device 10 of the first vehicle 200A via the communication I / F 202 of the second vehicle 200B.

[0632] When the response signal 214 is sent from the central brain 15 of the second vehicle 200B, the communication I / F 202 of the first vehicle 200A receives the response signal 214. The central brain 15 of the first vehicle 200A acquires the response signal 214 received by the communication I / F 202.

[0633] The response signal 214 includes driving information 214A, identification information 214B, and control information 214C. In addition, the response signal 214 also includes priority information 209 assigned to the second vehicle 200B.

[0634] The driving information 214A is information indicating whether the second vehicle 200B is in autonomous driving. The identification information 214B is information indicating whether the first vehicle 200A is recognized by the information processing device 10 of the second vehicle 200B. The control information 214C is information for the central brain 15 of the second vehicle 200B to control the autonomous driving of the second vehicle 200B.

[0635] The first determination unit 15A refers to the tag information 210 and the driving information 214A to determine whether the type recognized as the type of the object appearing in the first image is the second vehicle 200B in autonomous driving.

[0636] When the type recognized as the type of the object reflected in the first image by the first determination unit 15A is the second vehicle 200B that is driving autonomously, the second determination unit 15B refers to the recognition information 214B to determine whether the information processing device 10 of the second vehicle 200B has recognized the first vehicle 200A.

[0637] When the second determination unit 15B determines that the information processing device 10 of the second vehicle 200B has recognized the first vehicle 200A, the third determination unit 15C determines whether the priority of the first vehicle 200A is higher than the priority of the second vehicle 200B. The determination by the third determination unit 15C is made using the priority information 209 included in the response signal 214 and the priority information 209 stored in the first memory 204. The priority information 209 included in the response signal 214 is the information given to the second vehicle 200B, and the priority information 209 stored in the first memory 204 is the information given to the first vehicle 200A. The third determination unit 15C determines whether the order indicated by the priority information 209 given to the first vehicle 200A is higher than the order indicated by the priority information 209 given to the second vehicle 200B.

[0638] Here, when it is determined that the order indicated by the priority information 209 given to the first vehicle 200A is higher than the order indicated by the priority information 209 given to the second vehicle 200B, the setting unit 15D sets the identifier 216, which indicates that the first vehicle 200A has been recognized by the second vehicle 200B. The setting of the identifier 216 is achieved by the setting unit 15D storing the identifier 216 in the second memory 206. As an example of the identifier 216, a flag can be cited.

[0639] Fig.45 It is a schematic diagram showing an example of the content of the processing performed by the inference unit 15E and the driving control unit 15G of the first vehicle 200A when the identifier 216 is not set by the setting unit 15D.

[0640] In the case where the identifier 216 is not set by the setting unit 15D, the central brain 15 controls the autonomous driving of the first vehicle 200A in a similar manner as described in the foregoing first embodiment and the like, based on the sensor information 218, the association information 219, and the unset information 222. The sensor information 218 is the same information as the sensor information described in the foregoing first embodiment and the like. The association information 219 is information that associates the first point information with the label information 210 in a similar manner as described in the foregoing first embodiment and the like. The first point information is the same information as the point information described in the foregoing first embodiment and the like. That is, the first point information is point information obtained by capturing, as points, objects (an example of the "first object" of the present disclosure) included in the surroundings of the first vehicle 200A based on a third image, which is an image obtained by photographing the surroundings of the first vehicle 200A at a frame rate higher than the frame rate of the photographing for obtaining the first image (hereinafter, also simply referred to as "third image"). The unset information 222 is information indicating that the identifier 216 has not been set.

[0641] It should be noted that, although an example of the manner in which the central brain 15 of the first vehicle 200A obtains the association information 219 is given here, in the central brain 15 of the second vehicle 200B, second vehicle-side association information (not shown) can also be obtained as information similar to the association information 219. The second vehicle-side association information is information that associates second point information obtained in a similar manner as the point information described in the foregoing first embodiment and the like with information obtained in a similar manner as the label information 210. The second point information is point information obtained by capturing, as points, objects (an example of the "second object" of the present disclosure) included in the surroundings of the second vehicle 200B based on a fourth image, which is an image obtained by photographing the surroundings of the second vehicle 200B at a frame rate higher than the frame rate of the photographing for obtaining the second image (hereinafter, also simply referred to as "fourth image"). The second vehicle-side association information is used by the central brain 15 of the second vehicle 200B in a process similar to that of the central brain 15 of the first vehicle 200A.

[0642] To control the autonomous driving of the first vehicle 200A, the central brain 15 calculates a first control variable 224 as a control variable for controlling the autonomous driving of the first vehicle 200A, based on the sensor information 218 and the association information 219. The first control variable 224 is a variable having the same definition as the control variable described in the foregoing first embodiment and the like.

[0643] The first control variable 224 is calculated by the inference unit 15E. The inference unit 15E has a deep learning model 226 and uses the deep learning model 226 to calculate the first control variable 224.

[0644] The deep learning model 226 is a learning model obtained by performing deep learning on a neural network using supervised data. As an example of the supervised data used herein, a data set obtained by associating example data with correct solution data assuming the first control variable 224 can be cited. As an example of the example data, data obtained in advance through experiments and / or computer simulations, etc. by an actual device as data assuming sensor information 218, data obtained in advance through experiments and / or computer simulations, etc. by an actual device as data assuming associated information 219, and data indicating whether the identifier 216 is set can be cited.

[0645] The inference unit 15E inputs the sensor information 218, the associated information 219, and the unset information 222 to the deep learning model 226. The deep learning model 226 outputs a first control variable 224 (for example, the control variable with the highest confidence) corresponding to the input sensor information 218, associated information 219, and unset information 222.

[0646] It should be noted that although a calculation method for calculating the first control variable 224 by using the deep learning model 226 is exemplified herein, this is merely an example, and the first control variable 224 can be calculated by using various calculation methods (for example, multivariate analysis based ...

Claims

1. An information processing device, wherein, the information processing device includes: a first processor that outputs point information obtained by capturing the object captured by the first camera as points, based on an image of the object; and a second processor that outputs identification information obtained by identifying the object captured by the second camera facing the direction corresponding to the first camera, based on an image of the object.

2. The information processing device according to claim 1, wherein, the information processing device includes a third processor, and the third processor associates the point information output from the first processor with the identification information output from the second processor.

3. The information processing device according to claim 1, wherein, the frame rate of the first camera is variable, and the first processor changes the frame rate of the first camera according to a specified factor.

4. The information processing device according to claim 3, wherein, the first processor calculates a score related to the external environment for a specified object.

5. The information processing device according to claim 4, wherein, the first processor changes the frame rate of the first camera according to the calculated score related to the external environment.

6. The information processing device according to claim 1, wherein, the first processor outputs coordinate values of at least two coordinate axes in a three-dimensional orthogonal coordinate system of points representing the existence position of the object captured by the first camera, based on an image of the object captured by the first camera, and the information processing device includes a third processor that associates the point information output from the first processor with the identification information output from the second processor.

7. The information processing device according to claim 6, wherein, the first processor outputs coordinate values of at least two diagonal points among the vertices of a polygon that encloses the contour of the object identified from the image captured by the first camera.

8. The information processing device according to claim 7, wherein, the first processor outputs coordinate values of a plurality of vertices of a polygon that encloses the contour of the object identified from the image captured by the first camera.

9. The information processing device according to claim 1, wherein, the information processing device includes a third processor, and the third processor associates the point information output from the first processor with the identification information output from the second processor, and controls the autonomous driving of a moving body based on the point information and the identification information.

10. The information processing device according to claim 9, wherein, the third processor calculates a control variable for controlling the autonomous driving of the moving body based on detection information detected by a detection unit, and controls the autonomous driving of the moving body based on the calculated control variable, the point information, and the identification information.

11. The information processing device according to claim 1, wherein, The first processor outputs the point information based on at least one of the visible light image and the infrared image of the object captured by the first camera. The information processing device includes a third processor that associates the point information output from the first processor with the identification information output from the second processor.

12. The information processing device according to claim 11, wherein, when the object cannot be captured from the visible light image of the object captured by the visible light camera included in the first camera due to a specified factor, the first processor outputs the point information based on the infrared image of the object captured by the infrared camera included in the first camera.

13. The information processing device according to claim 12, wherein, the first processor synchronizes the timing of capturing the visible light image by the visible light camera with the timing of capturing the infrared image by the infrared camera.

14. The information processing device according to claim 1, wherein, the first processor outputs the point information based on the image of the object captured by the first camera and a radar signal based on an electromagnetic wave irradiated on the object and reflected from the object. The information processing device includes a third processor that associates the point information output from the first processor with the identification information output from the second processor.

15. The information processing device according to claim 14, wherein, the first processor synchronizes the timing of capturing the image by the first camera with the timing of the radar acquiring three-dimensional point cloud data of the object based on the radar signal.

16. The information processing device according to claim 14, wherein, the number of images per unit time captured by the first camera and the number of three-dimensional point cloud data per unit time acquired by the radar are greater than the number of images per unit time captured by the second camera.

17. The information processing device according to claim 1, wherein, the second processor outputs label information indicating the category of the captured object based on the image of the object captured by the second camera. The information processing device includes a third processor that associates the point information output from the first processor with the label information output from the second processor.

18. The information processing device according to claim 17, wherein, the third processor associates the position information of the object represented by the point information with the label information related to the object existing at the position represented by the position information.

19. The information processing device according to claim 18, wherein, the third processor associates the point information output from the first processor at the same timing as the timing when the second processor outputs the label information with the label information.

20. The information processing device according to claim 18, wherein, When, after associating the point information with the tag information, new point information is output from the first processor, the third processor also associates the new point information with the tag information.

21. The information processing apparatus according to claim 1, wherein, the information processing apparatus includes a third processor, the third processor associates the point information output from the first processor with the identification information output from the second processor, the first processor derives, as the point information, a coordinate value in the depth direction of the object in a three-dimensional orthogonal coordinate system of a point indicating the existence position of the object from an image of the object captured by the first camera.

22. The information processing apparatus according to claim 21, wherein, the first processor derives, as the point information, the coordinate value in the depth direction from images of the object captured by a plurality of the first cameras.

23. The information processing apparatus according to claim 21, wherein, the first processor derives, as the point information, coordinate values in the width direction, height direction, and depth direction of the object from an image of the object captured by the first camera and a radar signal based on an electromagnetic wave irradiated onto the object by a radar and reflected from the object.

24. The information processing apparatus according to claim 21, wherein, the first processor derives, as the point information, coordinate values in the width direction, height direction, and depth direction of the object from an image of the object captured by the first camera and a result of photographing structured light irradiated onto the object by an irradiating device.

25. The information processing apparatus according to claim 21, wherein, the first processor derives, as the point information, a coordinate value in the depth direction at a second time point from coordinate values in the width direction, height direction, and depth direction of the object in the three-dimensional orthogonal coordinate system at a first time point and coordinate values in the width direction and height direction at the second time point, the second time point being the next time point after the first time point.

26. The information processing apparatus according to claim 1, wherein, the information processing apparatus includes a third processor, the third processor associates the point information output from the first processor with the identification information output from the second processor, the third processor derives, as the point information, a coordinate value in the depth direction of the object in a three-dimensional orthogonal coordinate system of a point indicating the existence position of the object from an image of the object captured by the first camera.

27. The information processing apparatus according to claim 1, wherein, the first processor outputs the point information based on an image of the object captured by an event camera, the second processor outputs the identification information based on an image of the object captured by a second camera facing a direction corresponding to the event camera. The information processing device includes a third processor that associates the point information output from the first processor with the recognition information output from the second processor.

28. The information processing device according to claim 27, wherein, when the object cannot be captured from the visible light image of the object captured by the visible light camera due to a specified factor, the first processor outputs the point information based on the image of the object captured by the event camera.

29. The information processing device according to claim 28, wherein, the specified factor includes at least one of a case where the moving speed of the object is equal to or higher than a specified value and a case where the change in the amount of light per unit time of the ambient light is equal to or higher than a specified value.

30. The information processing device according to claim 27, wherein, the event camera is a camera that outputs an event image, and the event image represents a difference portion between an image captured at the current moment and an image captured at the previous moment.

31. The information processing device according to claim 1, wherein, the second processor outputs the image of the captured object at a first frame rate, the first processor outputs motion information representing the motion of the captured object at a second frame rate higher than the first frame rate, the information processing device includes a third processor that performs driving control of the vehicle based on the image and the motion information.

32. The information processing device according to claim 31, wherein, the second frame rate is 10 times or more the first frame rate.

33. The information processing device according to claim 31, wherein, the second frame rate is 100 frames per second or more.

34. The information processing device according to claim 31, wherein, the first processor outputs vector information representing the motion of a point indicating the existence position of the object along a specified coordinate axis.

35. The information processing device according to claim 34, wherein, using two of the first processors, vector information representing the motion of a point indicating the existence position of the object along each of three coordinate axes in a three-dimensional orthogonal coordinate system is output.

36. The information processing device according to claim 35, wherein, the third processor has the ability to process a plurality of information in units of one billionth of a second.

37. The information processing device according to claim 1, wherein, from an image of an object captured by a camera with a frame rate of 100 frames per second or more, only points indicating the existence position of the object are extracted, and vector information representing the motion of the points indicating the existence position of the object along a specified coordinate axis is output from the processor.

38. The information processing device according to claim 35, wherein, the first processor outputs the vector information for at least two diagonal points among the vertices of a quadrilateral surrounding the contour of the object.

39. The information processing device according to claim 1, wherein, The first processor extracts points representing the existence position of the object from the image of the object, and outputs motion information at a frame rate of 1000 frames per second or more, where the motion information represents the motion of the points representing the existence position of the object along a specified coordinate axis.

40. The information processing apparatus according to claim 39, wherein, the first processor outputs vector information of the motion of the center point or centroid point of the object along a specified coordinate axis as the motion information.

41. The information processing apparatus according to claim 39, wherein, the first processor outputs, as the motion information, vector information of the motion along a specified coordinate axis for at least two diagonal points among the vertices of the quadrilateral surrounding the contour of the object.

42. The information processing apparatus according to claim 39, wherein, the image includes an infrared image.

43. The information processing apparatus according to claim 39, wherein, the image includes a visible light image and an infrared image that are synchronized with each other.

44. The information processing apparatus according to claim 39, wherein, using two of the first processors, vector information of the motion of the points representing the existence position of the object along each of the three coordinate axes in a three-dimensional orthogonal coordinate system is output as the motion information.

45. The information processing apparatus according to claim 39, wherein, the first processor derives the distance to the object based on the reflected wave of the electromagnetic wave irradiated on the object and reflected from the object, and outputs vector information of the motion of the points representing the existence position of the object along each of the three coordinate axes in a three-dimensional orthogonal coordinate system as the motion information.

46. The information processing apparatus according to claim 39, wherein, the information processing apparatus further includes: a second processor that outputs an image of the object at a frame rate less than 1000 frames per second; and a third processor that performs response control of the object based on the motion information and the image output from the second processor.

47. The information processing apparatus according to claim 1, wherein, the first processor extracts points representing the existence position of the object from the image showing the object, and outputs the points representing the existence position of the object.

48. The information processing apparatus according to claim 47, wherein, the information processing apparatus includes a camera capable of changing the frame rate, the first processor calculates a score related to the external environment, determines the frame rate of the camera according to the score, outputs a control signal indicating to capture an image at the determined frame rate to the camera, extracts points representing the existence position of the object from the image captured by the camera, and outputs the points representing the existence position of the object.

49. The information processing apparatus according to claim 48, wherein, the information processing apparatus is mounted on a vehicle, the first processor calculates a risk level related to the driving of the vehicle as the score related to the external environment, Determine the frame rate of the camera according to the degree of danger. Output a control signal to the camera indicating to capture an image at the determined frame rate. Extract points representing the existence position of the object from the image captured by the camera, and output the points representing the existence position of the object.

50. The information processing apparatus according to claim 47. Wherein, The first processor Extracts an object from the image. When the existence position of the object is in a specified area, extracts points representing the existence position of the object, and outputs the points representing the existence position of the object.

51. The information processing apparatus according to claim 47. Wherein, The first processor Extracts an object from the image. Calculates a score for each object. Extracts points representing the existence position of the object whose score is above a specified threshold, and outputs the points representing the existence position of the object.

52. The information processing apparatus according to claim 1. Wherein, The information processing apparatus includes: A first camera with a first horizontal viewing angle; A second camera with a second horizontal viewing angle wider than the first horizontal viewing angle; and An adjustment unit that adjusts the shooting direction of the first camera. When the movement of an object located in the blind spot of the first camera is detected in the image acquired by the second camera, the first processor controls the adjustment unit to direct the shooting direction of the first camera towards the detected object.

53. The information processing apparatus according to claim 52. Wherein, The horizontal viewing angle of the second camera is 360°.

54. The information processing apparatus according to claim 53. Wherein, The adjustment range of the shooting direction of the adjustment unit in the horizontal direction is within a range of ±135° with respect to the reference direction.

55. The information processing apparatus according to claim 52. Wherein, The resolution per unit viewing angle of the first camera is higher than that of the second camera.

56. The information processing apparatus according to claim 52. Wherein, The first processor uses the image acquired by the first camera and the image acquired by the second camera to obtain information on the distance of the object in the image.

57. The information processing apparatus according to claim 1. Wherein, The information processing apparatus includes: The first processor, which outputs motion information representing the motion of the object extracted from the image; The second processor, which outputs category information representing the category of the object extracted from the image; The third processor, which performs response control on the object based on the motion information and the category information; And An acquisition unit that acquires the image acquired by an external camera provided outside the mobile body on which this apparatus is mounted and the position information of the external camera. The second processor associates and outputs the category information representing the category of the object extracted from the image acquired by the external camera with the position information of the external camera.

58. The information processing apparatus according to claim 57. Wherein, In a case where an object extracted from an image acquired by an external camera is an object that may affect the travel of the moving body, the second processor attaches identification information indicating that it is an object that may affect travel, and outputs the category information of the object.

59. The information processing apparatus according to claim 58, wherein an object that may affect the travel of the moving body is a person or an animal.

60. The information processing apparatus according to claim 57, wherein the acquisition unit acquires an image and position information from an external camera located in front of the travel route of the moving body.

61. The information processing apparatus according to claim 60, wherein the acquisition unit acquires an image and position information only from the external camera closest to the apparatus in front of the travel route of the moving body.

62. The information processing apparatus according to claim 60, wherein the acquisition unit determines an external camera located in front of the travel route of the moving body based on route information of a navigation system that guides the travel route of the moving body.

63. The information processing apparatus according to claim 1 ， wherein it includes: a photographing unit that sequentially moves a camera to a first position and a second position, the second position being at least different from the first position in the horizontal direction, and causes the camera to photograph images at the first position and the second position respectively; and a processing unit that calculates three-dimensional positions of objects respectively shown in the first image photographed at the first position and the second image photographed at the second position.

64. The information processing apparatus according to claim 63, wherein the photographing unit includes a moving unit that rotates a member on which the camera is mounted so that the camera circularly moves along an annular path passing through each of the first position and the second position.

65. The information processing apparatus according to claim 64, wherein the processing unit calculates a score related to the external environment, and determines the frame rate of the camera according to the calculated score, and the moving unit changes the rotation speed of the member according to the frame rate determined by the processing unit.

66. The information processing apparatus according to claim 63, wherein the processing unit corrects a deviation in the two-dimensional position of the object in the first image and the second image caused by a photographing time difference between the first image and the second image, and calculates the three-dimensional position of the object.

67. The information processing apparatus according to claim 66, wherein the processing unit performs matching between a first image photographed at the first position at a first time and a first image photographed at the first position at a second time, and estimates the two-dimensional position of the object in a virtual first image based on the matching result, thereby correcting the deviation in the two-dimensional position of the object, the virtual first image being an image obtained when photographing is performed at the first position at a third time, which is the photographing time at the second position.

68. The information processing apparatus according to claim 1, wherein, the information processing apparatus includes a third processor, the third processor associates the point information output from the first processor with the identification information output from the second processor, when the image captured by the first camera is unclear, the third processor uses the image captured by the second camera as a substitute for the image captured by the first camera.

69. The information processing apparatus according to claim 68, wherein, when the image captured by the first camera is unclear, the third processor performs processing to clarify the image captured by the first camera.

70. The information processing apparatus according to claim 1, wherein, the second processor uses the image of the object captured by another device existing near the present device and the position information of the object to obtain the position information of the image captured by the second camera.

71. The information processing apparatus according to claim 70, wherein, the second processor considers the transmission time of the image of the object from the other device and the position information of the object, and synchronizes the frames of the second camera with the frames of the image of the object from the other device.

72. The information processing apparatus according to claim 1, wherein, the first processor switches and outputs a first coordinate value or a second coordinate value at a specified timing according to the image of the object captured by the first camera. Among at least two coordinate axes of a three-dimensional orthogonal coordinate system constituting a point representing the existence position of the captured object, the first coordinate value represents the existence position of the center point or the centroid point of the object on the coordinate axis, and the second coordinate value represents the existence position of at least two diagonal points among the vertices of the polygon surrounding the object.

73. The information processing apparatus according to claim 72, wherein, when the second processor cannot identify the captured object from the image of the object captured by the second camera due to a specified factor, the first processor switches the output coordinate value from the first coordinate value to the second coordinate value.

74. The information processing apparatus according to claim 72, wherein, when the moving speed of the object is below a specified threshold value, or when the moving direction of the object is a specified direction, the first processor switches the output coordinate value from the second coordinate value to the first coordinate value.

75. The information processing apparatus according to claim 72, wherein, the information processing apparatus includes a third processor, the third processor associates the first coordinate value or the second coordinate value output from the first processor with the identification information output from the second processor.

76. The information processing apparatus according to claim 1, wherein, When it is possible to determine that the possibility of contact between the object present in the passage path of the moving body and the moving body is low, the first processor stops outputting the point information related to the object.

77. The information processing apparatus according to claim 76, wherein when the object is not on the moving route of the moving body in the passage path and the object is gradually moving away from the moving route, the first processor determines that the possibility of contact between the object and the moving body is low.

78. The information processing apparatus according to claim 1, wherein the first processor outputs the point information based on the image of the object captured by the first camera, and calculates and outputs motion information representing the motion of the points according to the time series of the point information, the information processing apparatus includes a third processor that associates the motion information output from the first processor with the recognition information output from the second processor, when motion information of a plurality of objects corresponding to the detected motion direction is detected, the third processor removes the motion information of the plurality of objects and associates the remaining motion information with the recognition information.

79. The information processing apparatus according to claim 78, wherein the first processor calculates the direction of motion of the points using the Hough transform according to the time series of the point information, and calculates the speed of the motion according to the change in the point information of the calculated direction of motion of the points.

80. The information processing apparatus according to claim 78, wherein the third processor associates the motion information output from the first processor with the recognition information output from the second processor, and controls the automatic driving of the moving body based on the motion information and the recognition information.

81. The information processing apparatus according to claim 1, wherein the information processing apparatus includes at least one processor, the processor outputs the point information based on the image of the object captured by the camera, when a certain number or more of points moving in either the up or down direction and in the same direction and having a moving amount in a certain range in the up and down directions are detected based on the point information output from multiple frames of images, only points other than the detected certain number or more of points are output from the points included in the point information.

82. The information processing apparatus according to claim 81, wherein when the moving body provided with the camera is going straight, the processor also outputs points having a moving amount in the left - right direction equal to or greater than a threshold value from among the detected certain number or more of points.

83. The information processing apparatus according to claim 1, wherein the information processing apparatus includes a first moving body processor for use with a first moving body, the first moving body processor identifies the types of first objects included in the surroundings of the first moving body based on a first image obtained by photographing the surroundings of the first moving body The second mobile body processor used for the second mobile body identifies the types of second objects included in the surroundings of the second mobile body based on a second image obtained by photographing the surroundings of the second mobile body. When the second mobile body is identified by the first mobile body processor as the type of the first object and the first mobile body is identified by the second mobile body processor as the type of the second object, the first mobile body processor sets an identifier indicating that the first mobile body is recognized by the second mobile body processor.

84. The information processing apparatus according to claim 83, wherein, the second mobile body is a mobile body capable of autonomous driving, the first mobile body processor When the second mobile body is identified by the first mobile body processor as the type of the first object, the first mobile body is identified by the second mobile body processor as the type of the second object, and the second mobile body is in autonomous driving, the first mobile body processor sets the identifier.

85. The information processing apparatus according to claim 83, wherein, the first mobile body and the second mobile body are mobile bodies capable of autonomous driving, the first mobile body processor When the identifier is set and the second mobile body is in autonomous driving, based on the behavior of the second mobile body, controls the autonomous driving of the first mobile body.

86. The information processing apparatus according to claim 85, wherein, the second mobile body processor controls the autonomous driving of the second mobile body, the first mobile body processor Obtains control information for the second mobile body processor to control the autonomous driving of the second mobile body as information indicating the behavior, and based on the control information, controls the autonomous driving of the first mobile body.

87. The information processing apparatus according to claim 86, wherein, A first priority is assigned to the first mobile body, A second priority is assigned to the second mobile body, The first mobile body processor obtains the control information on the condition that the first priority is higher than the second priority.

88. The information processing apparatus according to claim 85, wherein, the first mobile body processor Based on a third image, outputs first point information obtained by capturing the first object as a point. The third image is an image obtained by photographing the surroundings of the first mobile body at a frame rate higher than the frame rate for obtaining the first image. Associates second mobile body information capable of determining the second mobile body identified by the first mobile body processor as the type of the first object with the first point information, and based on the associated second mobile body information and the first point information, controls the autonomous driving of the first mobile body.

89. The information processing apparatus according to claim 88, wherein, the first mobile body processor includes a first processor, a second processor, and a third processor, the first processor outputs the first point information, The second processor identifies the type of the first object based on the first image. The third processor associates the second moving body information with the first point information and controls the autonomous driving of the first moving body.

90. The information processing apparatus according to claim 83, wherein, the second moving body is a moving body capable of autonomous driving, the second moving body processor outputs second point information obtained by capturing the second object as a point based on a fourth image, where the fourth image is an image obtained by capturing the surroundings of the second moving body at a frame rate higher than the frame rate of the shooting for obtaining the second image, associates first moving body information with the second point information, where the first moving body information can determine the first moving body whose type is identified as the second object by the second moving body processor, performs control processing for controlling the autonomous driving of the second moving body based on the associated first moving body information and the second point information, when the second moving body is identified as the type of the first object by the first moving body processor, the first moving body is identified as the type of the second object by the second moving body processor, and the control processing is performed by the second moving body processor, the first moving body processor sets the identifier.

91. The information processing apparatus according to claim 85, wherein, the first moving body processor controls the autonomous driving of the first moving body by using control content capable of avoiding contact between the first moving body and the second moving body.

92. The information processing apparatus according to claim 83, wherein, when the identifier is set, the first moving body processor notifies the notification device that the first moving body has been identified by the second moving body processor.

93. The information processing apparatus according to claim 1, wherein, is an information processing apparatus including at least one processor, the processor outputs an image of an object captured at a first frame rate, for each of the images output at the first frame rate, derives label information indicating the category of the object included in the image, outputs position information indicating the existence position of the object at a second frame rate higher than the first frame rate based on the image of the object captured, associates the position information at each time point corresponding to the output timing of the image output at the first frame rate among the position information sequentially output at the second frame rate with the label information derived for the same object as the object corresponding to the position information.

94. The information processing apparatus according to claim 93, wherein, for the position information output at the second frame rate that has not been associated with the label information, the processor associates the previously immediately associated label information.

95. The information processing apparatus according to claim 93, wherein, the processor outputs at least one point indicating the existence position of the object as the position information.

96. The information processing apparatus according to claim 93, wherein, the processor derives the tag information based on a first image with a relatively high resolution output at the first frame rate, and derives the position information based on a second image with a relatively low resolution.

97. The information processing apparatus according to claim 2, wherein, the third processor determines an object that satisfies a specified condition among the objects reflected in the first image captured by the first camera and the objects reflected in the second image captured by the second camera as the same object, and associates the point information with the identification information for the objects determined to be the same.

98. The information processing apparatus according to claim 97, wherein, the specified condition includes integrating the positional relationship of the object reflected in the first image and the object reflected in the second image as a first condition, integrating the contour of the object reflected in the first image and the object reflected in the second image as a second condition, and integrating the category of the object estimated for the object reflected in the first image and the category of the object estimated for the object reflected in the second image as a third condition. When at least one of the first condition, the second condition, and the third condition is satisfied, the third processor determines that the object reflected in the first image and the object reflected in the second image are the same object.

99. The information processing apparatus according to claim 1, wherein, the information processing apparatus includes an extraction unit, an integration unit, an estimation unit, and an update unit. The extraction unit extracts first planar coordinates from an image in which an object is reflected, and the first planar coordinates represent the existence position of the object in a two-dimensional space at time t. The integration unit combines the first planar coordinates with first depth information to generate three-dimensional coordinates of the object at time t, and the first depth information is information detected for the depth direction of the object in a three-dimensional space. The extraction unit extracts second planar coordinates, and the second planar coordinates represent the existence position of the object in a two-dimensional space at the next moment of time t. The estimation unit estimates second depth information corresponding to the next moment based on the shape information of the space where the object exists and the change from the first planar coordinates to the second planar coordinates. The update unit integrates the second planar coordinates with the estimated second depth information to update the three-dimensional coordinates of the object at the next moment.

100. The information processing apparatus according to claim 99, wherein, the integration unit detects the first depth information from the point cloud data detected by the sensor.

101. The information processing apparatus according to claim 100, wherein, when the second planar coordinates at the next moment after the object moves are extracted, if the depth information obtained from the point cloud data corresponding to the next moment cannot be acquired, the estimation of the second depth information and the update are performed.

102. The information processing apparatus according to claim 99, wherein, the shape information includes the shape of the road surface and the shape of the object preset, and the shape of the road surface is obtained from the image acquired by the high-resolution camera and the point cloud data detected by the sensor, the estimation unit estimates the second depth information by calculating the amount of movement in the depth direction when the shape of the object moving relative to the shape of the road surface changes from the first planar coordinates to the second planar coordinates.

103. The information processing apparatus according to claim 99, wherein, a point corresponding to the coordinates indicating the existence position of the object is set as each of a plurality of points at the vertices of the frame representing the contour of the object, the extraction unit extracts, for each of the points, the first planar coordinates and the second planar coordinates of the point, the integration unit generates, for each of the points, the three-dimensional coordinates of the object at the point, the estimation unit estimates, for each of the points, the second depth information of the point, the update unit updates, for each of the points, the three-dimensional coordinates of the object at the point.

104. The information processing apparatus according to claim 1, wherein, the information processing apparatus includes a third processor, the third processor associates the point information output from the first processor with the recognition information output from the second processor, when there are a plurality of pieces of point information, the first processor outputs the point information at the next time point according to the priority determined in advance for the recognition information associated with the point information.

105. The information processing apparatus according to claim 104, wherein, at a first time point, the first processor derives, from the image of the object captured by the first camera, the coordinate value in the depth direction of the object of the point indicating the existence position of the object in the three-dimensional orthogonal coordinate system as the point information, when there are a plurality of pieces of point information, the first processor outputs the coordinate value in the depth direction at a second time point according to the priority determined in advance for the recognition information associated with the point information, and the second time point is the next time point after the first time point.

106. The information processing apparatus according to claim 104, wherein, regarding the priority, the first processor determines a high priority for the object having an action according to the presence or absence of the action of the object indicated by the recognition information, and determines a priority lower than the high priority for the object not having an action.

107. The information processing apparatus according to claim 106, wherein, regarding the priority, the first processor determines a relatively high priority corresponding to the risk factor of the object for the object having the action.

108. The information processing apparatus according to claim 107, wherein, when the number of the point information is equal to or more than a specified number, the first processor derives the point information at the next time point according to the priority.

109. The information processing apparatus according to claim 1, wherein, Regarding the point information, at each time point that satisfies the conditions related to the image, the first processor derives, from the image of the object captured by the first camera, the coordinate value in the depth direction of the object in a three-dimensional orthogonal coordinate system of the point representing the existence position of the object as the point information.

110. The information processing apparatus according to claim 109, wherein, the first processor sets each time point that satisfies the specified conditions as a second time point, and derives the coordinate value in the depth direction at the second time point as the point information from the coordinate values in the width direction, height direction, and depth direction of the object in the three-dimensional orthogonal coordinate system at a first time point, and the coordinate values in the width direction and height direction at the second time point, and the second time point is the next time point after the first time point.

111. The information processing apparatus according to claim 110, wherein, the first processor sets the specified conditions as the change amounts of the coordinate values in the width direction and height direction at the second time point with respect to the coordinate values in the width direction and height direction at the first time point, and derives the coordinate value in the depth direction at a predetermined frequency for the change amounts.

112. The information processing apparatus according to claim 1, wherein, regarding the point information, the first processor derives, from the image of the object captured by the first camera, the coordinate value in the depth direction of the object in a three-dimensional orthogonal coordinate system of the point representing the existence position of the object as the point information, the first processor learns correction information for the coordinate value in the depth direction based on specified sensor information.

113. The information processing apparatus according to claim 112, wherein, when the difference between the derived coordinate value in the depth direction and the coordinate value in the depth direction detected by a specified sensor is equal to or greater than a threshold value, the first processor collects the sensor information, and learns the correction information for a specific driving condition in the collected sensor information.

114. The information processing apparatus according to claim 113, wherein, the first processor derives the coordinate value in the depth direction at the second time point as the point information from the coordinate values in the width direction, height direction, and depth direction of the object in the three-dimensional orthogonal coordinate system at a first time point, and the coordinate values in the width direction and height direction at the second time point, and the second time point is the next time point after the first time point.

115. The information processing apparatus according to claim 1, wherein, the information processing apparatus includes: an information acquisition unit that can acquire a plurality of pieces of information related to a vehicle; an inference unit that uses deep learning to infer a plurality of index values based on the plurality of pieces of information acquired by the information acquisition unit; and a driving control unit that performs driving control of the vehicle based on the plurality of index values.

116. The information processing apparatus according to claim 115, wherein, the inference unit infers the plurality of index values based on the multiple pieces of information through multivariate analysis using the integration method of the deep learning.

117. The information processing apparatus according to claim 115, wherein, the information acquisition unit acquires the multiple pieces of information in units of one billionth of a second, and the inference unit and the driving control unit use the multiple pieces of information acquired in units of one billionth of a second to perform the inference of the multiple index values and the driving control of the vehicle in units of one billionth of a second.

118. The information processing apparatus according to claim 115, wherein, the information processing apparatus further includes a strategy setting unit, the strategy setting unit sets a driving strategy for the vehicle to reach the destination, the driving strategy includes at least one of a theoretical value of an optimal route, a driving speed, an inclination, and a braking until the destination, the driving control unit includes a strategy update unit, and the strategy update unit updates the driving strategy based on a difference between the multiple index values and the theoretical values.

119. The information processing apparatus according to claim 115, wherein, the information acquisition unit includes a sensor, and the sensor is provided under the vehicle and can detect the temperature, material, and inclination of the ground on which the vehicle travels.

120. The information processing apparatus according to claim 1, wherein, the information processing apparatus includes: an acquisition unit that acquires a detection result of an object detected in relation to the operation of a control device for controlling the autonomous driving of a vehicle, the control device being mounted on the vehicle; and an execution unit that performs cooling in the control device based on the detection result.

121. The information processing apparatus according to claim 120, wherein, as the detection result, the acquisition unit extracts points representing the existence positions of the object from a single-frame image of the object and acquires motion information at a frame rate of 100 frames per second or more, the motion information representing the motion of the points representing the existence positions of the object along a specified coordinate axis.

122. The information processing apparatus according to claim 121, wherein, the information processing apparatus further includes a prediction unit, the prediction unit uses the detection result to predict the operation of the control device, the prediction unit uses a learning model generated through machine learning to predict the operation of the control device, and the machine learning uses the detection result and the operation status of the control device when the detection result is acquired as learning data.

123. The information processing apparatus according to claim 122, wherein, the prediction unit also predicts the respective temperature changes of multiple parts in the control device, and the execution unit controls the cooling of the parts in the control device.

124. The information processing apparatus according to claim 1, wherein, the information processing apparatus includes: A calculation unit that calculates, for each of a plurality of combinations of a predetermined number of pieces of sensor information among the plurality of sensor information provided in a vehicle, respective wheel speeds, tilts, and index values of each suspension supporting the wheels for controlling the four wheels of the vehicle, and aggregates the index values to calculate control variables for the wheel speeds, the tilts, and each of the suspensions; and A control unit that controls autonomous driving based on the control variables.

125. The information processing device according to claim 124, wherein the calculation unit calculates the control variables based on the index values through multivariate analysis using an integration method based on deep learning.

126. The information processing device according to claim 124, wherein the control unit controls the autonomous driving in units of one billionth of a second based on the control variables.

127. The information processing device according to claim 1, wherein the information processing device includes: An acquisition unit that acquires a plurality of pieces of information related to the vehicle from a detection unit including sensors as a cycle for detecting the surrounding conditions of the vehicle, and the sensors detect the surrounding conditions of the vehicle at a second cycle shorter than a first cycle for photographing the surrounding of the vehicle; A calculation unit that calculates an index value related to the surrounding conditions of the vehicle based on the acquired plurality of pieces of information, and calculates control variables for controlling the behavior of the vehicle based on the calculated index value; and A control unit that controls the behavior of the vehicle based on the calculated control variables.

128. The information processing device according to claim 127, wherein the calculation unit calculates the control variables based on the index values through multivariate analysis using an integration method based on deep learning.

129. The information processing device according to claim 127, wherein the acquisition unit acquires the plurality of pieces of information in units of one billionth of a second, and the calculation unit uses the plurality of pieces of information acquired in units of one billionth of a second to perform the calculation of the index value and the control variables in units of one billionth of a second.

130. The information processing device according to claim 127, wherein the calculation unit predicts a collision of an object with the vehicle based on the acquired plurality of pieces of information, and when the predicted result indicates an inevitable collision, calculates control variables corresponding to damage to the vehicle in the inevitable collision that is less than or equal to a predetermined threshold as the control variables.

131. The information processing device according to claim 130, wherein the damage to the vehicle is at least one of the deformation position and the deformation amount of the vehicle.

132. The information processing device according to claim 130, wherein the control variables corresponding to the damage to the vehicle are at least one of the collision angle and the vehicle speed of the vehicle.

133. The information processing device according to claim 1, wherein The first processor extracts points representing the existence position of the object from the image of the reflected object, and outputs points representing the existence position of the object.

134. The information processing apparatus according to claim 133, wherein, the information processing apparatus includes a camera capable of changing the frame rate, the first processor calculates a score related to the external environment, determines the frame rate of the camera according to the score, outputs a control signal indicating to capture an image at the determined frame rate to the camera, extracts points representing the existence position of the object from the image captured by the camera, and outputs points representing the existence position of the object.

135. The information processing apparatus according to claim 134, wherein, the information processing apparatus is mounted on a vehicle, the first processor calculates, as a score related to the external environment, the degree of danger related to the driving of the vehicle, determines the frame rate of the camera according to the degree of danger, outputs a control signal indicating to capture an image at the determined frame rate to the camera, extracts points representing the existence position of the object from the image captured by the camera, and outputs points representing the existence position of the object.

136. The information processing apparatus according to claim 133, wherein, the first processor extracts an object from the image, when the existence position of the object is in a specified area, extracts points representing the existence position of the object, and outputs points representing the existence position of the object.

137. The information processing apparatus according to claim 133, wherein, the first processor extracts an object from the image, calculates a score for each object, extracts points representing the existence position of the object whose score is above a specified threshold, and outputs points representing the existence position of the object.

138. A vehicle, wherein, the vehicle includes a pair of camera units and a processor including a camera, the first camera unit extracts points representing the existence position of the object from the image of the object reflected in front of the vehicle, and outputs points representing the existence position of the object, the second camera unit extracts points representing the existence position of the object from the image of the object reflected behind the vehicle, and outputs points representing the existence position of the object, the processor controls the driving of the vehicle based on the existence positions of the objects output from the first camera unit and the second camera unit.

139. The vehicle according to claim 138, wherein, the camera included in each camera unit is capable of changing the frame rate, one of the first camera unit and the second camera unit sets the frame rate of the camera according to the detection status of the object in the other of the first camera unit and the second camera unit, extracts points representing the existence position of the object from the image of the camera captured at the set frame rate, and outputs points representing the existence position of the object.

140. The vehicle according to claim 138 or 139, wherein, The vehicle is equipped with a high-resolution camera having a resolution higher than that of the camera, and is also equipped with other camera units each connected to each of the camera units, each of the camera units being capable of acquiring information on an object determined based on information from the high-resolution camera, one of the first camera unit and the second camera unit acquires information on an object that the other of the first camera unit and the second camera unit has.

141. An information processing method, wherein, the following processing is performed by a computer: Based on an image of an object captured by a first camera, point information obtained by capturing the captured object as a point is output, Based on an image of the object captured by a second camera facing a direction corresponding to the first camera, identification information obtained by identifying the captured object is output.

142. An information processing program, wherein, the information processing program causes a computer to perform the following processing, Based on an image of an object captured by a first camera, point information obtained by capturing the captured object as a point is output, Based on an image of the object captured by a second camera facing a direction corresponding to the first camera, identification information obtained by identifying the captured object is output.

Images