Information processing apparatus, sensing apparatus, mobile body, information processing method, and information processing system

Abstract

An information processing apparatus has an input interface, a processor, and an output interface. The input interface acquires observation data obtained from an observation space. The processor detects a subject image of a detection object from the observation data, calculates a plurality of individual indexes each representing a reliability degree related to at least any one of identification information and measurement information of the detection object, and further calculates a comprehensive index that synthesizes the plurality of calculated individual indexes. The output interface outputs the comprehensive index.

Description

[0001] Cross-reference of related applications

[0002] This application claims priority to Japanese Patent Application No. 2019-235115, filed on December 25, 2019, the entire disclosure of which is incorporated herein by reference. Technical Field

[0003] This invention relates to information processing apparatus, sensing device, mobile body, information processing method, and information processing system. Background Technology

[0004] Conventionally, a device has been provided that detects the type, position, size, etc., of an object based on the output of sensors mounted on vehicles, such as vehicle cameras. Such devices may be prone to false detections due to sensor performance and the surrounding environment, as well as detection errors. Therefore, a device or system capable of determining the object being detected and outputting the reliability of the detection result has been proposed (for example, see Patent Document 1).

[0005] For example, Patent Document 1 discloses a camera system that identifies a subject from an image captured by the camera, sequentially measures the three-dimensional position of the subject, and creates information together with the measurement information as an indicator of the reliability of the measurement information.

[0006] Existing technical documents

[0007] Patent documents

[0008] Patent document 1: Japanese Patent Application Publication No. 9-322048. Summary of the Invention

[0009] The information processing device of the present invention includes an input interface, a processor, and an output interface. The input interface acquires observation data obtained from an observation space. The processor detects a target object from the observation data and calculates multiple individual indicators, wherein each individual indicator represents the reliability related to at least one of the target object's identification information and measurement information. The processor also calculates a comprehensive indicator by combining the calculated individual indicators. The output interface outputs the comprehensive indicator.

[0010] The sensing device of the present invention includes a sensor, a processor, and an output interface. The sensor is configured to sense an observation space and acquire observation data of a detected object. The processor detects the detected object from the observation data and calculates multiple individual indicators, wherein each of the multiple individual indicators represents the reliability related to at least one of the object's identification information and measurement information. The processor also calculates a comprehensive indicator by combining the calculated multiple individual indicators. The output interface outputs the comprehensive indicator.

[0011] The mobile body of the present invention includes an information processing device. The information processing device includes an input interface, a processor, and an output interface. The input interface acquires observation data obtained from an observation space. The processor detects a target object from the observation data and calculates multiple individual indicators, wherein each of the multiple individual indicators represents the reliability related to at least one of the identification information and measurement information of the target object. The processor also calculates a comprehensive indicator by combining the calculated multiple individual indicators. The output interface outputs the comprehensive indicator.

[0012] The information processing method of the present invention includes the following steps: acquiring observation data from the observation space, detecting a target object from the observation data, and calculating multiple individual indicators, wherein the multiple individual indicators respectively represent the reliability related to at least one of the identification information and measurement information of the target object. The information processing method further includes the following step: calculating a comprehensive indicator by combining the calculated multiple individual indicators, and outputting the comprehensive indicator.

[0013] The information processing system of the present invention includes: a sensor, an information processing device, and a determination device. The sensor acquires observation data from an observation space. The information processing device includes a processor. The processor detects a target object from the observation data and calculates multiple individual indicators, wherein each of the multiple individual indicators represents the reliability related to at least one of the target object's identification information and measurement information. The processor also calculates a comprehensive indicator by combining the calculated multiple individual indicators. Based on the comprehensive indicator, the determination device determines whether information based on at least one of the identification information and the measurement information from the sensor can be used. Attached Figure Description

[0014] Figure 1 This is a block diagram illustrating the general structure of an information processing system, i.e., an image processing system, in one implementation.

[0015] Figure 2 It means to carry Figure 1 An example of an image processing system for vehicles and subjects.

[0016] Figure 3 yes Figure 1 Functional block diagram of the control section of the image processing device.

[0017] Figure 4 This is a diagram representing an example of a subject image on a moving image.

[0018] Figure 5 It is a diagram illustrating the relationship between the subject in real space, the subject image in a moving image, and the particles in virtual space.

[0019] Figure 6 This is a diagram representing an example of the movement of a point mass in virtual space.

[0020] Figure 7 It is a diagram of the error ellipse representing the estimated position of the subject.

[0021] Figure 8 This is a diagram illustrating the calculation method for the first indicator.

[0022] Figure 9 This is a graph representing the cumulative distribution function in the calculation method of the second index.

[0023] Figure 10 It means Figure 1 A flowchart illustrating an example of processing performed by the control unit of an image processing device.

[0024] Figure 11 This is a block diagram illustrating the schematic structure of a sensing device, i.e., an imaging device, in one embodiment.

[0025] Figure 12 This is a block diagram illustrating an example of the schematic structure of a sensing device containing millimeter-wave radar.

[0026] Figure 13 It means Figure 12 A flowchart illustrating an example of the processing performed by the information processing unit of a sensing device.

[0027] Figure 14 This is a diagram representing an example of observation data mapped to virtual space.

[0028] Figure 15 It means to Figure 14 A clustered graph of the observation data. Detailed Implementation

[0029] In information processing devices that process observation data acquired by sensing devices such as vehicle-mounted cameras, there are situations where the detected objects are diverse and involve multiple individuals. Furthermore, the detected information may include quantitative measurement information as well as identification information such as the category of the detected object. In such cases, creating reliability information for each individual's specific data could become extremely complex.

[0030] If information from multiple reliability metrics can be combined to calculate a reliability metric, it may be easier to use.

[0031] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. The drawings used in the following description are schematic diagrams. The dimensions and proportions in the drawings may not necessarily correspond to reality.

[0032] An example of an information processing system according to one embodiment of the present invention, namely an image processing system 1, includes: an imaging device 10, an image processing device 20, and a determination device 30. The imaging device 10 is an example of a sensor that senses the observation space. The image processing device 20 is an example of an information processing device. Figure 2 As illustrated, the image processing system 1 is mounted on a vehicle 100, which is an example of a moving body.

[0033] like Figure 2 As shown, in this embodiment, in the coordinate system of actual space, the x-axis direction is the width direction of the vehicle 100 on which the imaging device 10 is installed. Actual space is the object used to acquire observation data, i.e., the observation space. The y-axis direction is the direction in which the vehicle 100 moves backward. The x-axis and y-axis directions are parallel to the road surface where the vehicle 100 is located. The z-axis direction is perpendicular to the road surface. The z-axis direction can be referred to as the vertical direction. The x-axis, y-axis, and z-axis directions are orthogonal to each other. The method of using the x-axis, y-axis, and z-axis directions is not limited to this. The x-axis, y-axis, and z-axis directions can be interchanged.

[0034] (Filming device)

[0035] like Figure 1 As shown, the imaging device 10 is configured to include: an imaging optical system 11, an imaging element 12, and a control unit 13.

[0036] The camera device 10 can be installed at various locations on the vehicle 100. The camera device 10 includes, but is not limited to, a front camera, a left-side camera, a right-side camera, and a rear camera. The front camera, left-side camera, right-side camera, and rear camera are respectively installed on the vehicle 100 to capture images of the surrounding areas in front of, to the left of, to the right of, and behind the vehicle 100. In the following embodiment, as an example, ... Figure 2 As shown, the shooting device 10 is mounted on the vehicle 100 with the optical axis pointing downwards from the horizontal direction in order to take pictures of the rear of the vehicle 100.

[0037] The imaging element 12 includes a CCD (Charge-Coupled Device Image Sensor) and a CMOS (Complementary Metal-Oxide-Semiconductor Image Sensor). The imaging element 12 converts the subject image, which is imaged by the imaging optical system 11 onto the imaging surface of the imaging element 12, into an electrical signal. The subject image is the image of the subject being detected. The imaging element 12 is capable of capturing moving images at a specified frame rate. A moving image is an example of observation data. Each still image constituting a moving image is called a frame. The number of images that can be captured in one second is called the frame rate. The frame rate can be set, for example, to 60 fps (frames per second), 30 fps, etc.

[0038] The control unit 13 controls the entire imaging device 10 and performs various image processing operations on the moving images output from the imaging element 12. The image processing performed by the control unit 13 can be any processing such as distortion correction, brightness adjustment, contrast adjustment, and gamma correction.

[0039] The control unit 13 may be composed of one or more processors. For example, the control unit 13 includes one or more circuits or units configured to perform one or more data calculation steps or processes by executing instructions stored in associated memory. The control unit 13 includes one or more processors, microprocessors, microcontrollers, application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), or any combination of these devices or structures, or combinations of other known devices or structures.

[0040] (Image processing device)

[0041] The image processing device 20 can be installed in any location on the vehicle 100. For example, the image processing device 20 can be located in the dashboard, trunk, or under the seats of the vehicle 100. The image processing device 20 is configured to include an input interface 21, a control unit 22, an output interface 23, and a storage unit 24.

[0042] The input interface 21 is configured to communicate with the imaging device 10 via a wired or wireless communication unit. The input interface 21 is configured to acquire moving images from the imaging device 10. The input interface 21 can correspond to the transmission method of the image signal sent by the imaging device 10. The input interface 21 can be referred to as an input unit or an acquisition unit. The imaging device 10 and the input interface 21 can be connected via an in-vehicle communication network such as CAN (Controller Area Network).

[0043] The control unit 22 controls the image processing device 20 as a whole. The control unit 22 detects a subject image from a moving image acquired via the input interface 21 and calculates multiple individual indicators, each representing a reliability related to at least one of the recognition information and measurement information of the subject image. The control unit 22 also calculates a comprehensive indicator, which combines the calculated individual indicators. The control unit 22 is the same as the control unit 13 of the imaging device 10, and is configured to include one or more processors. Furthermore, like the control unit 13, the control unit 22 can be configured to combine multiple types of devices. The control unit 22 can be referred to as a processor or a controller.

[0044] In this embodiment, "identification information" refers to information representing the characteristics of the subject 40 detected from the moving image. The "identification information" includes the "type," "color," and "brightness" of the subject 40 detected from the moving image. The "type" of the subject 40 is the type of object categorized according to its class. The type of the subject 40 can be referred to as the "class" of the subject. The "identification information" is determined from one of several classes. Examples of classes for the "type" of the subject 40 include "vehicle," "pedestrian," and "two-wheeled vehicle." When the subject 40 is an object on the road, the control unit 22 determines the "type" as one of several classes. Examples of classes for the "color" of the subject 40 include "red," "blue," and "yellow." When the subject 40 is a traffic light located on the roadside, the control unit 22 can determine "red," "blue," and "yellow" as the color class. The "brightness" of the subject 40 is divided into multiple categories such as "bright" and "dark" by setting a threshold based on the brightness detected by the imaging element 12.

[0045] In this embodiment, "measurement information" is quantitative information obtained by processing and calculating the image of the subject 40 contained in the moving image in consecutive frames. "Measurement information" includes, for example, the "position," "distance," and "size" of the subject 40 detected from the moving image. "Position" refers, for example, to the coordinates of the subject 40 in actual space. "Distance" is, for example, the distance from the vehicle 100, especially the capturing device 10, to the subject 40. "Size" refers to the dimensions of the subject 40, such as its width and height in actual space.

[0046] In this embodiment, "reliability" refers to the accuracy of the identification and measurement information calculated by the control unit 22. "Reliability" is a relative concept. The "reliability" of this invention can be represented by a reliability index using a value from 0 to 1. For example, a smaller reliability index indicates lower reliability, and vice versa.

[0047] like Figure 3 As shown, the control unit 22 is configured to include functional blocks comprising: an image recognition unit 51, a measurement information calculation unit 52, a measurement information correction unit 53, a first index calculation unit 54, a second index calculation unit 55, and a comprehensive index calculation unit 56. The processing performed by each functional block will be described in detail below.

[0048] The output interface 23 is configured to output an output signal as output information from the image processing device 20. The output interface 23 can be referred to as an output unit. The output interface 23 can output the recognition information and measurement information calculated by the control unit 22, as well as a comprehensive index representing the reliability of this information, to the outside of the image processing device 20. For example, the output interface 23 can output the aforementioned recognition information, measurement information, and comprehensive index to a determination device 30 outside the image processing device 20. The determination device 30 may be included in collision prevention devices, alarm devices, etc.

[0049] The output interface 23 may be configured to include at least one of a physical connector and a wireless communication device. In one of several embodiments, the output interface 23 may be connected to a network of the vehicle 100, such as CAN. The output interface 23 may be connected to a determination device 30, a collision avoidance device of the vehicle 100, and a distance warning device via a communication network such as CAN.

[0050] Storage unit 24 is a storage device for data and programs required for processing by storage control unit 22. For example, storage unit 24 temporarily stores moving images acquired from imaging device 10. For example, storage unit 24 sequentially stores data generated by processing performed by control unit 22. Storage unit 24 can be configured using any one or more of the following: semiconductor memory, magnetic memory, and optical memory. Semiconductor memory can include volatile memory and non-volatile memory. Magnetic memory can include, for example, hard disk and magnetic tape. Optical memory can include, for example, CD (Compact Disc), DVD (Digital Versatile Disc), and BD (Blu-ray Disc).

[0051] The determination device 30 determines whether information from at least one of the identification information and measurement information related to each subject 40 can be used based on a comprehensive index output from the image processing device 20. The determination device 30 can be integrated into other devices that use identification and measurement information. Devices having the determination device 30 include those providing functions such as collision warning, automatic braking, and automatic steering. For example, the determination device 30 assumes that the image processing device 20 detects a specific type of subject 40 and its distance to the subject 40 in the direction of travel of the vehicle 100. When the value of the comprehensive index is close to 0 and the reliability is extremely low, the determination device 30 can choose not to use the information related to the subject 40 output by the image processing device 20. When the value of the comprehensive index is close to 1 and the reliability is high, the determination device 30 can use the information related to the subject 40 output by the image processing device 20 to perform vehicle control, etc.

[0052] (Functions of the control department)

[0053] The following is for reference Figures 3 to 9 The functional blocks of the control unit 22 will be described below. Each functional block of the control unit 22 can be a hardware module or a software module. The control unit 22 is capable of executing the actions performed by each functional block described below. The control unit 22 can execute all actions of each functional block. The actions performed by each functional block can be referred to as actions performed by the control unit 22. The processing performed by the control unit 22 using any one of the functional blocks can be considered as processing performed by the control unit 22 itself.

[0054] (Image Recognition Department)

[0055] The image recognition unit 51 acquires frames of the moving image from the capturing device 10 via the input interface 21. Figure 4 An example of a one-frame image of a moving image is shown. Figure 4In the example, a subject image 42a of a pedestrian attempting to cross behind vehicle 100 and subject images 42b of other vehicles traveling behind vehicle 100 are displayed in a two-dimensional image space 41 defined by the u-v coordinate system. Image space is an example of display space. Display space is the space in which the detected object is represented two-dimensionally for visual recognition by a user or for use by other devices. The u-coordinate is the horizontal coordinate of the image. The v-coordinate is the vertical coordinate of the image. Figure 4 In this diagram, the origin of the uv coordinate system is the point at the top left of image space 41. Furthermore, the positive direction of the u-coordinate is defined as the direction from left to right, and the positive direction of the v-coordinate is defined as the direction from top to bottom.

[0056] The image recognition unit 51 detects subject images 42a and 42b (hereinafter, collectively referred to as subject image 42) from each frame of the moving image. The detection method for subject image 42 includes various known methods. For example, the recognition method for subject image 42 includes: methods based on shape recognition of objects such as vehicles and pedestrians, methods based on template matching, methods that calculate feature quantities from the image and use them for matching, etc. In the case of a subject image 42 detection method using feature quantities, a function approximator capable of learning the relationship between input and output can be used in the calculation of the feature quantities. The function approximator capable of learning the relationship between input and output can use a neural network. For example, the image recognition unit 51 can use a function approximator that has previously learned feature quantities of various categories of subject 40 types such as vehicles, pedestrians, and two-wheeled vehicles through deep learning based on a neural network to calculate the feature quantities.

[0057] The image recognition unit 51 outputs information about the type of the subject 40 as an example of recognition information. For each detected subject image 42, the image recognition unit 51 calculates the category of the subject 40 and the probability of belonging to each category. The probability of belonging to a category represents the probability of being assigned to that category. For example, when the category of the subject image 42a is set to "vehicle," "pedestrian," or "two-wheeled vehicle," the image recognition unit 51 calculates the probability of belonging to "vehicle" as 0.7, "pedestrian" as 0.2, and "two-wheeled vehicle" as 0.1. In this case, there are three categories. The image recognition unit 51 determines "vehicle," which has the highest probability of belonging, as the category of the subject image 42a.

[0058] The image recognition unit 51 can output information representing the position and size in the image space 41 for each subject image 42 detected for each frame. For example, the image recognition unit 51 can determine the horizontal (u direction) and vertical (v direction) range occupied by the subject image 42 in the image space 41 as the size of the subject image 42. For example, in Figure 4The area occupied by the subject image 42 within the image space 41 is represented by a rectangular frame. Furthermore, the image recognition unit 51 determines the position of the subject image 42 within the image space 41. For example, the image recognition unit 51 determines the point at the center of the lowest part of the area occupied by each subject image 42a, 42b within the image space 41 as representative points 43a, 43b (hereinafter, appropriately referred to collectively as representative point 43) representing the positions of the subject images 42a, 42b. It is assumed that this representative point 43 is the position where the subject 40 corresponding to the subject image 42 contacts the road surface or ground.

[0059] The image recognition unit 51 can output the detected recognition information to the outside of the image processing device 20 via the output interface 23. The image recognition unit 51 transmits the recognition information of each subject 40 and the probability of belonging to each category of the recognition information to the first index calculation unit 54. The image recognition unit 51 transmits the position and size information of the subject image 42 in the image space 41 to the measurement information calculation unit 52.

[0060] (Measurement Information Calculation Department)

[0061] The measurement information calculation unit 52 calculates measurement information based on the position and size of the subject image 42 detected in the image recognition unit 51 in the image space 41. For example, the measurement information calculation unit 52 calculates the position of the subject 40 by mapping the position of the representative point 43 determined by the image recognition unit 51 to the actual space.

[0062] For example, in Figure 5The diagram illustrates the relationship between the subject 40 located in three-dimensional actual space and the subject image 42 on two-dimensional image space 41. Given the internal parameters of the imaging device 10, the direction from the center of the imaging optical system 11 of the imaging device 10 toward the corresponding coordinates (x, y, z) in actual space can be calculated based on the coordinates (u, v) in image space 41. The internal parameters of the imaging device 10 include information such as the focal length and distortion of the imaging optical system 11 and the pixel size of the imaging element 12. In actual space, the point where the straight line pointing in the direction corresponding to the representative point 43 in image space intersects the reference plane 44 at z = 0 is taken as the mass point 45 of the subject 40. The reference plane 44 corresponds to the road surface or ground where the vehicle 100 is located. The mass point 45 has three-dimensional coordinates (x, y, 0). Therefore, when the two-dimensional plane at z = 0 is taken as the virtual space 46, the coordinates of the mass point 45 can be represented by (x', y'). The virtual space is the virtual space used in the control unit 22 to describe the motion of the object. The coordinates (x', y') of particle 45 in virtual space 46 are equivalent to the coordinates (x, y) of a specific point of the subject 40 on the xy plane (z = 0) in actual space when viewed from along the z-axis. The specific point is the point corresponding to particle 45.

[0063] The method of detecting the position of subject 40 using subject image 42 is not limited to... Figure 5 This is a mapping transformation-based method as shown. For example, by using multiple imaging devices 10 to construct a stereo camera, the measurement information calculation unit 52 can detect the three-dimensional position of the subject 40. In this case, the measurement information calculation unit 52 can calculate the distance from the imaging device 10 to each pixel of the subject image 42 based on the parallax of the images acquired from the multiple imaging devices 10, whose optical axes are set to be parallel to each other. In this case, the moving images acquired from the multiple imaging devices 10 are included in the observation data.

[0064] Additionally, the measurement information calculation unit 52 can obtain the distance to the subject 40 from a distance measuring device mounted on the vehicle 100, and combine it with information from the subject image 42 detected by the image recognition unit 51 to calculate the position of the subject 40 in actual space. The distance measuring device includes LiDAR (Laser Imaging Detection and Range), millimeter-wave radar, lidar, ultrasonic sensors, and stereo cameras. The distance measuring device is not limited to the aforementioned devices; various devices capable of distance measurement can also be used. In this case, in addition to the moving image obtained from the imaging device 10, the observation data also includes distance data to the subject 40 obtained from the distance measuring device.

[0065] The measurement information calculation unit 52 can calculate the size of the subject 40 in the actual space based on the size of the subject image 42 in the image space 41 and the distance to the subject 40.

[0066] (Measurement Information Correction Department)

[0067] When subject images 42 overlap in image space 41, the image recognition unit 51 sometimes fails to perform correct detection. Furthermore, the detection of measurement information by the measurement information calculation unit 52 may include errors caused by various factors such as the accuracy of the imaging device 10, vibration during shooting, and ambient brightness. Therefore, the measurement information correction unit 53 estimates and corrects the measurement information based on the measurement information of the subject 40 obtained from consecutive frames of images. The case where the measurement information calculation unit 52 calculates the coordinates (x', y') of the mass point 45 in virtual space 46 as the position of the subject 40 will be explained. Figure 6 As shown, the measurement information correction unit 53 tracks the position (x', y') and velocity (v) of the mass point 45 in the virtual space 46, which is mapped from the representative point 43 of the subject image 42 to the virtual space 46. x' v y' ). Assumption Figure 6 In this context, k-1, k, and k+1 represent the frame numbers corresponding to particle 45. Since particle 45 has a position (x', y') and a velocity (v... x' v y' The control unit 22 can predict the range of the position (x', y') of the particle 45 in consecutive frames by receiving information from the image of the subject 42. The control unit 22 can identify the particle 45 located within the predicted range in the next frame as the particle 45 corresponding to the tracked subject image 42. Each time a new frame is received, the control unit 22 sequentially updates the position (x', y') and velocity (v) of the particle 45. x' v y' ).

[0068] For example, estimations using a Kalman filter based on a state-space model can be used for tracking point 45. By performing predictions / estimations using a Kalman filter, robustness against undetectable or falsely detected objects 40 to be tracked is improved. Typically, it is difficult to describe the object image 42 in image space 41 using an appropriate model for describing motion. Therefore, it is difficult to simply estimate the position with high accuracy. In the image processing apparatus 20 of the present invention, by mapping and transforming the object image 42 into a point 45 in a virtual space corresponding to the xy plane of the actual space, a model for describing motion in the actual space can be applied, thus improving the accuracy of tracking the object 40. Furthermore, by treating the object 40 as a point 45 without size, simple and easy tracking can be achieved.

[0069] Whenever a frame of a moving image is acquired from the capturing device 10, the measurement information correction unit 53 estimates the correct position of the subject 40 at that time point. In estimating the position of the subject 40, the measurement information correction unit 53 can assume a normal distribution. The measurement information correction unit 53 calculates the estimated position of the subject 40 and calculates the error covariance matrix. By using the error covariance matrix, the measurement information correction unit 53 can define the probability density distribution of the position of the subject 40 in the virtual space 46. If the error covariance matrix is ​​used, the range of error can be represented by an error ellipse, thereby enabling the image processing device 20 to evaluate the accuracy of the estimated range as defined by specifications. An error ellipse is an ellipse representing a statistically determinable range in which the true value lies with a specified probability.

[0070] like Figure 7 As shown, the error ellipse 47 can be displayed as an elliptical region around the particle 45 of the subject 40 in the virtual space 46. When viewed along the x' and y' axes, the probability density distributions of the particle 45 are one-dimensional normal distributions, respectively. Figure 7 In the diagram, the probability density distributions along the x' and y' axes are shown as curves on the x' and y' axes, respectively. The two intrinsic values ​​λ1 and λ2 of the error covariance matrix correspond to the major and minor axis lengths of the error ellipse. λ1λ2π is the area of ​​the error ellipse.

[0071] Whenever measurement information is acquired from images in each frame, the measurement information correction unit 53 corrects the measurement information calculated by the measurement information calculation unit 52 based on estimated values ​​such as the position of the mass point 45. The measurement information correction unit 53 outputs the corrected measurement information via the output interface 23. The measurement information correction unit 53 calculates the error covariance matrix or its intrinsic value and transmits it to the second index calculation unit 55.

[0072] (First Indicator Calculation Department)

[0073] The first index calculation unit 54 calculates a first index related to the recognition information based on the classification probability of each category of the recognition information calculated by the image recognition unit 51. The first index is also called an object recognition index. The first index is an individual index. When a vector containing the classification probability of each category of the recognition information is set as the classification probability vector v... p When the number of categories is set to N, the first index calculation unit 54 calculates the first index related to the identification information using the following mathematical formula.

[0074] [Mathematical Expression 1]

[0075]

[0076] in,

[0077] [Mathematical Expression 2]

[0078] ||v p ||2

[0079] This refers to the attribution probability vector v p The L2 norm. The L2 norm is also called the Euclidean norm.

[0080] The first index is a value above 0 and below 1. The first index increases when the probability of belonging to a specific category is close to 1. The first index decreases when there is no difference in the probability of belonging to categories. For example, in the case of two categories with probabilities p1 and p2 respectively, equation (1) is as follows. The sum of p1 and p2 is always 1.

[0081] [Mathematical Expression 3]

[0082]

[0083] Figure 8 This means setting the values ​​of p1 and p2 as the coordinates of the two axes, and using them as the belonging probability vector v. p Example of v p1 v p2 Attribution probability vector v p The front end lies on the straight line connecting points (1, 0) and (0, 1). When the combination of the probability of belonging to the two categories (p1, p2) is (0.5, 0.5), the value of equation (2) is the minimum value of 0. The closer the combination of the probability of belonging to the two categories (p1, p2) is to (1, 0) or (0, 1), the closer the value of equation (2) is to 1. It can be said that the closer the value of equation (2) is to 1, the more clearly the image recognition unit 51 determines the category of the recognition information.

[0084] Even when N is 3 or more, similar determinations remain valid. For example, if the subject 40 is categorized into three types: vehicle, pedestrian, and two-wheeled vehicle, and its classification probability vector v is used... p For v p The case of (0.9, 0.1, 0.0) is different from v. p Comparing the cases with (0.6, 0.2, 0.2), the first index is 0.77650 in the former case and 0.20341 in the latter case. In both cases, the image recognition unit 51 identifies the subject 40 as a vehicle, but the first index shows that the reliability of the former is higher than that of the latter.

[0085] The first index calculation unit 54 outputs the first index to the comprehensive index calculation unit 56. The first index is not limited to a reliability index related to the type of the subject 40. The first index may include, for example, indices such as color and brightness. The first index is not limited to one and may include multiple indices.

[0086] (Second Indicator Calculation Department)

[0087] The second index calculation unit 55 calculates the second index based on the inherent value of the error covariance matrix calculated by the measurement information correction unit 53. The second index is an individual index. The inherent value of the error covariance matrix can be used to evaluate the accuracy of the estimated range of the measurement information of the subject 40. The second index calculation unit 55 calculates the second index based on the accuracy of the estimated range of the position of the subject 40.

[0088] Generally, it can be assumed that the estimation error of the position of an object on a two-dimensional plane follows a chi-square distribution with 2 degrees of freedom, which is used in statistics. In a chi-square distribution with 2 degrees of freedom, the probability density function (PDF) is known to be Equation (3).

[0089] [Mathematical Expression 4]

[0090]

[0091] In addition, in a typical chi-square distribution with 2 degrees of freedom, the cumulative distribution function (CDF) is known to be Equation (4).

[0092] [Mathematical Expression 5]

[0093]

[0094] The second index calculation unit 55 calculates the chi-square value χ using the following formula (5) when the inherent values ​​of the error covariance matrix are set to λ1 and λ2, and the one-sided tolerance margins in the x and y directions, which are allowed as the error range of the image processing device 20, are set to d1 and d2. 2 One of the x-direction and the y-direction is the first direction, and the other is the second direction. The tolerance margins in the x-direction and the y-direction represent the error range relative to the expected true value guaranteed by the image processing device 20 to the user or downstream system. For example, if the measured value in the x-direction is 10m and the tolerance guaranteed by the image processing device 20 is 10%, the one-sided tolerance margin in the x-direction is 1m.

[0095] [Mathematical Expression 6]

[0096]

[0097] On the right-hand side of equation (5), the denominator is equivalent to the product of the major and minor axes of the error ellipse representing the range of the standard deviation σ of the error variance. The χ calculated using equation (5) 2 The value is equivalent to the ratio of the area of ​​the tolerance circle to the area of ​​the error ellipse. The second index calculation unit 55 calculates χ using equation (5). 2 The second index is calculated using equation (4). The second index can be called the object detection index.

[0098] As is known in statistics, the cumulative distribution function with 2 degrees of freedom is derived from... Figure 9 That kind of graph is represented. The horizontal axis represents x. 2 The vertical axis represents the cumulative distribution, which is equivalent to the second indicator. χ² 2 The closer the value of χ² is to 0, the larger the error ellipse, and the closer the second index becomes to 0. In this case, the reliability of the position estimation can be judged to be low. 2 The larger the value of the second index, the closer it is to infinity, the smaller the error ellipse, and the closer the second index becomes to 1. In this case, the reliability of the position estimation can be judged to be relatively high. The second index takes a value above 0 and below 1.

[0099] The second index calculation unit 55 outputs the second index to the comprehensive index calculation unit 56. The second index is not limited to an index related to the object's position. The second index may include an index that quantifies the reliability of the distance to the subject 40 and the size of the subject 40. The second index is not limited to one, but may include multiple.

[0100] (Comprehensive Indicator Calculation Department)

[0101] The comprehensive index calculation unit 56 calculates a comprehensive index, which is an index that combines multiple individual indices, including the first index calculated by the first index calculation unit 54 and / or the second index calculated by the second index calculation unit 55. The comprehensive index is also called the object detection index. Each individual index takes a value greater than 0 and less than 1. The multiple individual indices are composed of n individual indices a. i In the case where i is a natural number from 1 to n, the comprehensive index calculation unit 56 calculates the comprehensive index a according to the following formula (6). total Comprehensive index a total Take a value greater than 0 and less than 1.

[0102] [Mathematical Expression 7]

[0103]

[0104] It can also replace equation (6) in individual indicators a i The composite index a is calculated by weighting the values ​​among them. total For example, comprehensive index a total It can be calculated using equation (7).

[0105] [Mathematical Expression 8]

[0106]

[0107] In equation (7), w i Indicates individual indicator a i The weight.

[0108] The comprehensive index calculation unit 56 can calculate the comprehensive index a total The output is sent to the outside of the image processing device 20, such as the determination device 30, via the output interface 23. The control unit 22 can output the comprehensive index a. total The image recognition unit 51 detects the type, color, and brightness of the subject image 42, and the measurement information, such as the corrected position, distance, and size of the subject 40, is output together with the measurement information correction unit 53. The comprehensive index calculation unit 56 can output a comprehensive index a for each subject image 42 identified by the image recognition unit 51. total .

[0109] (Processing flow of the image processing device)

[0110] Next, refer to Figure 10The flowchart below details an image processing method according to an embodiment of the present invention. The image processing method is an example of an information processing method. The image processing apparatus 20 may be configured to read a program recorded on a non-transitory computer-readable medium to execute the processing performed by the control unit 22 described below. Non-transitory computer-readable media include, but are not limited to, magnetic storage media, optical storage media, opto-magnetic storage media, and semiconductor storage media. Magnetic storage media include magnetic disks, hard disks, and magnetic tapes. Optical storage media include optical discs such as CDs (Compact Discs), DVDs, and Blu-ray discs. Semiconductor storage media include ROM (Read Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), and flash memory.

[0111] Figure 10 The flowchart describes the process of acquiring consecutive frames of a moving image and executing it by the control unit 22. The control unit 22 of the image processing apparatus 20, according to... Figure 10 The flowchart shows that each time a frame of a moving image is acquired, the processing from step S101 to step S108 is performed.

[0112] First, the control unit 22 acquires frames of moving images from the imaging device 10 via the input interface 21 (step S101).

[0113] If a frame of a moving image is acquired in step S101, the control unit 22 detects the subject image 42 from the image of that frame using the image recognition unit 51 (step S102). The control unit 22, together with the detected subject image 42, calculates the category of the subject 40's recognition information and the probability of belonging to that category. Additionally, the control unit 22 calculates information such as the position and size of the subject image 42 in the image space 41.

[0114] In the first index calculation unit 54, the control unit 22 calculates a first index representing the reliability of the identification information based on the belonging probability of each category of the identification information calculated in step S102 (step S103). The first index takes a value greater than or equal to 0 and less than or equal to 1.

[0115] The control unit 22 executes steps S104 to S106 in parallel with step S103, or before or after step S103.

[0116] In the measurement information calculation unit 52, the control unit 22 calculates the position, distance, size and other measurement information of the subject 40 in the actual space based on the position and size information of the subject image 42 in the image space 41 calculated in step S102 (step S104).

[0117] In the measurement information correction unit 53, the control unit 22 corrects the measurement information calculated by the measurement information calculation unit 52 (step S105). The control unit 22 uses the estimated values ​​of measurement information from past frames and the measurement information calculated in step S104 to estimate the correct measurement information. The control unit 22 calculates the error distribution together with the estimated values ​​of the calculated measurement information. The error distribution is calculated as an error covariance matrix.

[0118] In the second index calculation unit 55, the control unit 22 calculates a second index representing the reliability of the measurement information using the inherent value of the error covariance matrix calculated in step S105 (step S106). The second index takes a value greater than or equal to 0 and less than or equal to 1.

[0119] In the comprehensive index calculation unit 56, the control unit 22 calculates a comprehensive index that combines the first and second indices based on the first and second indices calculated in steps S103 and S106 (step S107). The comprehensive index takes a value greater than or equal to 0 and less than or equal to 1.

[0120] The control unit 22 outputs the comprehensive index calculated in step S107 to the outside (step S108). The control unit 22 can output the recognition information of the subject image 42 detected in step S102 and the measurement information of the subject 40 corrected in step S105 together with the comprehensive index.

[0121] As described above, in the image processing apparatus 20 of this embodiment, the control unit 22 can integrate multiple reliability information and calculate a reliability index for a subject 40 of a detection target. Therefore, the control unit 22 can output a comprehensive index for each subject 40 identified as a subject image 42 through image recognition.

[0122] Furthermore, by using a single index to represent the reliability of the information detected for each subject 40, it becomes easier for the determination device 30, located downstream of the image processing device 20, to determine the degree to which the output of the image processing device 20 should be trusted. For example, assuming the determination device 30 is integrated into a system such as a collision avoidance or lane departure warning device for a vehicle 100, the determination device 30 acquires the combined index calculated by multiple image processing devices 20. The determination device 30 can perform processing corresponding to the reliability based on the values ​​of the multiple combined indices output from the multiple image processing devices 20. Assuming there are differences between the information from the multiple image processing devices 20, if the value of the combined index output by one image processing device 20 is closer to 1, the determination device 30 can trust the information based on that image processing device 20 more and prioritize using that information.

[0123] The first index in this embodiment is calculated according to Equation (1) based on the probability of belonging to each category of identification information, which includes multiple categories. Thus, the reliability of the identification information can be calculated as a consistent index from 0 to 1 through simple calculation.

[0124] The second index in this embodiment is calculated according to equations (4) and (5) based on the error covariance matrix of the measurement information and the tolerance guaranteed by the image processing device 20. Thus, the reliability of the measurement information can be calculated as a consistent index from 0 to 1 through simple calculation.

[0125] The image processing apparatus 20 of the present invention can consistently process reliability indices because it uses values ​​of 0 or higher and 1 or lower to represent all reliability indices of the first index, the second index, and the comprehensive index. For example, even when the number of reliability indices to be considered increases, the reliability indices can be easily combined into a single index using equation (6) or equation (7).

[0126] (The imaging device used to calculate reliability indicators)

[0127] The image processing apparatus 20 of the present invention described in the above embodiments can be incorporated into a shooting device. Figure 11 This is a schematic diagram illustrating an imaging device 60 of one embodiment of the present invention, which includes the functions of an image processing device 20. The imaging device 60 includes: an imaging optical system 61, an imaging element 62, a control unit 63, an output interface 64, and a storage unit 65. The imaging optical system 61 and the imaging element 62 are... Figure 1 The imaging optical system 11 and imaging element 12 of the imaging device 10 have similar structural elements. The output interface 64 and the storage unit 65 are similar to those of the imaging device 10. Figure 1The image processing device 20 has similar structural elements as its output interface 23 and storage unit 24. The control unit 63 is also a combination of... Figure 1 The structural elements that control the camera 10 control unit 13 and the image processing device 20 control unit 22.

[0128] In the imaging device 60, the imaging element 62 captures a moving image of the subject 40 imaged by the imaging optical system 61. Regarding the moving image output by the imaging element 62, the control unit 63 performs [operations related to...]. Figure 10 The process described in the flowchart is the same as or similar to the process described in the flowchart. Therefore, the imaging device 60 can calculate the recognition information and measurement information of the subject image 42, and can calculate a comprehensive index representing the reliability of this information. The calculated comprehensive index is output to the outside of the imaging device 60 via the output interface 64. Thus, it is possible to obtain... Figure 1 The image processing device 20 of the image processing system 1 shown has a similar effect.

[0129] In the above embodiment, the information processing device is described as the image processing device 20, and the sensor is described as the imaging device 10. The sensor is not limited to an imaging device that detects visible light, but also includes a far-infrared camera that acquires images using far-infrared light. Furthermore, the information processing device of the present invention is not limited to a device that acquires moving images as observation data and detects a detected object through image recognition. For example, the sensor can be a sensor other than an imaging device capable of sensing the observation space of the observed object to detect its direction and size. Sensors include, for example, sensors that use electromagnetic waves or ultrasound. Sensors using electromagnetic waves include millimeter-wave radar and LiDAR (Laser Imaging Detection and Ranging). Therefore, the detected object is not limited to the subject captured as an image. The information processing device can acquire observation data output from the sensor, including information such as the direction and size of the detected object, thereby detecting the detected object. Furthermore, the display space is not limited to an image space that displays moving images, but can also be a space capable of displaying the detected object in two dimensions.

[0130] (Including sensing devices for millimeter-wave radar)

[0131] As an example, refer to Figure 12 A sensing device 70 according to one embodiment will be described. The sensing device 70 includes a millimeter-wave radar 71, an information processing unit 72, and an output unit 73, which are examples of sensors. The sensing device 70 can be mounted in various locations on a vehicle, similar to the imaging device 10.

[0132] The millimeter-wave radar 71 can use millimeter-wave electromagnetic waves to detect the distance, speed, and direction of a target object. The millimeter-wave radar 71 includes a signal generation unit 74, a high-frequency circuit 75, a transmitting antenna 76, a receiving antenna 77, and a signal processing unit 78.

[0133] The transmitting signal generation unit 74 generates a chirp signal after frequency modulation. A chirp signal is a signal whose frequency rises or falls at regular time intervals. The transmitting signal generation unit 74 is, for example, installed in a DSP (Digital Signal Processor). The transmitting signal generation unit 74 can be controlled by the information processing unit 72.

[0134] After the linear frequency modulated signal is converted by a D / A converter, it undergoes frequency conversion in the high-frequency circuit 75 to become a high-frequency signal. The high-frequency circuit 75 transmits the high-frequency signal as a radio wave into the observation space via the transmitting antenna 76. The high-frequency circuit 75 can receive the reflected wave from the radio wave transmitted from the transmitting antenna 76 after it has been reflected by the target object as a received signal via the receiving antenna 77. The millimeter-wave radar 71 may have multiple receiving antennas 77. The millimeter-wave radar 71 can estimate the direction of the target object by detecting the phase difference between each receiving antenna in the signal processing unit 78. The orientation detection method in the millimeter-wave radar 71 is not limited to the phase difference method. The millimeter-wave radar 71 can also detect the orientation of the target object by using a beam scan in the millimeter-wave band.

[0135] The high-frequency circuit 75 amplifies the received signal and mixes it with the transmitted signal to transform it into a beat signal representing the frequency difference. The beat signal is converted into a digital signal and output to the signal processing unit 78. The signal processing unit 78 processes the received signal, performing estimation processing for range, velocity, and direction, etc. The estimation methods for range, velocity, and direction in the millimeter-wave radar 71 are well known, so the processing performed by the signal processing unit 78 is omitted. The signal processing unit 78 is, for example, installed in a DSP. The signal processing unit 78 can be installed in the same DSP as the transmitted signal generation unit 74.

[0136] The signal processing unit 78 outputs the estimated distance, speed, and direction, etc., as observation data of the detected object to the information processing unit 72. The information processing unit 72 can map and transform the detected object onto a virtual space based on the observation data and perform various processes. The information processing unit 72 consists of one or more processors similar to the control unit 13 of the imaging device 10. The information processing unit 72 can control the entire sensing device 70. The processing performed by the information processing unit 72 will be described further later.

[0137] The output unit 73 is an output interface that outputs the results processed by the information processing unit 72 to an external display device or an ECU inside the vehicle, which is connected to the sensing device 70. The output unit 73 may include a communication processing loop and a communication connector for connecting to a vehicle network such as CAN.

[0138] The following is for reference Figure 13 The flowchart illustrates a portion of the processing performed by the information processing unit 72.

[0139] The information processing unit 72 acquires observation data from the signal processing unit 78 (step S201).

[0140] Next, the information processing unit 72 maps the observation data onto the virtual space (step S202). Figure 14 An example of observation data mapped onto a virtual space is shown. The observation data from the millimeter-wave radar 71 is obtained as point information, where each point includes information on distance, velocity, and direction. The information processing unit 72 maps each observation data onto a horizontal plane. Figure 14 In the diagram, the horizontal axis is the x-axis, with its center at 0 and measured in meters. The vertical axis is the y-axis, with its nearest point at 0 and measured in meters, representing the distance in the depth direction.

[0141] Next, the information processing unit 72 clusters the set of points in the virtual space to detect the target object (step S203). Clustering refers to extracting a group of points as a set of points from the data representing each point. For example... Figure 15 As shown in the dashed ellipse, the information processing unit 72 can extract a set of points representing the points of the observation data. The information processing unit 72 can determine that a detection object actually exists in a portion of the multiple sets of observation data. Conversely, it can determine that the observation data corresponding to each discrete point is generated due to observation noise. The information processing unit 72 can set thresholds such as the number or density of points corresponding to the observation data to determine whether the set of observation data is a detection object. The information processing unit 72 can estimate the size of the detection object based on the size of the area occupied by the point set.

[0142] Next, the information processing unit 72 tracks the position of each detected point group within the virtual space (step S204). The information processing unit 72 can use the center of the area occupied by each point group or the average of the coordinates of the points contained within the point group as the position of each point group. By tracking the movement of the point groups, the information processing unit 72 grasps the movement of the detected object in a time-series manner. Each time the position of a point group in the virtual space is acquired, the information processing unit 72 estimates the correct position of the detected object at that time point. In estimating the position of the detected object, the information processing unit 72... Figure 3Similarly, the measurement information correction unit 53 can calculate the estimated position and the error covariance matrix.

[0143] Following or in parallel with step S204, the information processing unit 72 estimates the category of the detected object corresponding to each point group (step S205). The categories of detected objects include "vehicles," "pedestrians," and "two-wheeled vehicles," etc. The category of the detected object can be determined using any one or more of the following: the object's speed, size, shape, position, the density of observation data points, and the intensity of the detected reflected wave. For example, the information processing unit 72 can accumulate the Doppler velocities of the detected objects obtained from the signal processing unit 78 in a time series manner and estimate the category of the detected object based on the distribution pattern of the Doppler velocities. Furthermore, the information processing unit 72 can estimate the category of the detected object based on the size information of the detected object estimated in step S203. Further, the information processing unit 72 can estimate the category of the detected object by obtaining the intensity of the reflected wave corresponding to the observation data from the signal processing unit 78. For example, since vehicles containing a large amount of metal have a large reflective cross-sectional area, the intensity of the reflected wave is stronger compared to pedestrians with smaller reflective cross-sectional areas. The information processing unit 72 can estimate the category of the object being detected and calculate the reliability of the estimate's accuracy.

[0144] After step S205, the information processing unit 72 maps and transforms the detected object from the virtual space to the display space (step S206). The display space can be a three-dimensional observation space represented by a two-dimensional plane, similar to image space, and viewed from the user's viewpoint. The display space can be a two-dimensional space viewing the observed object from the z-axis direction (vertical direction). Alternatively, the information processing unit 72 can directly map the observation data acquired from the signal processing unit 78 in step S201 to the display space without going through steps S203 to S205.

[0145] The information processing unit 72 can perform further data processing (step S207) based on the detected object mapped to the display space and the data such as the position, speed, size, and category of the detected object obtained through steps S203 to S206. For example, the information processing unit 72 can use the category of the detected object estimated in step S205 as identification information representing the type of the detected object, and use the reliability as the probability of belonging to each category, to perform matching. Figure 3 The first index calculation unit 54 performs similar processing. Furthermore, the information processing unit 72 can use the information such as the position and size of the detection object estimated in steps S203 and S204 as measurement information, and use the error covariance matrix to perform similar processing. Figure 3 The second indicator calculation unit 55 performs similar processing. Therefore, the information processing unit 72 can use the observation data from the millimeter-wave radar 71 to perform similar processing. Figure 10 The processing is similar to that shown. Furthermore, the information processing unit 72 can output data from the output unit 73 for processing in other devices (step S207).

[0146] As described above, when using millimeter-wave radar 71 as a sensor, sensing device 70 can also perform similar processing and achieve similar results as when using imaging device as a sensor. Figure 12 The sensing device 70 has a built-in millimeter-wave radar 71 and an information processing unit 72. However, the millimeter-wave radar and the information processing device with the function of the information processing unit 72 can also be set up independently.

[0147] The embodiments of the present invention have been described based on the accompanying drawings and examples, but it should be noted that those skilled in the art can easily make various modifications or variations based on the present invention. Therefore, it should be understood that these modifications or variations are included within the scope of the present invention. For example, the functions included in each component or step can be reconfigured in a logically consistent manner, and multiple components or steps can be combined into one or divided. The embodiments of the present invention have been described primarily in the context of a device, but the embodiments of the present invention can also be implemented as methods including steps performed by each component of the device. The embodiments of the present invention can be implemented as methods, programs executed by a processor included in the device, or storage media containing programs. It should be understood that these contents are also included within the scope of the present invention.

[0148] The term "mobile body" in this invention includes vehicles, ships, and aircraft. "Vehicles" in this invention include automobiles and industrial vehicles, but are not limited to these; they may also include railway vehicles, residential vehicles, and fixed-wing aircraft that travel on runways. Automobiles include, but are not limited to, cars, trucks, buses, two-wheeled vehicles, and trolleybuses, and may include other vehicles that travel on roads. Industrial vehicles include agricultural and construction vehicles. Industrial vehicles include, but are not limited to, forklifts and golf carts. Agricultural industrial vehicles include tractors, transplanters, balers, combine harvesters, and lawnmowers, but are not limited to these. Construction industrial vehicles include bulldozers, scrapers, excavators, cranes, dump trucks, and loading / unloading vehicles, but are not limited to these. Vehicles include human-powered vehicles. Furthermore, the classification of vehicles is not limited to the examples above. For example, automobiles may include industrial vehicles capable of traveling on roads, and the same vehicle may be included in multiple classifications. Ships in this invention include jet-powered seaplanes, boats, and oil tankers. Aircraft in this invention include fixed-wing aircraft and rotary-wing aircraft, etc.

[0149] Explanation of reference numerals in the attached figures:

[0150] 1: Image processing system (information processing system)

[0151] 10: Camera (Sensor)

[0152] 11: Camera optical system

[0153] 12: Control Department

[0154] 20: Image processing device (information processing device)

[0155] 21: Input Interface

[0156] 22: Control Unit (Processor)

[0157] 23: Output Interface

[0158] 24: Storage Department

[0159] 30: Determination device

[0160] 40: Subject (Object to be tested)

[0161] 41: Image Space

[0162] 42, 42a, 42b: Subject images

[0163] 43: Representative point

[0164] 44: Reference plane

[0165] 45: Point Mass

[0166] 46: Virtual Space

[0167] 51: Image Recognition Department

[0168] 52: Measurement Information Calculation Department

[0169] 53: Measurement Information Correction Department

[0170] 54: First Indicator Calculation Department

[0171] 55: Second Indicator Calculation Department

[0172] 56: Comprehensive Indicator Calculation Department

[0173] 60: Camera (Sensing Device)

[0174] 61: Camera optical system

[0175] 62: Camera components

[0176] 63: Control Department

[0177] 64: Output Interface

[0178] 70: Sensing device

[0179] 71: Millimeter-wave radar (sensor)

[0180] 72: Information Processing Department

[0181] 73: Output Department

[0182] 74: Signal Generation Unit

[0183] 75: High-frequency circuits

[0184] 76: Transmitting antenna

[0185] 77: Receiving antenna

[0186] 78: Signal Processing Department

[0187] 100: Vehicles (moving objects)

[0188] v1, v2: Attribution probability vectors

Claims

1. An information processing device, disposed on a moving body, for determining the reliability of detection results for detecting objects located around the moving body, wherein, have: The input interface acquires observation data obtained by the sensor from the observation space. The processor detects the object from the observation data and calculates multiple individual indicators, wherein each of the multiple individual indicators represents the reliability of information related to at least the measurement information in the identification information and measurement information of the object, and also calculates a comprehensive indicator after combining the calculated multiple individual indicators. as well as The output interface outputs a comprehensive index representing the reliability of the detection results. The processor calculates the position and error covariance matrix of the detected object on the two-dimensional plane based on the observation data, and calculates the individual indicators related to the position based on the error covariance matrix and the estimated range of the detected object's position. The measurement information is information related to the location.

2. The information processing apparatus according to claim 1, wherein, Let the intrinsic values ​​of the error covariance matrix be λ1 and λ2. The one-sided tolerance margins of the first direction of the two-dimensional plane, which is allowed as the estimated range, and the second direction intersecting the first direction are set as d1 and d2, respectively. The processor uses χ calculated by the following mathematical formula 2 , [Mathematical Expression 3] ， The individual indicators related to the location are calculated using the following mathematical formula. [Mathematical Expression 4] 。 3. The information processing apparatus according to claim 1 or 2, wherein, Each of the individual indicators and the comprehensive indicator has a value greater than 0 and less than 1.

4. The information processing apparatus according to claim 3, wherein, The multiple individual indicators are composed of n individual indicators a i The composition, where i is a natural number from 1 to n, is used by the processor to calculate the comprehensive index using the following mathematical formula: [Mathematical Expression 1] 。 5. The information processing apparatus according to claim 3, wherein, The multiple individual indicators are composed of n individual indicators a i The structure is formed by i being a natural number from 1 to n, when w i When assigning weights to each indicator, the processor calculates the comprehensive indicator using the following mathematical formula: [Mathematical Expression 2] 。 6. An information processing device, disposed on a moving body, for determining the reliability of detection results for detecting objects located around the moving body, wherein, have: The input interface acquires observation data obtained by the sensor from the observation space. The processor detects the object to be detected from the observation data and calculates multiple individual indicators, wherein the multiple individual indicators respectively represent the reliability of information containing at least the identification information in the identification information and measurement information of the object to be detected, and also calculates a comprehensive indicator after combining the calculated multiple individual indicators; as well as The output interface outputs a comprehensive index representing the reliability of the detection results. The identification information is determined from one of multiple categories, when a vector with the belonging probability of each category as its element is set as the belonging probability vector v. p When the number of categories is set to N, the processor calculates the individual indicators related to the identification information using the following mathematical formula: [Mathematical Expression 5] 。 7. The information processing apparatus according to claim 6, wherein, The identification information includes at least one of the type, color, and brightness of the detected object.

8. An information processing method, executed by a processor, for determining the reliability of a detection result for detecting an object located around a moving body, wherein, include: Acquire observational data obtained by sensors from the observation space; The object to be detected is detected from the observation data; Calculate multiple individual indicators, wherein each of the multiple individual indicators represents the reliability of information containing at least the measurement information from the identification information and measurement information of the detected object; It also calculates the composite index, which combines the calculated individual indicators; The output represents the comprehensive index indicating the reliability of the detection results. In the information processing method, the processor calculates the position and error covariance matrix of the detected object on the two-dimensional plane based on the observation data, and calculates the individual indicators related to the position based on the error covariance matrix and the estimated range of the detected object's position. The measurement information is information related to the location.

9. An information processing method, executed by a processor, for determining the reliability of a detection result for detecting an object located around a moving body, wherein, include: Acquire observational data obtained by sensors from the observation space; The object to be detected is detected from the observation data; Calculate multiple individual indicators, wherein each of the multiple individual indicators represents the reliability of information containing at least the identification information in the identification information and measurement information of the detected object; It also calculates the composite index, which combines the calculated individual indicators; The output represents the comprehensive index indicating the reliability of the detection results. In the information processing method, the identification information is determined from one of multiple categories. When a vector with the belonging probability of each category as its element is set as the belonging probability vector v... p When the number of categories is set to N, the processor calculates the individual indicators related to the identification information using the following mathematical formula: [Mathematical Expression 5] 。 10. A sensing device, wherein, have: The information processing apparatus as described in claim 1 or 6, and The sensor is configured to sense the observation space and acquire the observation data.

11. A mobile body, wherein, have: The information processing apparatus as described in claim 1 or 6.

12. An information processing system, wherein, have: The information processing apparatus as described in claim 1; Sensors acquire the observation data from the observation space; and The determination device, based on the comprehensive index, determines whether at least the measurement information based on the sensor can be used.

13. The information processing system according to claim 12, wherein, The information processing system includes multiple sensors and multiple information processing devices. The determining device determines which sensor's information should be preferentially used based on multiple comprehensive indicators calculated by multiple information processing devices.

14. An information processing system, wherein, have: The information processing apparatus as described in claim 6; Sensors acquire the observation data from the observation space; and The determination device, based on the comprehensive index, determines whether at least the identification information based on the sensor can be used.

15. The information processing system according to claim 14, wherein, The information processing system includes multiple sensors and multiple information processing devices. The determining device determines which sensor's information should be preferentially used based on multiple comprehensive indicators calculated by multiple information processing devices.

Images