Information processing device, information processing system, and information processing method

Abstract

To provide an information processing device for supporting smooth driving by discriminating a person between an occupant of a vehicle and a person other than the occupant.SOLUTION: An information processing device comprises: an image acquisition section 201 for acquiring an image from an imaging section 100; an object detection section 202 for detecting an object from the image so as to form a detection area to enclose the object; an object analysis section 203 for analyzing the kind of the object within the detection area; an occupant determination section 204 for determining whether a person is an occupant of a vehicle when it is analyzed that the kind of the object includes the vehicle and a person; and an information processing section 205 for processing the person as vehicle information of the vehicle when the occupant determination section determines that the person is the occupant of the vehicle.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing system, and an information processing method. [Background technology]

[0002] Technologies have been developed that use cameras mounted on vehicles, cameras installed on the roadside, etc. to detect objects such as people walking on sidewalks and vehicles traveling on roads, and use these detection results to assist vehicle driving. In this technology, image information from the camera is processed to estimate the type, position, shape, etc. of objects present on sidewalks, roads, etc., and collect information about the vehicle's surroundings. As a method for estimating the type, position, shape, etc. of objects, a method using a neural network has been proposed (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2023 / 190081 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with conventional methods, when a vehicle in an image and its occupants such as the driver and passengers are identified as objects using bounding boxes, it is not possible to distinguish between the occupants and people working outside the vehicle or people walking on the sidewalk, etc., and there is a problem in that this can hinder smooth driving assistance, such as by recognizing the occupants as an obstacle when the vehicle is moving and causing the vehicle to stop.

[0005] The present disclosure has been made to solve the above-mentioned problems, and aims to provide an information processing device that can distinguish between vehicle occupants and non-occupants and support smooth driving, as well as an information processing system and an information processing method. [Means for solving the problem]

[0006] The information processing device according to the present disclosure includes an image acquisition unit that acquires an image from an imaging unit, an object detection unit that detects an object from the image and forms a detection area surrounding the object, an object analysis unit that analyzes the type of object within the detection area, a passenger determination unit that determines whether the person is a passenger in the vehicle when the analysis finds that the type of object includes a vehicle and a person, and an information processing unit that processes the person as vehicle information of the vehicle when the passenger determination unit determines that the person is a passenger in the vehicle.

[0007] The information processing system according to the present disclosure also includes an imaging unit that captures images of the surroundings of the autonomously driven vehicle, and an information processing device according to the present disclosure that acquires images from the imaging unit and processes vehicle information about the surroundings of the autonomously driven vehicle.

[0008] The information processing method according to the present disclosure also includes the steps of acquiring an image of the surroundings of an autonomous vehicle, detecting an object from the image and forming a detection area surrounding at least a portion of the object, analyzing the type of object within the detection area, determining whether the person is a passenger in the vehicle if the analysis finds that the type of object includes a vehicle and a person, and processing the person as vehicle information of the vehicle if the person is determined to be a passenger in the vehicle. [Effects of the Invention]

[0009] According to the present disclosure, smooth driving can be supported by determining whether a detected person is a vehicle occupant and processing the object determined to be a occupant as vehicle information. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a schematic block diagram illustrating an example of an information processing system according to a first embodiment. [Figure 2] FIG. 2 is an explanatory diagram showing an example of acquiring an image of a vehicle traveling on a road according to the first embodiment. [Figure 3] FIG. 2 is an explanatory diagram showing an example of analyzing an image of a vehicle traveling on a road according to the first embodiment. [Figure 4] FIG. 2 is an explanatory diagram showing an example of analyzing the position of a vehicle traveling on a road according to the first embodiment. [Figure 5] FIG. 2 is an explanatory diagram showing an example of analyzing vehicles and people on an image according to the first embodiment. [Figure 6] FIG. 2 is an explanatory diagram showing an example of analyzing vehicles and people on an image according to the first embodiment. [Figure 7] FIG. 4 is an explanatory diagram showing an example of a code for setting a parameter according to the first embodiment. [Figure 8] FIG. 4 is an explanatory diagram showing an example of analyzing the overlap ratio of detection regions of a vehicle and a person according to the first embodiment. [Figure 9] FIG. 2 is an explanatory diagram showing an example of analyzing vehicles and people on an image according to the first embodiment. [Figure 10] FIG. 2 is an explanatory diagram showing an example of analyzing vehicles and people on an image according to the first embodiment. [Figure 11] 4 is an explanatory diagram showing an example of analyzing the relationship between a detection area of ​​a person and a vehicle according to the first embodiment. FIG. [Figure 12] FIG. 4 is an explanatory diagram showing an example of a code for setting a parameter according to the first embodiment. [Figure 13] FIG. 2 is an explanatory diagram showing an example of analyzing vehicles and people on an image according to the first embodiment. [Figure 14] FIG. 2 is an explanatory diagram showing an example of analyzing vehicles and people on an image according to the first embodiment. [Figure 15] FIG. 4 is an explanatory diagram showing an example of a code for setting a parameter according to the first embodiment. [Figure 16] FIG. 4 is an explanatory diagram showing an example of analyzing the overlap ratio of detection regions of a vehicle and a person according to the first embodiment. [Figure 17] FIG. 2 is an explanatory diagram showing an example of analyzing vehicles and people on an image according to the first embodiment. [Figure 18] FIG. 2 is an explanatory diagram showing an example of analyzing vehicles and people on an image according to the first embodiment. [Figure 19] 10 is a flowchart showing a processing routine executed by an information processing device according to a second embodiment. [Figure 20] FIG. 10 is a schematic block diagram showing an example of a processing circuit that realizes each function of the information processing device according to the second embodiment. [Figure 21] FIG. 11 is a schematic block diagram illustrating an example of an information processing system according to a third embodiment. [Figure 22] FIG. 11 is an explanatory diagram showing an example of analyzing vehicles and people on an image according to the third embodiment. [Figure 23] FIG. 11 is an explanatory diagram showing an example of analyzing vehicles and people on an image according to the third embodiment. [Figure 24] FIG. 11 is an explanatory diagram showing an example of analyzing vehicles and people on an image according to the third embodiment. [Figure 25] FIG. 10 is a schematic block diagram illustrating an example of an information processing system according to a fourth embodiment. [Figure 26] FIG. 13 is an explanatory diagram showing an example of analyzing vehicles and people on an image according to the fourth embodiment. [Figure 27] FIG. 13 is an explanatory diagram showing an example of a code for setting a parameter according to the fourth embodiment. [Figure 28] FIG. 13 is an explanatory diagram showing an example of analyzing vehicles and people on an image according to the fourth embodiment. [Figure 29] FIG. 13 is an explanatory diagram showing an example of analyzing vehicles and people on an image according to the fourth embodiment. [Figure 30] FIG. 11 is a schematic block diagram illustrating an example of an information processing system according to a fifth embodiment. [Figure 31] FIG. 13 is an explanatory diagram showing an example of analyzing a person on an image according to the fifth embodiment. [Figure 32] FIG. 13 is an explanatory diagram showing an example of analyzing a person on an image according to the fifth embodiment. [Figure 33] FIG. 13 is an explanatory diagram showing an example of a code for setting a vehicle type according to the fifth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] The embodiments will be described with reference to the drawings, in which the same contents and corresponding parts are designated by the same reference numerals and detailed description thereof will be omitted.

[0012] Embodiment 1 Fig. 1 is a schematic block diagram showing an example of an information processing system 10 according to the first embodiment. The information processing system 10 captures an image within an imaging range 1 from an imaging unit 100 configured with a camera or the like installed on a structure 21 around a road 2 shown in Fig. 2, for example, into an information processing device 200. The information processing device 200 includes an image acquisition unit 201 that acquires an image from the imaging unit 100, and an object detection unit 202 that detects an object from the image and forms a detection area B using a bounding box that surrounds the detected object. The image acquired by the image acquisition unit 201 may be a still image or a video.

[0013] The object analysis unit 203 analyzes the type of object within the detection area B. When the object analysis unit 203 analyzes that the type of object includes a vehicle VE and a person P, the occupant determination unit 204 determines whether or not the person P is an occupant of the vehicle VE. When the occupant determination unit 204 determines that the person P is an occupant of the vehicle VE, the person P is processed by the information processing unit 205 as vehicle information linked to the vehicle VE. A process may be performed to remove the person P determined to be an occupant. An occupant is a person who is in the vehicle VE, including the driver and passengers of the vehicle VE. Object information is used for analyzing the object and is stored in the object information storage unit 203a. In addition, various parameter information described below is used to determine whether or not the person P is an occupant of the vehicle VE and is stored in the parameter setting unit 204a. Object analysis, occupant determination, and information processing are performed based on this object information and parameters. This information may be input from a user interface.

[0014] The imaging unit 100 captures images within the imaging range 1 using a camera or the like. The camera may be, for example, a CCD (Charge Coupled Device) camera that converts the image into an electrical signal, extracts it, and forms pixels. It may also be a video camera that arranges the electrical signals in time series to generate a video signal. It may be a monocular camera with a single lens, a stereo camera, or a camera system that combines multiple cameras. A network camera that communicates over the Internet or an IP (Internet Protocol) camera may also be used. Furthermore, the structure 21 on which the imaging unit 100 is installed may be, for example, a utility pole, a street light, a traffic light, a building wall, etc., but is not particularly limited thereto. It may be located in a position where the imaging unit 100 can secure the imaging range 1. It may be mounted on the autonomous vehicle AD, which is the subject vehicle.

[0015] The image acquisition unit 201 acquires images captured by the imaging unit 100 and outputs them to the object detection unit 202. The object detection unit 202 performs processes such as scaling and normalization using an object detection algorithm, an object detection model, etc., to detect objects. Images are captured at intervals of, for example, several fps to 30 fps, and transmitted to the image acquisition unit 201 of the information processing device 200 via wireless communication. The images may also be transmitted via wired communication using a USB (Universal Serial Bus) / LAN (Local Area Network) cable or the like.

[0016] An example of image acquisition by the image acquisition unit 201 will be described. For example, as shown in Fig. 2, a roadside camera is installed as the imaging unit 100 at a height of, for example, about 6 m above the road 2, and an imaging range 1 to be detected is secured by looking down on the object from diagonally above. The imaging range 1 is, for example, a radial area including the road 2, and the objects to be detected are a vehicle VE, a person P, etc. that need to be grasped in order to perform autonomous driving. Fig. 2 shows an example in which a vehicle VE traveling on the road 2 and an autonomously driving vehicle AD performing autonomous driving are captured at a certain time.

[0017] This section explains the detection of a vehicle VE to support the driving of an autonomous driving vehicle AD. The object detection unit 202 outputs a rectangular detection area B that inscribes and surrounds the object in the image acquired by the image acquisition unit 201. Then, the object detection unit 202 recognizes the image pattern in the detection area B using neural network technology or the like, and estimates the type of the object. 3 is an explanatory diagram showing an example of analyzing an image of a vehicle VE traveling on a road 2. The diagram shows an example in which the object detection unit 202 detects an object in an image acquired by a roadside camera serving as the imaging unit 100. The object surrounded by the detection area Bv is the vehicle VE, and the object surrounded by the detection area Bp is the person P. In FIG. 3, the image coordinates are shown in pixel (pix) units, with the upper left corner of the image as the origin, the right direction as the positive direction of the x-axis, and the downward direction as the positive direction of the y-axis.

[0018] Figure 4 shows the positions of the detected vehicle VE, person P, and autonomous vehicle AD. The object's position coordinates can be calculated by assuming that the detected object exists on the ground and superimposing any position coordinate, such as the center of the bottom edge of detection area B, with the external parameters of the roadside camera and converting it to world coordinates (coordinates in the real world). Images can also be superimposed and converted to world coordinates using a homography matrix, which represents the projective transformation between two planes. World coordinates are defined as the x-axis representing longitude, the y-axis representing latitude, and the z-axis representing height. The world coordinates are expressed in meters, with the origin being an arbitrary position, the positive direction of the x-axis representing east, the positive direction of the y-axis representing north, and the positive direction of the z-axis representing height. Because person P, who is riding in vehicle VE shown in Figure 3, is detected at a position far from the ground, it is estimated to be located at a different position from vehicle VE in the world coordinates shown in Figure 4. In this case, autonomous vehicle AD may determine person P, who is riding in vehicle VE, as an obstacle and stop, which would disrupt smooth driving.

[0019] Therefore, the object analysis unit 203 further analyzes object information such as the size, position, and overlap ratio of the detection area B for the object detected by the object detection unit 202. The object information analysis in the object analysis unit 203 includes, for example, analyzing the type of vehicle VE and the overlap ratio of the detection area B of the vehicle VE and the person P.

[0020] For example, if the vehicle VE is a passenger car, the occupant is usually located inside the vehicle VE, and therefore the detection area Bp of person P is completely contained within the detection area Bv of the vehicle VE. Therefore, when the overlap ratio R1 = 1.0, person P can be estimated as an occupant of the vehicle VE. For example, if the vehicle VE is a light truck, it is also possible that person P is present in the bed of the light truck, as shown in Figure 5. Furthermore, if the vehicle VE is a special vehicle such as a reach lift car, it is also possible that the detection area Bp of person P is not completely contained within the detection area Bv of the vehicle VE, as shown in Figure 6. Therefore, it is preferable that the object analysis unit 203 analyzes the vehicle type of the vehicle VE and sets an overlap ratio R1 for each vehicle type. The analysis of the vehicle type of the vehicle VE may be performed by the object detection unit 202, and this data may be used by the object analysis unit 203. The object information analyzed here is stored, for example, in the object information storage unit 203a.

[0021] Then, the occupant determination unit 204 calls the object information analyzed by the object analysis unit 203 from the object analysis unit 203 or from the object information storage unit 203a, and determines whether or not the person P is an occupant of the vehicle VE. Fig. 7 is an explanatory diagram showing an example of code for setting parameters in the parameter setting unit 204a, and Fig. 8 is an explanatory diagram showing an example of analyzing the overlap ratio R1 between the detection area B of the vehicle VE and the detection area B of the person P. As described above, the degree of overlap between the detection area Bp of the person P and the detection area Bv of the vehicle VE differs depending on the vehicle type, so the overlap ratio threshold R1 is set for each vehicle type of the vehicle VE. For example, in Fig. 7, if the vehicle VE is a passenger car "car" or a truck "truck", the overlap ratio threshold R1 is set to 1.0, and if the vehicle VE is a light truck "light_truck" or a lift car "lift_car", R1 is set to 0.9. The occupant determination unit 204 determines that person P is an occupant of the vehicle VE if the overlap ratio, which is the ratio of the overlapping portion of person P's detection area Bp and the detection area Bv of the vehicle VE to person P's detection area Bp, is equal to or greater than the overlap ratio threshold R1 set for each vehicle model of the vehicle VE.

[0022] For example, the area S "overlap_s" of the overlap between the detection area Bv of the vehicle VE and the detection area Bp of the person P shown in Fig. 8 is calculated. Then, as shown in the following formula (1), if the area S is equal to or greater than the value obtained by multiplying the area Ap of the detection area Bp by the overlap ratio threshold R1, the person P is determined to be an occupant of the vehicle VE.

[0023]

number

[0024] When the occupant determination unit 204 determines that the person P is an occupant of the vehicle VE, the information processing unit 205 processes the person P as vehicle information of the vehicle VE. Since the information of the occupant does not become an obstacle to the traveling of the vehicle VE, the information of the person P may be removed.

[0025] As described above, the system includes the image acquisition unit 201 that acquires an image from the imaging unit 100, the object detection unit 202 that detects an object from the image and forms a detection area B surrounding the object, the object analysis unit 203 that analyzes the type of object within the detection area B, the occupant determination unit 204 that determines whether the person P is an occupant of the vehicle VE when the analysis finds that the object types include the vehicle VE and the person P, and the information processing unit 205 that processes the person P as vehicle information for the vehicle VE when the occupant determination unit 204 determines that the person P is an occupant of the vehicle VE. This makes it possible to link the information about the occupant, person P, to the vehicle VE and use it as vehicle information, or to remove the information about the occupant, person P. Therefore, it is possible to prevent the occupant, person P, from being recognized as an obstacle and causing the autonomous vehicle AD to stop. Furthermore, when the occupant determination unit 204 determines that the person P is not an occupant of the vehicle VE, it is possible to assist driving by allowing the autonomous vehicle AD to avoid the person P and continue driving. In other words, it is possible to distinguish between an occupant of the vehicle VE and an object that is not an occupant, thereby assisting smooth driving.

[0026] Furthermore, the occupant determination unit 204 determines that person P is an occupant of the vehicle VE if the overlap ratio, which is the ratio of the overlapping portion between the detection area Bp of person P and the detection area Bv of the vehicle VE to the detection area Bp of person P, is equal to or greater than the overlap ratio threshold R1 set for each vehicle type of vehicle VE, and therefore can make a determination based on the vehicle type and can also make a determination for special vehicles VE such as lift cars. Furthermore, when the detection area B formed by the object detection unit 202 is a two-dimensional area, accurate determination can be achieved by determining the numerical value of the area S where the two-dimensional areas formed for the person P and the vehicle VE overlap.

[0027] Although the example has been described with a single vehicle VE, information processing is also possible when multiple vehicles are detected. FIG. 9 shows an example in which a forklift (vehicle VE2) approaches the bed of a truck (vehicle VE1) and loads or unloads cargo from the bed of the truck. In this case, the detection area Bp of person P is contained within the detection area Bv2 of vehicle VE2, i.e., the forklift, and since the above formula (1) is satisfied in relation to the forklift, person P is processed as an occupant of vehicle VE2. Furthermore, since the detection area Bv2 of vehicle VE2 is contained within the detection area Bv1 of the truck (vehicle VE1), vehicle VE2 can be linked to vehicle VE1. When the forklift moves away from the truck, the overlapping area S of the detection area Bv1 and the detection area Bv2 becomes smaller, so similarly, vehicle VE1 and vehicle VE2 can be processed without being linked from formula (1). In other words, person P is treated as an occupant of vehicle VE2, and the detection area Bp is removed, and the autonomously driven vehicle AD is controlled to travel while avoiding vehicles VE1 and VE2 as separate objects. Even in such a case, it is possible to prevent the occupant of vehicle VE2 from being recognized as an obstacle unrelated to vehicle VE2 and the driving of automatically driven vehicle AD from being controlled.

[0028] Furthermore, although an example in which there is only one person P has been described, information processing is also possible when multiple people are detected. FIG. 10 shows an example in which two people P1 and P2 are riding in the bed of a truck, which is a vehicle VE. In this case, the detection area Bp1 of person P1 is included in the detection area Bv of the vehicle VE, and since this satisfies the above formula (1), person P1 is processed as an occupant of the vehicle VE. Similarly, the detection area Bp2 of person P2 is included in the detection area Bv of the vehicle VE, and since this satisfies the above formula (1), person P2 is also processed as an occupant of the vehicle VE.

[0029] Furthermore, it may be possible to determine whether or not the person P is a passenger in the vehicle VE using the positional relationship between the detection area Bp of the person P and the detection area Bv of the vehicle VE as a parameter. For example, as shown in FIG. 11, when the distance from the bottom edge Bvu of the detection area Bv of the vehicle VE to the bottom edge Bpu of the detection area Bp of the person P is expressed as R2×Bvh using the height Bvh of the detection area Bv of the vehicle VE, the following equation (2) is obtained:

[0030]

number

[0031] If the above condition is satisfied, person P is determined to be a passenger. In FIG. 11, the upward direction of the y2 axis is positive, and the height ratio threshold R2 is a value set for each vehicle type. For example, as shown in FIG. 12, R2 is set to 0.3 for passenger cars and trucks, R2 is set to 0.4 for light trucks, and R2 is set to 0.2 for lift cars.

[0032] 13, when a person P is detected passing in front of a truck, the distance between the bottom edge Bpu of the detection area Bp of the person P and the bottom edge Bvu of the detection area Bv of the vehicle VE is smaller than the height Bvh of the detection area Bv of the vehicle VE multiplied by the height ratio threshold R2 determined for each vehicle type, and therefore does not satisfy equation (2). Therefore, it is determined that the person P is not a passenger in the vehicle VE.

[0033] In this way, when the object analysis unit 203 analyzes that the object types include a vehicle VE and a person P, and the overlap ratio is equal to or greater than the overlap ratio threshold set for each type of vehicle VE, the positional relationship between the detection area Bv of the vehicle VE and the detection area Bp of the person P can be analyzed to take into account the position where the person P is located, for example, the relationship with the road 2, etc. For example, if the distance between the lower end Bpu of person P's detection area Bp and the lower end Bvu of vehicle VE's detection area Bv is smaller than the height Bvh of vehicle VE multiplied by the height ratio threshold R2 set for each vehicle type, person P is determined to be not a passenger in vehicle VE, thereby taking into account the position of person P and enabling an accurate determination.

[0034] In addition, an example has been described in which the detection area Bv of the vehicle VE and the detection area Bp of the person P are two-dimensional areas, the area S where the detection area Bv of the vehicle VE and the detection area Bp of the person P overlap is calculated, and the area S is used to determine whether or not the person is a passenger.However, a similar determination may also be made based on the volume V using the detection areas B3v and B3p which are three-dimensional areas as shown in Figure 14.

[0035] When the detection area B is a three-dimensional area, the object detection unit 202 is capable of detecting objects in three dimensions. The object detection unit 202 forms a three-dimensional detection area B3 that inscribes and surrounds the object in the image acquired by the image acquisition unit 201. The object analysis unit 203 then recognizes the image pattern in the three-dimensional detection area B3 from the three-dimensional image using neural network technology or the like, and estimates the type of object. Fig. 15 is an explanatory diagram showing an example of code for setting parameters in the parameter setting unit 204a, and Fig. 16 is an explanatory diagram showing an example of analyzing the overlap of the three-dimensional detection area B3. In Fig. 15, for example, if the vehicle VE is a passenger car "car" or a truck "truck," the overlap ratio threshold R3 is set to 1.0, and if the vehicle VE is a light truck "light_truck" or a lift car "lift_car," R3 is set to 0.9. Then, the volume V "overlap_v" of the overlap between the three-dimensional detection area B3v of the vehicle VE and the three-dimensional detection area B3p of the person P is calculated. Then, as shown in Fig. 16 and the following equation (3), if the volume V is equal to or greater than the value obtained by multiplying the volume A3p of the three-dimensional detection area B3p by the overlap ratio threshold R3, the person P is determined to be an occupant of the vehicle VE.

[0036]

number

[0037] For example, in the case of a passenger on the bed of a light truck shown in Figure 17, the three-dimensional detection area B3p of the passenger, person P, may not be completely contained within the three-dimensional detection area B3v of the vehicle VE, so accurate determination can be made by setting an overlap ratio threshold R3 for each vehicle type. Also, as shown in Figure 18, even when the three-dimensional detection area B3p of person P who is not a passenger in the vehicle VE overlaps with the three-dimensional detection area B3v of the vehicle VE, determination is made easier than with a two-dimensional detection area. For example, if the overlap ratio threshold for a cargo truck is set to R3 = 1, equation (3) is not satisfied, and person P is determined not to be a passenger in the vehicle VE.

[0038] Also, similar to the two-dimensional detection area, if the distance between the lower end B3pu of person P's three-dimensional detection area B3p and the lower end B3vu of vehicle VE's three-dimensional detection area B3v is smaller than the height B3vh of vehicle VE multiplied by a height ratio threshold R4 determined for each vehicle type, person P may be determined not to be an occupant of vehicle VE.

[0039] This can further improve the accuracy of the judgment, and by linking the information of the occupant P to the vehicle VE and using it as vehicle information, or by removing the information of the occupant P, it is possible to reduce false detections of the occupant as an obstacle and prevent unnecessary stops of the autonomous vehicle AD.

[0040] Embodiment 2 The operation of the information processing device 200 will be described. 19 is a flowchart showing a processing routine executed by information processing device 200 according to the second embodiment. Information processing device 200 first acquires an image of the periphery of autonomously driven vehicle AD (step S101). Then, it detects an object from the acquired image and forms a detection area B surrounding the object (step S102). Next, it analyzes the type of object within detection area B (step S103). If the analysis finds that the object types include vehicle VE and person P (YES in step S104), it determines whether person P is an occupant of vehicle VE (step S105). If it is determined that person P is an occupant (YES in step S105), it processes person P as vehicle information of vehicle VE (step S106). If the analysis does not find that the object types include vehicle VE and person P (NO in step S104), it returns to step S101. If person P is not determined to be an occupant (NO in step S105), it processes person P as an obstacle in the traveling of autonomously driven vehicle AD, and returns to step S101. These processes are repeated at an arbitrary cycle of, for example, about 10 Hz. These operations are executed in accordance with the example explained in detail in the first embodiment.

[0041] Here, each function of the information processing device 200 is realized by a processing circuit. Fig. 20 is a schematic configuration diagram showing an example of a processing circuit that realizes each function of the information processing device 200. The information processing device 200 has a processor 80, a storage device 81, a communication I / F (interface) 82, a CAN (Controller Area Network) I / F (interface) 83, etc. For example, a CPU (Central Processing Unit) is used as the processor 80. The storage device 81 transmits and receives data to and from the processor 80 and stores the data. An image captured by the imaging unit 100 is acquired by the image acquisition unit 201 via the communication I / F 82. The calculations and determinations of the image acquisition unit 201, object detection unit 202, object analysis unit 203, occupant determination unit 204, and information processing unit 205 are executed by the processor 80. Object information, parameters, calculation formulas, etc. are stored in the storage device 81. Commands from the information processing device 200 to the autonomously driven vehicle AD are transmitted via the CANI / F 83 to a control unit, etc. that controls the operation of the autonomously driven vehicle AD.

[0042] The processor 80 and the storage device 81 may be one shared device or multiple devices may be used. The processor 80 may also be equipped with, for example, a logic circuit using an ASIC (Application Specific Integrated Circuit), an IC (Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or various signal processing circuits. Multiple processors 80, either of the same type or of different types, may be provided, so that each process is shared and executed by multiple arithmetic processing devices.

[0043] The multiple storage devices 81 may include, for example, RAM (Random Access Memory) configured to allow data to be read and written from the processor 80, ROM (Read Only Memory) configured to allow data to be read from the processor 80, a hard disk (HDD), etc.

[0044] Each function of the information processing device 200 is realized by the processor 80 executing software or a program stored in the storage device 81 and working in cooperation with the hardware. Setting data to be set in the information processing device 200 may be stored in the storage device 81 as part of the software or program, or may be input by the user. A non-transitory recording medium 85 on which an information processing program 84 is recorded may be distributed and installed in the storage device 81 of the information processing device 200.

[0045] In this way, an image of the area around the autonomous vehicle AD is acquired, an object is detected from the image, a detection area surrounding the detected object is formed, and the type of object within the detection area is analyzed. If the analysis finds that the object type includes a vehicle and a person P, it is determined whether the person P is a vehicle occupant. If the person P is determined to be a occupant of the vehicle VE, the person P is processed as vehicle information for the vehicle VE, thereby suppressing erroneous detection of the occupant as an obstacle and preventing unnecessary stops of the autonomous vehicle AD. Furthermore, if it is determined that the person P is not a occupant of the vehicle VE, driving assistance can be provided to allow the autonomous vehicle AD to avoid the person P. Therefore, smooth driving can be supported by distinguishing between occupants of the vehicle VE and objects that are not occupants.

[0046] Embodiment 3 21 is a schematic block diagram showing an example of an information processing system 10 according to embodiment 3. When the object analysis unit 203 analyzes that the object types include a vehicle VE and a person P, the information processing device 200 identifies a person-specific portion PTp corresponding to the person P below the detection area Bp of the person P, and determines whether or not the person P is an occupant of the vehicle VE based on the type of object present in an adjacent portion PTa adjacent to the person-specific portion PTp. The other configurations are the same as those of the information processing device 200 according to embodiment 1.

[0047] 22, when the object analysis unit 203 analyzes an image acquired by the image acquisition unit 201 to include a vehicle VE and a person P as object types, it extracts a vehicle area RGv where the vehicle VE exists corresponding to the detection area Bv of the vehicle VE, and a person area RGp where the person P exists corresponding to the detection area Bp of the person P. The image information of the vehicle area RGv and the person area RGp includes image information of surrounding objects 22 such as a road 2, a sidewalk, and a building, which are objects present in the vicinity. Furthermore, in the image information obtained by integrating the vehicle area RGv and the person area RGp shown in Fig. 23, a person-specific portion PTp corresponding to the person P is identified below the detection area Bp of the person P, and the type of object present in the adjacent portion PTa adjacent to the person-specific portion PTp is analyzed. In Fig. 23, the type of the adjacent portion PTa is the vehicle VE, and therefore the occupant determination unit 204 determines that the person P is an occupant of the vehicle VE.

[0048] Here, detailed analysis of the person-specific region PTp and adjacent region PTa may be similar to the type analysis within the detection region B in embodiment 1, but for example, the foreground and background of the extracted vehicle region RGv, person region RGp, identified person-specific region PTp, adjacent region PTa, etc. may be separated and the type, such as person P (person) or vehicle VE (vehicle), may be identified for the pixels representing each region based on information obtained by subtracting the background. The image may be divided by segmentation, and the type to which each pixel belongs may be identified. If multiple types of pixels exist within an area, the type may be determined by the most frequent pixel value.

[0049] FIG. 24 shows an example of capturing a person region RGp in which a person P who is not a passenger in the vehicle VE exists. The object analysis unit 203 identifies a region in which the pixels at the bottom of the person region RGp identified by the region identification unit 206 are classified as person P (person) as a person-identified region PTp. The object analysis unit 203 then identifies an adjacent region PTa adjacent to the person-identified region PTp and classifies the pixels of the adjacent region PTa. In an enlarged portion of the person-identified region PTp and the adjacent region PTa shown in FIG. 24, the pixels of the adjacent region PTa are classified as road 2. Therefore, it is determined that the person P is not a passenger in the vehicle VE.

[0050] Here, we have described an example in which the part identification unit 206 identifies the person-specific part PTp and the adjacent part PTa from image information that integrates the person area RGp and the vehicle area RGv. However, if the adjacent part PTa is included within the person area RGp, the person-specific part PTp and the adjacent part PTa may be identified from only the person area RGp without integrating the person area RGp and the vehicle area RGv. Immediate analysis can be performed by storing pixel information such as vehicle, person, road, and sidewalk that represent classes of vehicles VE, people P, roads 2, and sidewalks that may be present in the person area RGp in the object information storage unit 203a. Other types, such as animals that appear on the road 2, and construction items such as road cones, may also be stored. The person-specific portion PTp is preferably specified from the bottom of the person area RGp, but may be specified at a position determined within the vehicle area RGv for each vehicle type.

[0051] In this way, when the object analysis unit 203 analyzes that the object types include a vehicle VE and a person P, it identifies a person-specific portion PTp corresponding to the person P below the detection area Bp of the person P, and when the type of object present in the adjacent portion PTa adjacent to the person-specific portion PTp is a vehicle VE, the occupant determination unit 204 determines that the person P is an occupant of the vehicle VE, making it possible to determine whether or not the person P is an occupant of the vehicle VE without analyzing the positional relationship between the vehicle VE and the person P. The accuracy of the determination can be improved, and by linking the information of the occupant person P to the vehicle VE as vehicle information or removing the information of the occupant person P, it is possible to suppress erroneous detection of the person P as an obstacle and suppress unnecessary stops of the autonomously driven vehicle AD.

[0052] Embodiment 4 25 is a schematic block diagram showing an example of an information processing system 10 according to embodiment 4. The information processing device 200 includes an object tracking unit 207 that tracks an object detected by an object detection unit 202, and determines whether or not a person P is a passenger in a vehicle VE using a tracking image 207a acquired from the object tracking unit 207. The other configurations are the same as those of the information processing device 200 according to embodiment 1.

[0053] The object tracking unit 207 tracks the object detected by the object detection unit 202 using a technique such as ByteTrack or DeepSort. For example, as shown in FIG. 26, the object tracking unit 207 assigns "person" as the tracking ID for the person P and "vehicle" as the tracking ID for the vehicle VE to the object detected by the object detection unit 202, and analyzes the tracked frames. If there are multiple persons P and multiple vehicles VE, the tracking IDs may be "person1", "person2", ..., "vehicle1", "vehicle2", ..., etc. For example, using the first frame and the nth frame (n is a natural number) image that is the tracking image 207a shown in FIG. 26, the movement of the detected object is understood from the position of the detection area B, the speed at which the frames progress, etc.

[0054] For example, when the person P and the vehicle VE are moving, if the speed difference between the person P and the vehicle VE is within the speed difference threshold, the occupant determination unit 204 determines that the person P is an occupant of the vehicle VE. If the speed of the person P is equal to or greater than the person speed threshold, the person P may be determined to be an occupant of the vehicle VE. The difference in the stopping time between the person P and the vehicle VE may also be used for the determination. Each setting parameter may be stored in the parameter setting unit 204a.

[0055] FIG. 27 shows an example of code specifying setting parameters. "range_vd:2.0" represents a command to set the speed difference between person P, who is presumed to be an occupant, and the vehicle VE as a parameter, assuming that person P and the target vehicle VE are moving; if the speed difference is within ±2 km / h, person P is determined to be an occupant of the vehicle VE. Therefore, the speed difference threshold is set to 2 km / h. For example, in the example shown in FIG. 26, if the speed of the vehicle VE analyzed by the object analysis unit 203 is 40 km / h and the speed of person P is 39 km / h, person P is determined to be an occupant of the vehicle VE based on "range_vd:2.0." "velocity_p:15.0" is a command to determine that person P is a passenger if their speed is 15 km / h or faster. Taking into account the limit speed at which person P can run and the environment, the person speed threshold is set to a realistic speed that person P cannot achieve. If both "range_vd" and "velocity_p" are satisfied, person P may be determined to be a passenger.

[0056] "range_vs:2.0" represents a command to determine that person P is a passenger if the vehicle VE's speed is within 2 km / h, assuming that person P is stopped and the target vehicle VE is moving. "range_vs:1.0" may be used to define a speed of less than 1 km / h as stopped. For example, in the example shown in FIG. 28, where the vehicle VE passes by a stopped person P at 15 km / h, "range_vs:2.0" determines that person P is not a passenger. Also, as shown in Figure 29, when the vehicle VE is stopped and the person P is moving, the tracking image 207a can be used to analyze the speed difference between the vehicle VE and the person P, and it can be determined that the person P is not a passenger in the vehicle VE.

[0057] "time:30" is a command to set a threshold for the stopping time [s], assuming that the person P and the target vehicle VE will stop. If the stopping time difference between the person P and the vehicle VE is less than the stopping time difference threshold, it is determined that the person P is a passenger in the vehicle VE.

[0058] In this way, the system is provided with an object tracking unit 207 that tracks objects detected by the object detection unit 202, and when the object analysis unit 203 analyzes that the object types include a vehicle VE and a person P, the occupant determination unit 204 can improve the accuracy of the determination by using the tracking image obtained from the object tracking unit 207 to determine whether or not the person P is an occupant of the vehicle VE.

[0059] Furthermore, the occupant determination unit 204 can determine that person P is an occupant of the vehicle VE if the speed difference between person P and vehicle VE calculated from the tracking image is equal to or less than the speed difference threshold, and can determine that person P is an occupant of vehicle VE if the stopping time difference between person P and vehicle VE calculated from the tracking image is equal to or less than the stopping time difference threshold. Furthermore, by combining a plurality of determination processes, it is possible to improve the accuracy of the determination.

[0060] Embodiment 5. 30 is a schematic block diagram showing an example of an information processing system 10 according to embodiment 5. The information processing device 200 includes a posture estimation unit 208 that estimates the posture of person P when the type of the object is analyzed as a person. The other configurations are the same as those of the information processing device 200 according to embodiment 1.

[0061] The posture estimation unit 208 estimates the posture of the person P detected by the object detection unit 202 using technology such as a neural network. For example, it outputs the posture status of the person P, such as "standing" for a standing person, "sitting" for a sitting person, or "unknown" if the posture is unknown. For example, in the example shown in FIG. 31, "standing" is output, and in the example shown in FIG. 32, "sitting" is output.

[0062] The object information storage unit 203a referenced by the object analysis unit 203 stores information about the posture of the person P. For example, if a forklift, golf cart, bicycle, motorcycle, etc. is considered to be a vehicle VE, occupants such as a driver and passengers are usually detected in a sitting posture. Therefore, the parameters shown in FIG. 33 are prepared. "pose_estimation_class:[bike, bicycle, forklift, golfcart, ...]" sets the vehicle type in which the passenger is expected to be sitting. When the object analysis unit 203 analyzes that the object types include a vehicle VE and a person P, the posture estimation unit 208 checks whether the vehicle type of the vehicle VE and the posture of the person P are reasonable. When the posture status of the person P is "sitting," if the vehicle type matches "sitting" in "pose_estimation_class:," the person P is estimated to be a passenger of the vehicle VE. If the posture status of person P is "standing," it may be confirmed that the vehicle is a special vehicle such as a reach lift car, or person P may not be determined to be a passenger, but may be processed by other determination means.

[0063] In this way, the accuracy of the determination can be improved by providing a posture estimation unit 208 that estimates the posture of person P when the type of object is analyzed as person P, estimating the type of vehicle VE based on the estimated posture, and using the object analysis unit 203 to confirm whether the type of vehicle and the posture of person P are reasonable and determining whether person P is an occupant of the vehicle VE.

[0064] Although each of the above-mentioned first to fifth embodiments has its own effect, some or all of these may be combined for processing, which can improve the accuracy of the determination.

[0065] <Summary of various aspects of the present application> Various aspects of the present application will be summarized below as appendices. (Appendix 1) an image acquisition unit that acquires an image from the imaging unit; an object detection unit that detects an object from the image and forms a detection area surrounding the object; an object analysis unit that analyzes the type of the object within the detection area; an occupant determination unit that determines whether the person is an occupant of the vehicle when the object type is analyzed to include a vehicle and a person; an information processing unit that processes the person as vehicle information of the vehicle when the occupant determination unit determines that the person is an occupant of the vehicle; An information processing device comprising:

[0066] (Appendix 2) The information processing device described in Appendix 1, wherein when the object analysis unit analyzes that the object type includes the vehicle and the person, the occupant determination unit determines that the person is an occupant of the vehicle if the overlap ratio, which is the ratio of the overlapping portion of the detection area of ​​the person and the detection area of ​​the vehicle to the detection area of ​​the person, is equal to or greater than an overlap ratio threshold set for each model of the vehicle.

[0067] (Appendix 3) The detection area formed by the object detection unit is a two-dimensional area, The information processing device described in Appendix 2, wherein the occupant determination unit determines that the person is an occupant of the vehicle if the overlapping area of ​​the two-dimensional areas formed on the person and the vehicle is equal to or greater than the value obtained by multiplying the area of ​​the two-dimensional area of ​​the person by the overlap ratio threshold set for each vehicle type.

[0068] (Appendix 4) The detection area formed by the object detection unit is a three-dimensional area, The information processing device described in Appendix 2, wherein the occupant determination unit determines that the person is an occupant of the vehicle if the volume of the overlapping three-dimensional areas formed on the person and the vehicle is equal to or greater than the value obtained by multiplying the volume of the person's three-dimensional area by the overlap ratio threshold set for each vehicle type.

[0069] (Appendix 5) 5. The information processing device according to claim 2, wherein the object analysis unit analyzes that the object types include the vehicle and the person, and when the overlap ratio is equal to or greater than an overlap ratio threshold set for each vehicle model, the information processing device analyzes the positional relationship between the detection area of ​​the vehicle and the detection area of ​​the person.

[0070] (Appendix 6) An information processing device as described in any one of Appendices 1 to 5, wherein the occupant determination unit determines that the person is not an occupant of the vehicle if the distance between the lower end of the detection area of ​​the person and the lower end of the detection area of ​​the vehicle is smaller than a value obtained by multiplying the height of the detection area of ​​the vehicle by a height ratio threshold set for each vehicle type.

[0071] (Appendix 7) An information processing device according to any one of appendices 1 to 6, wherein, when the object analysis unit analyzes that the object types include the vehicle and the person, a person-specific part corresponding to the person is identified below the detection area of ​​the person, and when the type of the object present in an adjacent part adjacent to the person-specific part is the vehicle, the occupant determination unit determines that the person is an occupant of the vehicle.

[0072] (Appendix 8) an object tracking unit that tracks the object detected by the object detection unit, An information processing device according to any one of appendices 1 to 7, wherein, when the object analysis unit analyzes that the object types include the vehicle and the person, the occupant determination unit determines whether the person is an occupant of the vehicle using a tracking image acquired from the object tracking unit.

[0073] (Appendix 9) The information processing device described in Appendix 8, wherein the occupant determination unit determines that the person is an occupant of the vehicle if the speed difference between the person and the vehicle calculated from the tracking image is less than or equal to a speed difference threshold.

[0074] (Appendix 10) The information processing device described in Appendix 8, wherein the occupant determination unit determines that the person is an occupant of the vehicle if the stopping time difference between the person and the vehicle calculated from the tracking image is less than or equal to a stopping time difference threshold.

[0075] (Appendix 11) a posture estimation unit that estimates a posture of the person when the type of the object is analyzed to be the person; 11. The information processing device according to any one of claims 1 to 10, wherein the object analysis unit estimates a type of the vehicle based on the posture estimated by the posture estimation unit.

[0076] (Appendix 12) an imaging unit that captures an image of the surroundings of the autonomous driving vehicle; The information processing device according to any one of Supplementary Note 1 to Supplementary Note 11, which acquires the image from the imaging unit and processes vehicle information around the autonomous driving vehicle; An information processing system comprising:

[0077] (Appendix 13) 13. The information processing system according to claim 12, wherein the imaging unit is mounted on the autonomous driving vehicle or installed on a structure around a road.

[0078] (Appendix 14) acquiring an image of an area surrounding the autonomous vehicle; detecting an object from the image and forming a detection region surrounding at least a portion of the object; analyzing the type of the object within the detection area; If the object type is analyzed to include a vehicle and a person, determining whether the person is a passenger in the vehicle; When the person is determined to be the passenger of the vehicle, processing the person as vehicle information of the vehicle; An information processing method comprising:

[0079] Although various exemplary embodiments are described in this disclosure, the various features, aspects, and functions described in one or more embodiments are not limited to the application of a particular embodiment, but may be applied to the embodiments alone or in various combinations. Therefore, countless variations not illustrated are contemplated within the scope of the technology disclosed herein. For example, this includes cases where at least one component is modified, added, or omitted, and even cases where at least one component is extracted and combined with components of another embodiment. [Explanation of symbols]

[0080] 1 imaging range, 2 road, 10 information processing system, 21 structure, 22 surrounding object, 100 imaging unit, 200 information processing device, 201 image acquisition unit, 202 object detection unit, 203 object analysis unit, 203a object information storage unit, 204 occupant determination unit, 204a parameter setting unit, 205 information processing unit, 206 part identification unit, 207 object tracking unit, 207a tracked image, 208 posture estimation unit, VE, VE1, VE2 vehicle, AD autonomous driving vehicle, P, P1, P2 person, Bv, Bv1, Bv2, B3v, Bp, Bp1, Bp2, B3p detection area, RGv vehicle area, RGp person area, PTp person identified part, PTa adjacent part

Claims

1. an image acquisition unit that acquires an image from the imaging unit; an object detection unit that detects an object from the image and forms a detection area surrounding the object; an object analysis unit that analyzes the type of the object within the detection area; an occupant determination unit that determines whether the person is an occupant of the vehicle when the object type is analyzed to include a vehicle and a person; an information processing unit that processes the person as vehicle information of the vehicle when the occupant determination unit determines that the person is an occupant of the vehicle; An information processing device comprising:

2. 2. The information processing device according to claim 1, wherein when the object analysis unit analyzes that the object type includes the vehicle and the person, the occupant determination unit determines that the person is an occupant of the vehicle if the overlap ratio, which is the ratio of the overlapping portion of the detection area of ​​the person and the detection area of ​​the vehicle to the detection area of ​​the person, is equal to or greater than an overlap ratio threshold set for each vehicle model.

3. The detection area formed by the object detection unit is a two-dimensional area, The information processing device described in claim 2, wherein the occupant determination unit determines that the person is an occupant of the vehicle if the area of ​​overlap between the two-dimensional areas formed on the person and the vehicle is greater than or equal to the area of ​​the two-dimensional area of ​​the person multiplied by the overlap ratio threshold set for each vehicle type.

4. The detection area formed by the object detection unit is a three-dimensional area, The information processing device described in claim 2, wherein the occupant determination unit determines that the person is an occupant of the vehicle if the volume of the overlapping three-dimensional areas formed on the person and the vehicle is equal to or greater than the value obtained by multiplying the volume of the person's three-dimensional area by the overlap ratio threshold set for each vehicle type.

5. 3. The information processing device according to claim 2, wherein when the object analysis unit analyzes that the object type includes the vehicle and the person and the overlap ratio is equal to or greater than an overlap ratio threshold set for each vehicle model, the information processing device analyzes the positional relationship between the detection area of ​​the vehicle and the detection area of ​​the person.

6. The information processing device described in claim 5, wherein the occupant determination unit determines that the person is not an occupant of the vehicle if the distance between the lower end of the detection area of ​​the person and the lower end of the detection area of ​​the vehicle is smaller than a value obtained by multiplying the height of the detection area of ​​the vehicle by a height ratio threshold set for each vehicle type.

7. 2. The information processing device according to claim 1, wherein when the object analysis unit analyzes that the object types include the vehicle and the person, the occupant determination unit identifies a person-specific part corresponding to the person at the bottom of the detection area of ​​the person, and when the type of the object present in an adjacent part adjacent to the person-specific part is the vehicle, the occupant determination unit determines that the person is an occupant of the vehicle.

8. an object tracking unit that tracks the object detected by the object detection unit, 2. The information processing device according to claim 1, wherein when the object analysis unit analyzes that the object type includes the vehicle and the person, the occupant determination unit determines whether the person is an occupant of the vehicle using a tracking image acquired from the object tracking unit.

9. The information processing device according to claim 8 , wherein the passenger determination unit determines that the person is a passenger of the vehicle when a speed difference between the person and the vehicle calculated from the tracking image is equal to or less than a speed difference threshold.

10. The information processing device according to claim 8 , wherein the occupant determination unit determines that the person is an occupant of the vehicle when a stop time difference between the person and the vehicle calculated from the tracking image is equal to or less than a stop time difference threshold.

11. a posture estimation unit that estimates a posture of the person when the type of the object is analyzed to be the person; The information processing device according to claim 1 , wherein the object analysis unit estimates a type of the vehicle based on the orientation estimated by the orientation estimation unit.

12. an imaging unit that captures an image of the surroundings of the autonomous driving vehicle; an information processing device according to any one of claims 1 to 11, which acquires the image from the imaging unit and processes vehicle information around the autonomous driving vehicle; An information processing system comprising:

13. The information processing system according to claim 12 , wherein the imaging unit is mounted on the autonomous driving vehicle or installed on a structure around a road.

14. acquiring an image of an area surrounding the autonomous vehicle; detecting an object from the image and forming a detection region surrounding at least a portion of the object; analyzing the type of the object within the detection area; If the object type is analyzed to include a vehicle and a person, determining whether the person is a passenger in the vehicle; When the person is determined to be the passenger of the vehicle, processing the person as vehicle information of the vehicle; An information processing method comprising:

Images