Information processing device and information processing method

Abstract

To suppress degradation in visibility derived from superposed display of information concerning a detected object.SOLUTION: An information processor according to the present disclosure includes an acquisition unit, a detection unit, a determination unit, and a display control unit. The determination unit determines a display form concerning a detected object on the basis of whether an object around a vehicle detected by the detection unit on the basis of sensing information acquired by the acquisition unit satisfies a predetermined requirement. As the display form concerning the detected object, when the detected object does not satisfy the predetermined requirement, the determination unit determines a first form in which detection information signifying a result of detection concerning the detected object is displayed in a superposed manner, and when the detected object satisfies the predetermined requirement, the determination unit determines a second form in which support information concerning the detected object is displayed in a superposed manner together with the detection information. The display control unit outputs display information for use in displaying in a superposed manner information concerning the detected object in the display form determined by the determination unit.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to an information processing device. [Background technology]

[0002] Conventionally, there is known a technology for supporting the driving of a moving body such as a vehicle by superimposing support information on a display image or a user's field of vision to draw attention and support the driving of a moving body such as a vehicle. For example, Patent Document 1 discloses a technology for superimposing an alternative image corresponding to a detected three-dimensional object on an overhead image generated based on an image captured by an on-board camera that captures the surrounding area of ​​the vehicle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2012-257106 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, by superimposing the displayed image or information about an object detected in the user's field of vision, visibility may be reduced, such as making the displayed image difficult to see, blocking the user's field of vision, or unnecessarily directing the user's gaze, slowing recognition.

[0005] One of the problems that the present disclosure aims to solve is to suppress a decrease in visibility that occurs when information about a detected object is displayed in a superimposed manner. [Means for solving the problem]

[0006] The information processing device according to the present disclosure includes an acquisition unit, a detection unit, a determination unit, and a display control unit. The acquisition unit acquires sensing information from an on-board sensor mounted on a vehicle. The detection unit detects an object around the vehicle based on the acquired sensing information. The determination unit determines a display mode for the detected object based on whether the detected object satisfies a predetermined condition. The display control unit outputs display information for superimposing information about the detected object in the display mode determined by the determination unit. When the detected object does not satisfy the predetermined condition, the determination unit determines a first mode in which detection information indicating a detection result about the detected object is superimposed as the display mode for the detected object. When the detected object satisfies the predetermined condition, the determination unit determines a second mode in which support information about the detected object is superimposed together with the detection information as the display mode for the detected object. Effect of the Invention

[0007] According to the present disclosure, it is possible to suppress a decrease in visibility that occurs when information about a detected object is displayed in a superimposed manner. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of a vehicle equipped with an information processing device according to an embodiment. [Diagram 2] FIG. 2 is a block diagram illustrating an example of a hardware configuration of the information processing device according to the embodiment. [Diagram 3] FIG. 3 is a block diagram illustrating an example of a functional configuration of the information processing device according to the embodiment. [Figure 4] FIG. 4 is a flowchart showing an example of the flow of information processing executed in the information processing device according to the embodiment. [Diagram 5] FIG. 5 is a diagram showing an example of a display of a detection result in a first mode in the information processing according to the embodiment. [Figure 6]FIG. 6 is a diagram showing an example of a display of a detection result in the second mode in the information processing according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of an information processing device, a vehicle, an information processing system, an information processing method, a program, and a storage medium according to the present disclosure will be described with reference to the drawings.

[0010] In the description of the present disclosure, components having the same or substantially the same functions as those described above with respect to the previously-mentioned drawings may be given the same reference numerals, and the description may be omitted as appropriate. In addition, even when the same or substantially the same parts are shown, the dimensions and ratios of the components may be different depending on the drawing. In addition, for example, from the viewpoint of ensuring the visibility of the drawings, reference numerals may be given only to the main components in the description of each drawing, and reference numerals may not be given to components having the same or substantially the same functions as those described above with respect to the previously-mentioned drawings.

[0011] The information processing system according to the present disclosure includes a vehicle 1 and an information processing device 3. Fig. 1 is a diagram illustrating an example of a vehicle 1 equipped with an information processing device 3 according to an embodiment.

[0012] Here, the vehicle 1 according to the embodiment is an example of a moving body. As the moving body according to the present disclosure, various vehicles such as wheelchairs, bicycles, electric kickboards, automobiles, motorcycles, and railroad cars can be appropriately used. In addition, the driving method of the moving body 2 may be any, for example, it may be driven by an installed battery, may be driven by an internal combustion engine, or may be driven by human power. In addition, the technology according to the present disclosure is not limited to vehicles, and can be appropriately applied to various moving bodies such as ships and aircraft.

[0013] In the information processing system according to the present disclosure, a part of the processing of the information processing device 3 may be executed by a device external to the information processing device 3. For example, a part or all of the information processing device 3 may be provided outside the mobile body. That is, the information processing system according to the present disclosure may be configured by an information processing device 3 provided outside the mobile body so as to be able to communicate with the mobile body, and the mobile body. Alternatively, the information processing system according to the present disclosure may be configured by an information processing device 3 mounted on the mobile body, an information processing device 3 provided outside the mobile body so as to be able to communicate with at least one of the mobile body and the in-vehicle information processing device 3, and the mobile body.

[0014] 1, the vehicle 1 has a vehicle body 12 and two pairs of wheels 13 arranged along a predetermined direction on the vehicle body 12. The two pairs of wheels 13 include a pair of front tires 13f and a pair of rear tires 13r.

[0015] 1 illustrates a vehicle 1 having four wheels 13, but is not limited thereto. The vehicle 1 may have at least one front tire 13f and at least one rear tire 13r. The number of wheels 13 of the vehicle 1 may be two, three, five or more.

[0016] The direction of at least one wheel (steered wheel) among the wheels 13 of the vehicle 1 is electrically or mechanically linked to the rotation angle, i.e., steering angle, of a steering wheel disposed in front of the driver's seat of the vehicle 1. In other words, the vehicle 1 can turn right or left by steering. The steered wheel may be the rear tire 13r, or both the front tire 13f and the rear tire 13r.

[0017] The vehicle body 12 is supported by wheels 13. The vehicle 1 has a driving machine (not shown) and can move by driving at least one wheel (driving wheel) of the wheels 13 of the vehicle 1 with the power of the driving machine. Any driving machine can be used as the driving machine, such as an engine fueled by gasoline or hydrogen, a motor using power from a battery, or a combination of an engine and a motor. In this case, the predetermined direction in which the two pairs of wheels 13 are arranged is the traveling direction of the vehicle 1. The vehicle 1 can move forward or backward by switching gears (not shown) or the like.

[0018] The vehicle body 12 has a front end portion F which is an end portion on the front tire 13f side, and a rear end portion R which is an end portion on the rear tire 13r side. The vehicle body 12 has a substantially rectangular shape when viewed from above, and the four corners of the substantially rectangular shape are sometimes referred to as ends.

[0019] A pair of bumpers 14 are provided at the front and rear ends F, R of the vehicle body 12 near the lower end of the vehicle body 12. Of the pair of bumpers 14, a front bumper 14f covers the entire front surface and part of the side surface near the lower end of the vehicle body 12. Of the pair of bumpers 14, a rear bumper 14r covers the entire rear surface and part of the side surface near the lower end of the vehicle body 12.

[0020] The vehicle 1 has at least one on-board sensor. Each of the at least one on-board sensor is connected to the information processing device 3 via an on-board network including a controller area network (CAN) or Ethernet (registered trademark).

[0021] FIG. 1 illustrates a sonar (Sound Navigation And Ranging) 211 and an all-around camera 212 as examples of on-board sensors of the vehicle 1.

[0022] The sonar 211 is provided, for example, at a predetermined end of the vehicle body 12, and transmits and receives sound waves such as ultrasonic waves. The sonar 211 includes wave transmitting and receiving units 211f, 211r. For example, one or more wave transmitting and receiving units 211f are disposed on the front bumper 14f, and one or more wave transmitting and receiving units 211r are disposed on the rear bumper 14r. The number and / or positions of the wave transmitting and receiving units 211f, 211r are not limited to the example shown in FIG. 1 and can be changed as appropriate. For example, the vehicle 1 may be provided with the wave transmitting and receiving units 211f, 211r on the left and right sides.

[0023] The sonar 211 detects objects around the vehicle 1 based on the results of transmitting and receiving sound waves. The sonar 211 also measures the distance between the vehicle 1 and objects around the vehicle 1 based on the results of transmitting and receiving sound waves. Here, the objects to be detected include various three-dimensional objects such as people (pedestrians), bicycles, automobiles, and obstacles such as structures.

[0024] In the present embodiment, the sonar 211 uses sound waves such as ultrasonic waves, but is not limited thereto. For example, instead of the sonar 211 or in addition to the sonar 211, the vehicle 1 may have a radar (Radio Detection and Ranging) that transmits and receives radio waves or a LiDAR (Light Detection and Ranging) that transmits and receives laser light.

[0025] The all-around camera 212 is provided on the vehicle 1 so as to be able to capture images of the surroundings of the vehicle 1. As an example, the vehicle 1 has, as the all-around camera 212, a front camera 212a that captures images in front, a rear camera 212b that captures images in the rear, a left side camera 212c that captures images on the left side, and a right side camera (not shown) that captures images on the right side.

[0026] The all-around camera 212 captures images of the surroundings of the vehicle 1. The all-around camera 212 is, for example, a camera that captures images based on visible light and / or infrared light. Note that the images captured by the all-around camera 212 may be moving images or still images.

[0027] The position and / or number of the all-around camera 212 is not limited to the example shown in FIG. 1 and can be changed as appropriate. For example, the vehicle 1 may have only two cameras, the front camera 212a and the rear camera 212b. Alternatively, the vehicle 1 may have other cameras in addition to the above-mentioned examples. For example, a part or all of the all-around camera 212 may be provided in the vehicle cabin space (inside the vehicle) of the vehicle 1. Furthermore, the all-around camera 212 may be a camera built into the vehicle 1, or may be a camera of a drive recorder attached to the vehicle 1.

[0028] The information processing device 3 is an example of an information processing device that can be mounted on the vehicle 1. The information processing device 3 is configured to be able to execute information processing according to the embodiment, for example, based on sensing information from an on-board sensor of the vehicle 1. The information processing device 3 is realized, for example, by at least one ECU (Electronic Control Unit) or OBU (On Board Unit) provided inside the vehicle 1.

[0029] The information processing device 3 may be realized by a DCU (Domain Control Unit) such as a CDC (Cockpit Domain Controller) that integrates multiple ECUs. This CDC is configured to be capable of executing various processes such as in-vehicle infotainment (IVI), meter control, display device control such as a HUD (Head Up Display) and an electronic mirror, and ADAS (Advanced Driver-Assistance Systems). For example, the information processing device 3 may also function as a car navigation device or the like. Alternatively, the information processing device 3 may be an external computer installed near the dashboard of the vehicle 1.

[0030] Fig. 2 is a diagram showing an example of a hardware configuration of an information processing device 3 according to an embodiment. As shown in Fig. 2, the information processing device 3 has a processor 31, a memory 32, an HMI 33, and a communication I / F (interface) 36. The HMI 33 includes, for example, a display 34 and an input I / F 35. The processor 31, the memory 32, the display 34, the input I / F 35, and the communication I / F 36 are connected to each other via a bus 39 or the like, and have a hardware configuration using a normal computer.

[0031] The processor 31 is an arithmetic unit that controls the entire information processing device 3. The processor 31 loads a program stored in a ROM (Read Only Memory) or an HDD (Hard Disk Drive) of the memory 32 into a RAM (Random Access Memory) of the memory 32 and executes the program, thereby implementing each process described below.

[0032] As the processor 31, various processors such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), or a dedicated arithmetic circuit realized by an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array) can be used as appropriate.

[0033] The memory 32 includes, for example, a ROM that stores programs, parameters, and the like that realize various processes by the processor 31. The memory 32 also includes, for example, a RAM as a main storage device that temporarily stores data necessary for various processes by the processor 31. Note that the memory 32 may include various storage media and storage devices, such as an HDD, an SSD (Solid State Drive), a Flash memory, and the like, that store various data, programs, and the like used by each device in the information processing system 9.

[0034] The HMI 33 is an interface for outputting various types of information, such as route guidance, notifications, and warnings, to the driver of the vehicle 1. The HMI 33 is also an interface for accepting input of various types of information by the driver of the vehicle 1. Here, the HMI 33 according to the embodiment is an example of a display unit and an input unit. The HMI 33 only needs to be able to output various types of information recognizable by the driver of the vehicle 1 and accept various operations by the driver of the vehicle 1, and is provided, for example, around the driver's seat of the vehicle 1, but may also be provided in other areas around the driver's seat, such as the back seat.

[0035] As an example, the HMI 33 includes a display that is provided on a dashboard or a console of the vehicle 1 and is configured to be able to output video. The display is, for example, a liquid crystal display (LCD) or an organic EL (Electro Luminescence) display.

[0036] The display may be configured as a touch panel display. Also, the display may be a part of a car navigation device mounted on the vehicle 1.

[0037] The display may be a projection-type display device such as a HUD that projects an image (virtual image) onto a display area provided in front of the driver, for example, on the windshield or dashboard (console).

[0038] The display may be a display device that uses XR (Cross Reality) technology, such as AR (Augmented Virtuality) glasses worn by a user, to superimpose and display support information related to a detected object on the field of view of the driver of the vehicle 1. This display device is not limited to a glasses-type HMD (Head Mounted Display) such as AR glasses, but may be a contact lens-type wearable device. This display device is not limited to AR technology, and various XR technologies such as VR (Virtual Reality), MR (Mixed Reality), and SR (Substitutional Reality) can be used.

[0039] As an example, the HMI 33 may be a display device that projects an image onto the retina. The display device may have, for example, a projection unit that emits laser light to display an image, and an optical system including a mirror that forms an image of the laser light from the projection unit onto the retina. Alternatively, the display device may have, for example, an optical system including a half mirror, and a projection unit that projects an image onto the half mirror.

[0040] The HMI 33 may include other output devices such as a speaker configured to output notification sounds, warning sounds, and audio.

[0041] As an example, the HMI 33 includes a touch panel of a touch panel display as an input device. The HMI 33 may include other input devices such as buttons, dials, switches, and a microphone. These input devices are arranged on, for example, the dashboard, instrument panel, steering wheel, console, and the like of the vehicle 1.

[0042] In addition, as the HMI 33, an operation terminal capable of transmitting or transmitting signals to the vehicle 1 from outside the vehicle 1, such as a tablet terminal, a smartphone, a remote controller, or an electronic key, may be used.

[0043] The communication I / F 36 is an interface for transmitting and receiving data to and from the outside of the information processing device 3. As an example, the communication I / F 36 receives data from other devices provided in the vehicle 1, such as on-board sensors. Note that the communication I / F 36 may transmit and receive information to and from other ECUs mounted on the vehicle 1 via an on-board network of the vehicle 1.

[0044] The communication I / F 36 may communicate with an information processing device outside the vehicle 1 via an electric communication line such as the Internet. For example, the communication I / F 36 may be configured to be capable of transmitting and receiving a radio signal for transmitting information between the vehicle 1 and a radio base station. In this case, the communication I / F 36 is configured to be capable of transmitting and receiving a radio signal between the vehicle 1 and a radio base station, for example, via an antenna (not shown) mounted on the vehicle 1. The transmission and reception of the radio signal is performed by any cellular V2X method, such as a mobile communication system of each generation according to the IMT2020 regulations or the specifications of 3GPP (registered trademark), such as 3G, LTE, 4G, or 5G, or a next-generation mobile communication system such as Beyond 5G (6G). The communication method for transmitting and receiving the radio signal may be another communication method, such as a DSRC method conforming to the IEEE or a vehicle-to-vehicle / road-to-vehicle communication (V2N) method.

[0045] 3 is a diagram showing an example of a functional configuration of an information processing device 3 according to an embodiment. The information processing device 3 realizes functions as an acquisition unit 301, a detection unit 302, an overlay determination unit 303, and a display control unit 304, for example, by causing a processor 31 to execute a program loaded in a RAM of a memory 32.

[0046] The acquisition unit 301 acquires sensing information from an on-board sensor mounted on the vehicle via the I / F 45, for example.

[0047] As an example, the acquisition unit 301 acquires images and measurement results (sensing information) from the sonar 211, the omnidirectional camera 212, RADAR, LiDAR, and IMU as on-board sensors.

[0048] The detection unit 302 detects objects around the vehicle 1 based on the sensing information acquired by the acquisition unit 301.

[0049] As an example, the detection unit 302 detects an object included in an image (sensing information) captured by the omnidirectional camera 212 of at least one of the front and rear of the vehicle 1, using image processing such as edge detection and object recognition processing.

[0050] As an example, the detection unit 302 detects an object present at least in front of or behind the vehicle 1 based on measurement results (sensing information) by the sonar 211, RADAR, or LiDAR relating to at least one of the front and rear of the vehicle 1.

[0051] The superimposition determination unit 303 determines a display mode for the detected object based on whether or not the object detected by the detection unit 302 satisfies a predetermined condition related to visibility. For example, the superimposition determination unit 303 determines whether or not an object has been detected by the detection unit 302. Also, for example, the superimposition determination unit 303 determines whether or not the detected object is difficult to see. Here, the superimposition determination unit 303 according to the embodiment is an example of a determination unit.

[0052] In the present disclosure, a situation in which a detected object is difficult to see refers to a situation in which the visibility of the detected object on the display screen of the HMI 33 or the visibility in the driver's field of vision is low. Here, a situation in which a detected object is difficult to see is an example of a case in which a predetermined condition related to visibility is satisfied.

[0053] As one example, the situation in which a detected object is difficult to see may be a time or time period that corresponds to a predetermined time or time period that is stored in memory 32, such as nighttime or twilight hours around sunset. Note that the situation in which a detected object is difficult to see may be a time or time period that does not correspond to a predetermined time or time period that is stored in memory 32, such as daytime.

[0054] As an example, a situation in which a detected object is difficult to see may be a situation in which the illuminance around the detected object, such as at night or inside a tunnel, or the illuminance around vehicle 1 is outside the threshold range, such as being lower than a predetermined threshold value stored in memory 32 or the like.

[0055] As an example, a situation in which a detected object is difficult to see may be a situation in which the contrast difference between the detected object and the background on the image of the object or in the driver's field of vision is smaller than a predetermined threshold value stored in memory 32 or the like, such as a case in which a person as the detected object is wearing gray clothes and standing in front of concrete, and falls outside the range of a threshold value. The contrast difference may be calculated using a color-related value such as lightness, refraction, or hue, or may be calculated using a luminance value on the image.

[0056] As an example, a situation in which a detected object is difficult to see may be a case in which the detected object is cut off in the image or in the field of view of the driver. Note that a case in which a detected object is cut off may be a case in which the entire detected object, or a portion of the detected object equal to or greater than a predetermined proportion stored in memory 32, is not within the angle of view of omnidirectional camera 212 or the field of view of the driver, or a case in which the detected object is hidden by another object such as an obstacle.

[0057] As an example, a situation in which a detected object is difficult to see may be a situation in which, when a process of synthesizing two or more images obtained by the omnidirectional camera 212 is performed, or when a process of viewpoint conversion is performed on one or more images obtained by the omnidirectional camera 212, distortion in the image of the detected object is greater than a predetermined threshold value stored in memory 32 or the like, or falls outside that range.

[0058] In addition, when the detected object is not difficult to see, i.e., when the specified conditions related to visibility are not satisfied, the overlay judgment unit 303 determines a first mode in which detection information indicating the detection result for the detected object is overlaid as the display mode by the HMI 33.

[0059] The display in the first mode may be a simpler drawing than the display in the second mode described below, and is not limited to a mode in which information indicating the detection result is displayed for each detected object. A simple drawing may be expressed as a simplified display, and refers to a small area or number of displays superimposed on the image or the driver's field of vision. In addition, in the display in the first mode, each display of information indicating the detection result may be displayed at a position unrelated to the position of the detected object.

[0060] As an example, the detection information indicating the detection result is a detection frame indicating the detected object. Note that, instead of or in addition to the detection frame, the detection information may be a frame that highlights the entire image or the driver's field of view, an icon and / or text indicating that an object has been detected, or a combination of these.

[0061] Furthermore, when the detected object is difficult to see, i.e., when a predetermined condition related to visibility is satisfied, the superimposition determination unit 303 determines a second mode in which a pictogram indicating the type of the detected object is superimposed at a position corresponding to the detected object together with detection information such as a detection frame, as the display mode of the detection result by the HMI 33. Here, the pictogram indicating the type of the detected object is an example of support information.

[0062] Note that the support information superimposed on the detected object in the second embodiment is not limited to a pictogram. For example, in the second embodiment, the silhouette of the detected object may be superimposed as the support information.

[0063] In the second embodiment, the color, size, and direction of the detected object may be reflected in the pictogram. For example, the pictogram may be displayed in a color tone corresponding to the color of the detected object. For example, when the detected object is a person, the pictogram displayed may be different depending on whether the object is a child or an adult. For example, when the detected object is a vehicle, the pictogram may be different between a truck and a passenger car. In other words, the pictograms may be prepared for classifications that are more detailed than the type.

[0064] In the second embodiment, a pictogram common to a plurality of types may be used as the pictogram. In other words, pictograms do not necessarily have to be prepared for each type of object, but may be prepared for each classification larger than the type.

[0065] Furthermore, the size of the pictogram superimposed in the second mode may be determined according to the distance from the vehicle 1. For example, the size of the pictogram becomes smaller as it is farther away from the vehicle 1. For example, the size of the pictogram may be determined according to the size of the detected object on the image or in the driver's field of vision. For example, if the detected object is cut off, the entire pictogram may be superimposed and displayed at a size determined based on the distance to the detected object.

[0066] In addition, the position of the pictogram superimposed on the image or in the field of view of the driver in the second aspect may be adjusted according to the reference position of the detected object. For example, if the detected object is a person, the position of the pictogram superimposed may be adjusted to match the foot position of the detected person. For example, if the detected object is a vehicle, the position of the pictogram superimposed may be adjusted to match the tire position of the detected vehicle. On the other hand, for example, if the reference position of the detected object, such as the foot position of the detected person or the tire position of the detected vehicle, is cut off, the position of the pictogram superimposed may not be adjusted.

[0067] In the second aspect, when any of the detected objects is difficult to see, a pictogram may be superimposed only on the object that is difficult to see. On the other hand, when any of the detected objects is difficult to see, pictograms may be superimposed on the other detected objects regardless of whether they are difficult to see.

[0068] In addition, the position of the pictogram superimposed on the image or in the field of vision of the driver in the second mode may be adjusted to be shifted to the left or right, for example, to ensure that the detected object is easily visible. As an example, in a situation where the detected object is easily visible, such as when it is bright, the pictogram may be superimposed at a position shifted to the side from the position of the detected object.

[0069] The display control unit 304 controls the display by the HMI 33. For example, the display control unit 304 displays information about the detected object by the HMI 33 in the display mode determined by the overlay determination unit 303. Specifically, the display control unit 304 generates display information for displaying information about the detected object in the display mode determined by the overlay determination unit 303, and outputs the generated display information to the HMI 33.

[0070] As an example, when the detected object is not difficult to see, the display control unit 304 causes the HMI 33 to display the detection result in a first manner.

[0071] As an example, when a detected object is difficult to see, the display control unit 304 displays the detection result on the HMI 33 in a second manner.

[0072] It should be noted that the detection unit 302 and the overlap determination unit 303 may be realized as one function.

[0073] Next, the flow of information processing executed in the information processing system configured as above will be described. Note that the flow of processing described below is one example, and it is possible to change the processing order, delete some processes, and add other processes.

[0074] FIG. 4 is a flowchart showing an example of the flow of information processing executed in the information processing device 3 according to the embodiment.

[0075] The acquisition unit 301 acquires sensor data (sensing information) from an on-board sensor via, for example, the communication I / F 36 (S1). Then, the detection unit 302 executes object detection based on the sensing information (S2).

[0076] The superimposition determination unit 303 determines whether an object has been detected (S3). If an object has not been detected (S3: No), the flow of Fig. 4 returns to the process of S1. On the other hand, if an object has been detected (S3: Yes), the superimposition determination unit 303 determines whether the detected object is difficult to see (S4).

[0077] If the detected object is not difficult to see (S4: No), the superimposition determination unit 303 determines the display mode of the detection result by the HMI 33 to be a first mode in which only the detection information is superimposed. Then, the display control unit 304 displays the detection result by the HMI 33 in the first mode.

[0078] FIG. 5 is a diagram showing an example of a display of a detection result in a first mode in the information processing according to the embodiment.

[0079] As an example, the display control unit 304 causes the HMI 33 to display a display screen 610 in which detection frames 601 indicating people P1 to P4 as detected objects are superimposed on the image acquired by the acquisition unit 301.

[0080] As an example, the display control unit 304 superimposes detection frames 601 at positions corresponding to the persons P1 to P4 as detected objects on the field of view of the driver, and displays them on the HMI 33. This provides the driver with the same field of view as the driver viewing the display screen 610.

[0081] In this way, when the detected object is not difficult to see, the superimposition determination unit 303 determines not to superimpose support information on the detected object.

[0082] On the other hand, if the detected object is difficult to see (S4: Yes), the superimposition determination unit 303 determines the display mode of the detection result by the HMI 33 to be a second mode in which support information is superimposed on the detected object in addition to the detection information. Then, the display control unit 304 displays the detection result by the HMI 33 in the second mode.

[0083] FIG. 6 is a diagram showing an example of a display of a detection result in the second mode in the information processing according to the embodiment.

[0084] As an example, the display control unit 304 displays, via the HMI 33, a display screen 620 in which support information for each of the persons P3 and P4 whose visibility is judged to be difficult is superimposed on the image acquired by the acquisition unit 301, together with detection frames 601 each showing the persons P1 to P4. Fig. 6 illustrates an example of support information in the form of a pictogram 603 showing a type of object.

[0085] For example, in the example of Fig. 6, the pictogram of a distant person P4 is displayed small and the pictogram 603 of a nearby person P3 is displayed large and superimposed depending on the distance from the vehicle 1. For example, in the example of Fig. 6, a pictogram 603 reflecting the direction of the detected person is displayed superimposed. Note that in the example of Fig. 6, when people P3 and P4 are detected in a situation determined to be difficult to see, pictograms may be displayed superimposed on the other detected people P1 and P2.

[0086] As an example, the display control unit 304 displays, in the field of view of the driver, detection frames 601 showing each of the detected persons P1 to P4, together with support information for each of the persons P3 and P4 whose visibility is determined to be difficult, on the HMI 33. This provides the driver with the same field of view as a driver viewing the display screen 620.

[0087] In this way, when a detected object is difficult to see, the superimposition determination unit 303 determines that support information should be superimposed on the detected object.

[0088] Thereafter, if it is determined that the process is not to end (S7: No), the process returns to S1 in the flow of Fig. 4. On the other hand, if it is determined that the process is to end (S7: Yes), the flow of Fig. 4 ends.

[0089] In this way, the information processing device 3 according to the embodiment is configured to superimpose only the detection information indicating the detection result of an object such as a person or an obstacle in a first simple manner such as a frame display in a situation where the detected object is easy to see, such as during the daytime. Also, the information processing device 3 is configured to superimpose a pictogram indicating the type together with the detection information in a situation where the detected object is difficult to see, such as at night or when the contrast difference with the background is small or when the object is cut off.

[0090] According to this configuration, in a situation where the detected object is difficult to see, the display of the support information makes it easier for the user to recognize the detected object. Also, according to the above configuration, by reducing the drawing of the support information in a situation where the detected object is easy to see, it is possible to prevent the display of information about the detected object from interfering with the viewing or visibility of the detected object, making it difficult for the user to recognize the detected object. Therefore, according to the technology according to the embodiment, it is possible to realize behavioral support and driving support such as appropriate attention and information provision to the user, and to improve the convenience of the user, such as preventing the occurrence of accidents.

[0091] In the information processing according to the above-described embodiment, the sensor data (sensing information) from the on-board sensor may be at least one of image data obtained by the omnidirectional camera 212 and measurement result data obtained by the sonar 211. For example, object detection may be performed using at least one of the image and the measurement result. For example, a determination as to whether a detected object satisfies a predetermined condition related to visibility may be performed using at least one of the image and the measurement result.

[0092] The vehicle 1 may be equipped with various sensors (not shown).

[0093] As an example, the vehicle 1 may have an illuminance sensor. In this case, the information processing device 3 may acquire data of the measurement result from the illuminance sensor as sensing information. For example, the transmittance of the pictogram superimposed and displayed in the second mode may be changed according to the illuminance of the detected object or the surroundings of the vehicle 1. For example, the transmittance of the pictogram is smaller as the illuminance is lower. For example, when it is dark, the transmittance of the pictogram can be lowered to make the display of the pictogram stand out. On the other hand, when it is bright, the transmittance of the pictogram can be increased so as not to obstruct the image or the driver's field of vision. In addition, for example, whether or not the position of the pictogram superimposed and displayed in the second mode on the image or in the driver's field of vision can be adjusted to be shifted in the left-right direction may be determined according to the illuminance of the detected object. For example, in a situation where the detected object is difficult to see, such as when it is dark, it is not necessary to adjust the position of the detected object to be shifted to the left.

[0094] As an example, the vehicle 1 may have a raindrop sensor or a sensor that acquires wiper operation. In this case, the information processing device 3 may acquire data of the measurement result from the raindrop sensor or the sensor that acquires wiper operation as sensing information related to weather. For example, the judgment of whether the detected object satisfies a predetermined condition related to visibility may be executed using the sensing information related to weather. As an example, the situation in which the detected object is difficult to see may correspond to a weather that is predetermined and stored in the memory 32, such as rainy weather or cloudy weather. Note that the situation in which the detected object is difficult to see may not correspond to a weather that is predetermined and stored in the memory 32, such as sunny weather. Note that the information processing device 3 may acquire information related to weather on the Internet by communicating with the outside of the vehicle 1 via the communication I / F 36, for example, and judge whether the predetermined condition related to visibility is satisfied based on the acquired information on weather.

[0095] As an example, the on-board sensor of the vehicle 1 includes a steering angle sensor that outputs a signal corresponding to the amount of steering wheel operation by the driver, i.e., the steering angle. As an example, the on-board sensor of the vehicle 1 includes a wheel speed sensor that outputs a signal corresponding to the rotation speed and rotation direction of the wheels 13. As an example, the on-board sensor of the vehicle 1 includes a brake sensor that detects the amount of brake pedal operation by the driver. As an example, the on-board sensor of the vehicle 1 includes an accelerator sensor that detects the amount of accelerator pedal operation by the driver. As an example, the on-board sensor of the vehicle 1 includes an acceleration sensor that outputs a signal corresponding to acceleration applied to the vehicle 1. As an example, the on-board sensor of the vehicle 1 includes a gyro sensor that outputs a signal corresponding to angular velocity applied to the vehicle 1. The acceleration sensor and the gyro sensor may each be provided with three axes and configured as an inertial measurement unit (IMU). As an example, the on-board sensor of the vehicle 1 includes a Global Navigation Satellite System (GNSS) sensor that outputs position information of the vehicle 1, such as a Global Positioning System (GPS) sensor. The GNSS sensor includes a GNSS antenna that receives radio waves from satellites, and a GNSS circuit that obtains position information based on radio waves from at least two satellites received by the GNSS antenna.

[0096] In the above embodiment, the situation where the detected object is difficult to see is exemplified as the case where the predetermined condition related to visibility is satisfied, but this is not limited thereto. For example, the situation where the predetermined condition related to visibility is satisfied may be a situation where the urgency of the detected object is high.

[0097] As one example, a situation of high urgency regarding a detected object may be a situation in which the distance to the detected object is equal to or less than a predetermined distance that is determined in advance and stored in the memory 32 or the like.

[0098] As one example, a situation of high urgency regarding a detected object may be a situation in which an object is detected behind the vehicle 1 while the vehicle 1 is traveling in reverse.

[0099] As an example, a situation of high urgency regarding a detected object may be a situation in which an assessment value of a collision risk, such as the time-to-collision (TTC) with a detected object, falls outside a threshold range, such as being smaller than a predetermined threshold value stored in memory 32 or the like.

[0100] As an example, a high urgency situation regarding a detected object may be when the detected object is included in an area that is not visible to the driver of the vehicle 1.

[0101] In the above embodiment, the sensing information from the on-board sensor of the vehicle 1 is used, but the present invention is not limited to this. The sensing information may be information from an external camera or measurement sensor, such as AR glasses worn on the user's head or chest.

[0102] It should be noted that some or all of the above-described on-board sensors may be included in the information processing device 3. On the other hand, some or all of the above-described HMI 33 may not be included in the information processing device 3.

[0103] In the above embodiment, the user of the information processing device 3 is a driver who moves by a moving body such as a vehicle 1, but is not limited to this. The user of the information processing device 3 is not limited to the driver of the moving body, but may be a passenger of the moving body, or a pedestrian who is not using the moving body. In other words, the information processing system according to the present embodiment may be a system that performs driving support for a moving body, or a system that performs behavior support for a pedestrian. For example, the information processing device 3 according to the embodiment may be configured as a device that outputs display information to a display terminal such as AR glasses worn by a pedestrian or a smartphone held by a pedestrian.

[0104] In the above embodiment, the detection information and support information are superimposed on an image obtained by combining a front image obtained by capturing an image in front of the vehicle 1, a rear image obtained by capturing an image in the rear, and a plurality of images obtained by capturing an image of the surroundings of the vehicle 1 including the front and rear, but the present invention is not limited to this. The image on which the detection information and support information are superimposed may be a top view (bird's-eye view image) obtained by combining a plurality of images obtained by capturing an image of the surroundings of the vehicle 1.

[0105] In each of the above-mentioned embodiments, "determining whether it is A" may mean determining that it is A, determining that it is not A, or determining whether it is A or not.

[0106] Each program executed by each device of the information processing system according to each of the above-described embodiments is provided by being recorded on a computer-readable storage medium such as a CD-ROM, FD, CD-R, or DVD in a file in an installable or executable format.

[0107] Also, each program executed by each device of the information processing system according to each embodiment described above may be configured to be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Also, each program executed by each device of the information processing system according to each embodiment described above may be configured to be provided or distributed via a network such as the Internet.

[0108] Furthermore, each program executed by each device in the information processing system according to each embodiment described above may be configured to be provided by being pre-installed in a ROM or the like.

[0109] In addition, the programs executed in each device of the information processing system according to each of the above-mentioned embodiments have a modular configuration including each of the functional units described above, and in terms of actual hardware, a processor 31 such as a CPU reads out and executes the program from a memory 32 such as a ROM or HDD, thereby loading each of the functional units described above into the RAM of the memory 32, and generating each of the functional units described above in the RAM of the memory 32.

[0110] According to at least one of the embodiments described above, it is possible to suppress a decrease in visibility caused by superimposed display of information related to a detected object.

[0111] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope of the invention and its equivalents described in the claims, as well as in the scope and spirit of the invention.

[0112] (Additional Note) The above description of the embodiments discloses the following techniques. (1) An acquisition unit that acquires sensing information from an on-board sensor mounted in the vehicle; A detection unit that detects objects around the vehicle based on the acquired sensing information; a determination unit that determines a display mode for the detected object based on whether the detected object satisfies a predetermined condition; a display control unit that outputs display information for superimposing and displaying information about the detected object in the display mode determined by the determination unit, The determination unit is When the detected object does not satisfy a predetermined condition, a first display mode is determined as a display mode for the detected object, in which detection information indicating a detection result for the detected object is superimposed; When the detected object satisfies a predetermined condition, a second display mode is determined as a display mode for the detected object, in which support information for the detected object is superimposed on the detection information. Information processing device. (2) The sensing information is at least one of an image captured around the vehicle and a measurement result of measuring the surroundings of the vehicle. The information processing device according to (1) above. (3) When the predetermined condition is satisfied, the detected object is at least one of a situation in which it is difficult to visually see the detected object and a situation in which the urgency of the detected object is high. The information processing device according to (1) or (2) above. (4) The situation in which the detected object is difficult to see includes a case in which the time when the object is detected corresponds to a predetermined time or time period. The information processing device according to (3) above. (5) The situation in which the detected object is difficult to see includes a case in which the illuminance of the detected object or the surroundings of the vehicle falls outside a predetermined threshold range. The information processing device according to (3) or (4) above. (6) The situation in which the detected object is difficult to see includes a situation in which a contrast difference between the detected object and a background of the object falls outside a predetermined threshold range. An information processing device according to any one of (3) to (5) above. (7) The situation in which the detected object is difficult to see includes a situation in which the detected object is cut off. An information processing device according to any one of (3) to (6) above. (8) The situation in which the detected object is difficult to see includes a case where the situation corresponds to a predetermined weather condition, An information processing device according to any one of (3) to (7) above. (9) The detection information is a detection frame showing the detected object. An information processing device according to any one of (1) to (8) above. (10) The support information is a pictogram indicating a type of the detected object or a silhouette of the detected object. An information processing device according to any one of (1) to (9) above. (11) The display information is information for superimposing and displaying information about the detected object on an image captured around the vehicle. An information processing device according to any one of (1) to (10) above. (12) The display information is information for superimposing and displaying information about the detected object on the field of view of the driver of the vehicle. An information processing device according to any one of (1) to (10) above. (13) Acquiring sensing information from an on-board sensor mounted on a vehicle; Detecting an object around the vehicle based on the acquired sensing information; and determining a display mode for the detected object based on whether the detected object satisfies a predetermined condition; outputting display information for superimposing and displaying information about the detected object in the display mode determined by the determination unit; Determining the display mode includes: When the detected object does not satisfy a predetermined condition, a first display mode is determined as a display mode for the detected object, in which detection information indicating a detection result for the detected object is superimposed; and determining, when the detected object satisfies a predetermined condition, a second display mode in which support information related to the detected object is superimposed on the detection information as a display mode related to the detected object. Information processing methods. (14) A program for causing a computer to execute the information processing method described in (13) above. (15) A storage medium (Computer Program Product) having recorded thereon the program described in (14) above, which is executed by a computer. [Explanation of symbols]

[0113] 1 vehicle 211 Sonar 212 All-around camera 3. Information processing equipment 301 Acquisition Department 302 Detection unit 303 Overlap Judgment Unit (Judgment Unit) 304 Display control section 31 Processors 32 Memory 33 Human Machine Interface 34 Display 35 Input I / F 36 Communication I / F 39 Bus

Claims

1. An acquisition unit that acquires sensing information from on-board sensors mounted on the vehicle, A detection unit that detects objects around the vehicle based on the acquired sensing information, A determination unit that determines the display mode for the detected object based on whether or not the detected object satisfies the conditions stored in memory, The system includes a display control unit that outputs information about the detected object to a display based on the aforementioned display mode, If the detected object does not satisfy the above conditions, the display mode is the first mode, and the information regarding the detected object does not include supporting information regarding the detected object. If the detected object satisfies the above conditions, the display mode is the second mode, and the information relating to the detected object includes the support information. Information processing device.

2. The information relating to the detected object includes detection information relating to the detected object, The information processing apparatus according to claim 1.

3. The sensing information is at least one of an image captured of the area around the vehicle and measurement results obtained by measuring the area around the vehicle. The information processing apparatus according to claim 1.

4. If the above conditions are met, then at least one of the following is the situation in which the detected object is difficult to see, or the situation in which there is a high degree of urgency regarding the detected object. The information processing apparatus according to claim 1.

5. The circumstances under which the detected object is difficult to see include cases where the time the object is detected falls within a specified time or time period. The information processing apparatus according to claim 4.

6. The situation in which the detected object is difficult to see includes cases where the illumination around the detected object or the vehicle falls outside a defined threshold range. The information processing apparatus according to claim 4.

7. The situation in which the detected object is difficult to see includes cases where the contrast difference between the detected object and the background of the object falls outside a defined threshold range. The information processing apparatus according to claim 4.

8. The situation in which the detected object is difficult to see includes cases where the detected object is partially obscured. The information processing apparatus according to claim 4.

9. The circumstances under which the detected object is difficult to see include cases that fall under the specified weather conditions. The information processing apparatus according to claim 4.

10. The detection information is a detection frame indicating the detected object. The information processing apparatus according to claim 2.

11. The support information is a pictogram indicating the type of the detected object or a silhouette of the detected object. An information processing apparatus according to any one of claims 1 to 10.

12. The display control unit further outputs an image of the area around the vehicle to the display, The display superimposes the image and the information regarding the detected object. An information processing apparatus according to any one of claims 1 to 10.

13. Acquire sensing information from an on-board sensor mounted on the vehicle, Based on the acquired sensing information, objects around the vehicle are detected. Based on whether the detected object satisfies the conditions stored in memory, the display mode for the detected object is determined. This includes outputting information about the detected object to a display based on the aforementioned display mode, If the detected object does not satisfy the above conditions, the display mode is the first mode, and the information regarding the detected object does not include supporting information regarding the detected object. If the detected object satisfies the above conditions, the display mode is a second mode, and includes support information relating to the detected object. Information processing methods.

14. The information relating to the detected object includes detection information relating to the detected object, The information processing method according to claim 13.

15. The sensing information is at least one of an image taken of the area around the vehicle and measurement results taken of the area around the vehicle. The information processing method according to claim 13.

16. If the above conditions are met, then at least one of the following is a situation in which the detected object is difficult to see, and a situation in which there is a high degree of urgency regarding the detected object. The information processing method according to claim 13.

17. The situation in which the detected object is difficult to see includes the case in which the time the object is detected falls within a specified time or time period. The information processing method according to claim 16.

18. The situation in which the detected object is difficult to see includes the case in which the illumination around the detected object or the vehicle falls outside a defined threshold range. The information processing method according to claim 16.

19. The situation in which the detected object is difficult to see includes the case in which the contrast difference between the detected object and the background of the object falls outside a defined threshold range. The information processing method according to claim 16.

20. The situation in which the detected object is difficult to see includes the case in which the detected object is partially obscured. The information processing method according to claim 16.

Images