Recognition system, recognition device, recognition method and recognition program

Abstract

To provide a recognition system that secures recognition accuracy for a light transmission body.SOLUTION: A processor of a recognition system that recognizes an environment of a host vehicle mounting a visible area camera and an infra-red area camera to generate recognition data Dr is configured to execute: acquiring visible area imaging data Dv captured by the visible area camera for the environment and infra-red imaging data Di captured by the infra-red area camera for the environment including a common imaging area Ac with the visible area camera; and outputting recognition data Dr reporting the presence of a light transmission body in the common imaging area Ac in response to a reduction of the degree of coincidence between brightness boundaries Bv, Bi recognized for the edges to outside an allowance range in each of the visible area imaging data Dv and the infrared imaging data Di.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to a recognition technology for recognizing the external environment of a vehicle. [Background technology]

[0002] The recognition technology disclosed in Patent Document 1 is capable of recognizing a translucent object present in the external environment of the vehicle by combining a visible range camera mounted on the vehicle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2017-220923 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with the technology disclosed in Patent Document 1, even when combined with a visible range camera mounted on a vehicle, it was difficult to ensure sufficient recognition accuracy in recognizing the presence of a translucent body, for example, glass, because the reflectance of the body is low for visible light.

[0005] An object of the present disclosure is to provide a recognition system that ensures recognition accuracy for translucent bodies. Another object of the present disclosure is to provide a recognition device that ensures recognition accuracy for translucent bodies. Yet another object of the present disclosure is to provide a recognition method that ensures recognition accuracy for translucent bodies. Yet another object of the present disclosure is to provide a recognition program that ensures recognition accuracy for translucent bodies. [Means for solving the problem]

[0006] The technical means of the present disclosure for solving the problems will be described below. Note that the claims and the reference characters in parentheses in this section indicate the correspondence with the specific means described in the embodiments described later in detail, and do not limit the technical scope of the present disclosure.

[0007] A first aspect of the present disclosure is A recognition system having a processor (12), which recognizes the external environment of a host vehicle (2) equipped with a visible range camera (40) and an infrared range camera (42) and generates recognition data (Dr), The processor Obtaining visible range photographing data (Dv) photographed by a visible range camera against the outside world, and infrared range photographing data (Di) photographed by an infrared range camera against the outside world including a common photographing area (Ac) with the visible range camera; In response to the degree of coincidence between the brightness boundaries (Bv, Bi) recognized as edges in each of the visible range photography data and the infrared range photography data decreasing to outside the allowable range, recognition data notifying the presence of a translucent body (30) in the common photography area is output.

[0008] A second aspect of the present disclosure is A recognition device configured to be mountable on a host vehicle (2) together with a visible range camera (40) and an infrared range camera (42), the recognition device having a processor (12) for recognizing the external environment of the host vehicle and generating recognition data (Dr), The processor Obtaining visible range photographing data (Dv) photographed by a visible range camera against the outside world, and infrared range photographing data (Di) photographed by an infrared range camera against the outside world including a common photographing area (Ac) with the visible range camera; In response to the degree of coincidence between the brightness boundaries (Bv, Bi) recognized as edges in each of the visible range photography data and the infrared range photography data decreasing to outside the allowable range, recognition data notifying the presence of a translucent body (30) in the common photography area is output.

[0009] A third aspect of the present disclosure is A recognition method executed by a processor (12) for recognizing an external environment of a host vehicle (2) equipped with a visible range camera (40) and an infrared range camera (42) and generating recognition data (Dr), comprising: Obtaining visible range photographing data (Dv) photographed by a visible range camera against the outside world, and infrared range photographing data (Di) photographed by an infrared range camera against the outside world including a common photographing area (Ac) with the visible range camera; and outputting recognition data that indicates the presence of a translucent body (30) in the common shooting area in response to a decrease in the degree of coincidence between the brightness boundaries (Bv, Bi) recognized as edges in each of the visible range shooting data and the infrared range shooting data falling outside the allowable range.

[0010] A fourth aspect of the present disclosure is A recognition program including instructions to be stored in a storage medium (10) and executed by a processor (12) for recognizing the external environment of a host vehicle (2) equipped with a visible range camera (40) and an infrared range camera (42) and generating recognition data (Dr), Obtaining visible range photographing data (Dv) photographed by a visible range camera against the outside world, and infrared range photographing data (Di) photographed by an infrared range camera against the outside world including a common photographing area (Ac) with the visible range camera; and outputting recognition data that indicates the presence of a translucent body (30) in the common shooting area in response to the degree of coincidence between the luminance boundaries (Bv, Bi) recognized as edges in each of the visible range shooting data and the infrared range shooting data decreasing to outside the allowable range.

[0011] In this way, in the first to fourth aspects, infrared range imaging data captured by an infrared range camera of the external world including a common imaging area with the visible range camera is acquired along with visible range imaging data captured by a visible range camera. At this time, if a light-transmitting object is present in the common imaging area, differences occur in the brightness boundaries whose edges are recognized in the visible range imaging data and the infrared range imaging data. Therefore, according to the first to fourth aspects, recognition data output in response to the degree of agreement between the brightness boundaries whose edges are recognized in the visible range imaging data and the infrared range imaging data dropping outside of an acceptable range can accurately indicate the presence of a light-transmitting object in the common imaging area. In other words, it is possible to generate recognition data that ensures recognition accuracy for light-transmitting objects. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram showing the overall configuration of a first embodiment. [Figure 2] FIG. 2 is a plan view showing a state in which a camera is mounted on a host vehicle to which the first embodiment is applied. [Figure 3] FIG. 1 is a block diagram showing a functional configuration of a recognition system according to a first embodiment. [Figure 4] 1 is a flowchart showing a recognition flow according to a first embodiment. [Figure 5] FIG. 4 is a schematic diagram for explaining a recognition flow according to the first embodiment. [Figure 6] FIG. 4 is a schematic diagram for explaining a recognition flow according to the first embodiment. [Figure 7] FIG. 4 is a schematic diagram for explaining a recognition flow according to the first embodiment. [Figure 8] FIG. 4 is a schematic diagram for explaining a recognition flow according to the first embodiment. [Figure 9] FIG. 4 is a schematic diagram for explaining a recognition flow according to the first embodiment. [Figure 10] FIG. 4 is a schematic diagram for explaining a recognition flow according to the first embodiment. [Figure 11] FIG. 4 is a schematic diagram for explaining a recognition flow according to the first embodiment. [Figure 12] FIG. 4 is a schematic diagram for explaining a recognition flow according to the first embodiment. [Figure 13] FIG. 4 is a schematic diagram for explaining a recognition flow according to the first embodiment. [Figure 14] FIG. 4 is a schematic diagram for explaining a recognition flow according to the first embodiment. [Figure 15] FIG. 4 is a schematic diagram for explaining a recognition flow according to the first embodiment. [Figure 16] FIG. 10 is a perspective view showing a state in which a camera is mounted on a host vehicle to which a second embodiment is applied. [Figure 17] FIG. 10 is a schematic diagram for explaining a recognition flow according to a second embodiment. [Figure 18] FIG. 10 is a schematic diagram for explaining a recognition flow according to a second embodiment. [Figure 19] FIG. 10 is a schematic diagram for explaining a recognition flow according to a second embodiment. [Figure 20] 10 is a flowchart showing a recognition flow according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, multiple embodiments of the present disclosure will be described with reference to the drawings. Note that corresponding components in each embodiment are designated by the same reference numerals, and redundant description may be omitted. Furthermore, when only a portion of the configuration is described in each embodiment, the configuration of another previously described embodiment may be applied to the remaining portions of the configuration. Furthermore, in addition to the combinations of configurations explicitly stated in the description of each embodiment, configurations of multiple embodiments may be partially combined together even if not explicitly stated, provided that there is no particular problem with the combination.

[0014] (First embodiment) As shown in Fig. 1, the recognition system 1 of the first embodiment generates recognition data Dr by recognizing the external environment of a host vehicle 2 shown in Fig. 2. The host vehicle 2 can be considered an ego-vehicle from a viewpoint centered on the host vehicle 2. The host vehicle 2 is, for example, an automobile that can travel on a road with an occupant on board. Therefore, directions in the following description are defined based on the host vehicle 2 on a horizontal plane.

[0015] The host vehicle 2 is provided with an autonomous driving mode that is divided into levels according to the degree of manual intervention by the occupant in the dynamic driving task. The autonomous driving mode may be realized by autonomous driving control, such as conditional driving automation, high driving automation, or full driving automation, in which the system performs all dynamic driving tasks when activated. The autonomous driving mode may also be realized by advanced driving assistance control, such as driving assistance or partial driving automation, in which the occupant performs some or all of the dynamic driving tasks. The autonomous driving mode may be realized by either autonomous driving control or advanced driving assistance control, or by a combination of these, or by switching between them.

[0016] The host vehicle 2 is equipped with a sensor system 4. As shown in FIGS. 1 to 3, the sensor system 4 includes a visible range camera 40 and an infrared range camera 42. The field of view Av of the visible range camera 40 and the field of view Ai of the infrared range camera 42 overlap as shown in FIG. 2, thereby forming a common imaging area Ac that can be commonly photographed by both of them. In this embodiment in particular, the common imaging area Ac is set in a range of the field of view Ai that partially overlaps with the field of view Av in both the horizontal and vertical views of the host vehicle 2.

[0017] As shown in FIGS. 1 and 3, the visible range camera 40 includes a visible range imaging element 400 and a visible range imaging circuit 402. The visible range imaging element 400 is a semiconductor element, such as a CMOS, having a plurality of pixels arranged vertically and horizontally. The visible range imaging element 400 receives light in the visible light range from a target 3 (see FIG. 5 described later) present within the field of view Av and captures a visible light image for each pixel. The visible range imaging circuit 402 is a semiconductor chip, such as an image processing circuit, that processes the imaging signal from each pixel of the visible range imaging element 400. The visible range imaging circuit 402 outputs visible range imaging data Dv by converting the luminance value for each pixel according to the intensity of received light of the visible light image from within the field of view Av into two-dimensional data.

[0018] On the other hand, the infrared camera 42 includes an infrared imaging element 420 and an infrared imaging circuit 422. The infrared imaging element 420 is a semiconductor element, such as a microbolometer, having a plurality of pixels arranged vertically and horizontally. The infrared imaging element 420 receives light in the infrared light range, particularly in the far-infrared light range (e.g., a wavelength range of 7 to 14 μm, etc.) from a target 3 (see FIG. 6 described later) present within the field of view Ai, and captures an infrared light image pixel by pixel. The infrared imaging circuit 422 is a semiconductor chip, such as an image processing circuit, that processes the imaging signal from each pixel of the infrared imaging element 420. The infrared imaging circuit 422 converts the luminance value of each pixel according to the received light energy of the infrared light image from within the field of view Ai into two-dimensional data, thereby outputting infrared imaging data Di.

[0019] In this embodiment in particular, the number of pixels for capturing the common imaging area Ac in the infrared imaging element 420 is different from the number of pixels for capturing the same area Ac in the visible imaging element 400 so that the former is smaller than the latter. As a result, between the infrared imaging data Di and the visible imaging data Dv, the former corresponds to low-resolution imaging data with a low resolution for the common imaging area Ac, while the latter corresponds to high-resolution imaging data with a higher resolution than the low-resolution imaging data.

[0020] 1, the recognition system 1 is configured to include at least one dedicated computer. The recognition system 1 is connected to a sensor system 4 via at least one of, for example, a LAN (Local Area Network) line, a wire harness, an internal bus, or a wireless communication line. When the recognition system 1 is configured with multiple dedicated computers, the connections between these dedicated computers are similar.

[0021] The dedicated computer constituting the recognition system 1 may be a driving control ECU (Electronic Control Unit) that controls the driving of the host vehicle 2. The dedicated computer constituting the recognition system 1 may be a navigation ECU that navigates the driving route of the host vehicle 2. The dedicated computer constituting the recognition system 1 may be a locator ECU that estimates the self-state quantity of the host vehicle 2. The dedicated computer constituting the recognition system 1 may be an actuator ECU that controls the driving actuator of the host vehicle 2. The dedicated computer constituting the recognition system 1 may be an HCU (Human Machine Interface (HMI) Control Unit) that controls the presentation of information in the host vehicle 2. The dedicated computer constituting the recognition system 1 may be a computer other than the host vehicle 2 that constitutes an external center or mobile terminal that can communicate via the communication system of the host vehicle 2.

[0022] The dedicated computer constituting the recognition system 1 has at least one memory 10 and one processor 12. The memory 10 is at least one type of non-transitory tangible storage medium, such as a semiconductor memory, a magnetic medium, or an optical medium, that non-temporarily stores computer-readable programs, data, and the like. Here, "storage" may refer to accumulation in which data is retained even when the host vehicle 2 is powered off, or may refer to temporary storage in which data is erased when the host vehicle 2 is powered off. The processor 12 includes at least one type of core, such as a central processing unit (CPU), a graphics processing unit (GPU), a reduced instruction set computer (RISC)-CPU, a data flow processor (DFP), or a graph streaming processor (GSP).

[0023] In the recognition system 1, the processor 12 executes a plurality of instructions included in a recognition program stored in the memory 10 in order to recognize the external environment of the host vehicle 2 and generate recognition data Dr. As a result, the recognition system 1 constructs a plurality of functional blocks for recognizing the external environment of the host vehicle 2 and generating recognition data Dr. The plurality of functional blocks constructed in the recognition system 1 include a data acquisition block 100 and a recognition processing block 120, as shown in FIG.

[0024] The recognition method in which the recognition system 1 recognizes the external environment of the host vehicle 2 and generates the recognition data Dr through the cooperation of these blocks 100 and 120 is executed according to the recognition flow shown in Figure 4. This recognition flow is executed repeatedly while the host vehicle 2 is running. Note that each "S" in this recognition flow represents a step executed by multiple commands included in the recognition program.

[0025] In S10, the data acquisition block 100 acquires data individually captured by the corresponding cameras 40, 42 within each field of view Av, Ai including a common imaging area Ac in the external world as visible range imaging data Dv and infrared range imaging data Di, respectively, as shown in Figures 5 and 6. At this time, it is preferable that the timing of capturing each imaging data Dv, Di is substantially synchronized.

[0026] As shown in Fig. 4, the recognition flow proceeds from S10 to S20. In S20, the data acquisition block 100 extracts the luminance values ​​of each pixel included in the common pixel areas Pcv and Pci, which are obtained by photographing the common photographing area Ac as shown in Figs. 5 and 6, from the acquired photographing data Dv and Di.

[0027] As shown in Fig. 4, the recognition flow proceeds from S20 to S30. In S30, the recognition processing block 120 matches the common pixel area Pcv of the visible range imaging data Dv, which is the high-resolution imaging data, with the common pixel area Pci of the infrared range imaging data Di, which is the low-resolution imaging data, as shown in Fig. 6, by downsampling, as shown in Fig. 7. At this time, the number of pixels in the common pixel area Pcv of the visible range imaging data Dv is made to substantially match the number of pixels in the common pixel area Pci of the infrared range imaging data Di. Therefore, it is preferable that the luminance values ​​of each pixel included in the common pixel area Pcv of the visible range imaging data Dv be converted to values ​​obtained by internal interpolation according to the position coordinates of each pixel included in the common pixel area Pci of the infrared range imaging data Di.

[0028] As shown in Fig. 4, the recognition flow proceeds from S30 to S40. In S40, the recognition processing block 120 binarizes (as shown in the examples of Figs. 8 and 9) or normalizes (not shown) the luminance values ​​of each pixel included in the common pixel areas Pcv and Pci of the matched photographic data Dv and Di by, for example, edge detection processing using an edge filter. As a result, the recognition processing block 120 in S40 recognizes, as edges, pixel groups that provide luminance boundaries Bv and Bi, shown by cross-hatching in Figs. 8 and 9, in the common pixel areas Pcv and Pci of the photographic data Dv and Di.

[0029] As shown in Fig. 4, the recognition flow proceeds from S40 to S50. In S50, the recognition processing block 120 determines whether the degree of match between the edge-recognized brightness boundaries Bv and Bi has fallen outside an allowable range. At this time, the allowable range is set taking into consideration the range of variation in the degree of match that appears between the brightness boundaries Bv and Bi when, among the targets 3 present in the external world of the host vehicle 2, a light-transmitting body 30 that can transmit visible light to the visible range camera 40 side is not present within the common photographing area Ac (see Fig. 2), as shown in Figs. 10 and 13.

[0030] Therefore, the translucent body 30 is defined as a target 3 made of glass, transparent resin, or the like, having a high transmittance of, for example, 70% or more, preferably 90% or more, to visible light to which the visible range camera 40 is sensitive. Furthermore, the translucent body 30 of this embodiment may have a lower transmittance to infrared light to which the infrared range camera 42 is sensitive, for example, 10% or less, preferably 1% or less, than the visible light to which the visible range camera 40 is sensitive. Examples of such a translucent body 30 include glass windows on the wall of a building or show windows. For the above reasons, it is expected that the degree of coincidence between the luminance boundaries Bv and Bi will be outside the acceptable range in a scene in which the translucent body 30 is present within the common shooting area Ac.

[0031] A specific assumed scenario is one in which at least one stationary object 32, among targets 3 present in the external world of the host vehicle 2, is captured by the visible-range camera 40 through a translucent body 30, and is included within the field of view Av of the camera 40, including the common capturing area Ac, as shown in FIGS. 10 and 11. In this case, as shown in FIGS. 10 and 12, the stationary object 32 present within the field of view Av is hidden by the translucent body 30 in front, making it difficult to capture. Consequently, in the common pixel area Pcv of the visible-range capturing data Dv, the pixel group of the luminance boundary Bv representing the outline of the stationary object 32 is difficult to recognize as an edge in the pixel group matching the pixel group in the common pixel area Pci of the infrared-range capturing data Di. Thus, in the assumed scenario of FIG. 10, the degree of match between the luminance boundaries Bv and Bi is outside the acceptable range, resulting in a positive determination in S50.

[0032] Another assumed scenario is when, among targets 3 present in the external world of the host vehicle 2, a wall 34 that is photographed by the visible range camera 40 through a translucent body 30 as shown in Fig. 13 is present spreading within the field of view Av of the camera 40, including the common photographing area Ac. In this case, with the infrared range camera 42, as shown in Figs. 13 and 15, the wall 34 spreading within the field of view Av is hidden by the translucent body 30 in front, making it difficult to photograph, and essentially only the translucent body 30 in front becomes easy to photograph. In this case, the outer contour of the wall 34 spreading within the field of view Av is difficult to recognize as an edge as a brightness boundary Bv, Bi, both in the common pixel area Pcv of the visible range photographing data Dv shown in Fig. 14 and in the common pixel area Pci of the infrared range photographing data Di shown in Fig. 15.

[0033] However, in this other assumed scene, if the temperature rises in the portion of the foreground translucent body 30 that receives radiant heat from the host vehicle 2, the temperature rise portion 300, which is difficult to capture with the visible range camera 40 as shown in Fig. 14, becomes easy to capture with the infrared range camera 42 as shown in Fig. 15. As a result, in the common pixel area Pci of the infrared range imaging data Di, the pixel group of the luminance boundary Bi that represents the outline of the temperature rise portion 300 becomes difficult to recognize as an edge in the pixel group that matches in the common pixel area Pcv of the visible range imaging data Dv. As a result, in the assumed scene of Fig. 13, the degree of match between the luminance boundaries Bv and Bi is outside the permissible range, and a positive determination is made in S50.

[0034] 4, if the degree of match between the luminance boundaries Bv and Bi is within the allowable range, the recognition flow proceeds to S60 in response to a negative determination in S50. In S60, the recognition processing block 120 generates and outputs recognition data Dr, which is primarily composed of visible range imaging data Dv and represents the luminance values ​​of all pixels, including those in the common pixel area Pcv. In this case, the recognition data Dr may also include infrared range imaging data Di, which represents the luminance values ​​of all pixels, including those in the common pixel area Pci.

[0035] On the other hand, if the degree of match between the luminance boundaries Bv, Bi has fallen outside the allowable range, the recognition flow proceeds to S70 in response to a positive determination in S50. In S70, the recognition processing block 120 generates and outputs recognition data Dr to notify the presence of the light-transmitting body 30 in the common imaging area Ac. At this time, the recognition data Dr may include visible range imaging data Dv representing the luminance values ​​of all pixels including the pixels in the common pixel area Pcv. In addition, the recognition data Dr may also include infrared range imaging data Di representing the luminance values ​​of all pixels including the pixels in the common pixel area Pci.

[0036] Here, the output in S60, 70 may be storing the recognition data Dr in the memory 10. The output in S60, 70 may be providing the recognition data Dr to, for example, a driving control ECU. The output of data in S60, 70 may be transmitting the recognition data Dr to an external center or a mobile terminal via the communication system of the host vehicle 2. Note that when either S60 or S70, which is selectively executed in the current recognition flow, is completed, the current recognition flow ends.

[0037] (Action and effect) The effects of the first embodiment described above will be explained below.

[0038] In the first embodiment, together with visible range imaging data Dv captured by the visible range camera 40 against the outside world, infrared range imaging data Di captured by the infrared range camera 42 against the outside world including a common imaging area Ac with the visible range camera 40 are acquired. At this time, if a light-transmitting body 30 is present in the common imaging area Ac, differences occur in the luminance boundaries Bv, Bi whose edges are recognized in the visible range imaging data Dv and the infrared range imaging data Di. Therefore, according to the first embodiment, the recognition data Dr output in response to the degree of coincidence between the luminance boundaries Bv, Bi whose edges are recognized in the visible range imaging data Dv and the infrared range imaging data Di respectively decreasing to outside the allowable range can accurately notify the presence of the light-transmitting body 30 in the common imaging area Ac. In other words, it is possible to generate recognition data Dr that ensures recognition accuracy for the light-transmitting body 30.

[0039] According to the first embodiment, a scene is assumed in which a stationary object 32 is photographed by the visible range camera 40 through a light-transmitting body 30, and infrared range imaging data Di is acquired by the infrared range camera 42, relative to the external world contained within a field of view Av including the common imaging area Ac of the visible range camera 40. In this assumed scene, a luminance boundary Bv representing the outline of the stationary object 32 in the visible range imaging data Dv photographed by the visible range camera 40 is difficult to recognize as an edge as a luminance boundary Bi in the infrared range imaging data Di. Accordingly, in response to a drop in the degree of agreement between the luminance boundaries Bv and Bi of the visible range imaging data Dv and the infrared range imaging data Di to an extent outside the allowable range due to the presence of a light-transmitting body 30, recognition data Dr that accurately notifies the presence of the light-transmitting body 30 can be output in a timely manner. This makes it possible to generate recognition data Dr with improved recognition accuracy for the light-transmitting body 30.

[0040] According to the first embodiment, a scenario is assumed in which infrared imaging data Di is acquired by the infrared camera 42, depicting a wall 34 captured by the visible camera 40 through the light-transmitting body 30, against the outside world extending within a field of view Av including the common imaging area Ac of the visible camera 40. In this scenario, a luminance boundary Bi representing the contour of a temperature-raised portion 300 in the light-transmitting body 30 due to radiant heat from the host vehicle 2 in the infrared imaging data Di is difficult to recognize as an edge in the luminance boundary Bv in the visible imaging data Dv captured by the visible camera 40. Accordingly, in response to a drop in the degree of agreement between the luminance boundaries Bv and Bi of the visible imaging data Dv and the infrared imaging data Di to an unacceptable level due to the presence of the light-transmitting body 30, recognition data Dr accurately reporting the presence of the light-transmitting body 30 can be output in a timely manner. This makes it possible to generate recognition data Dr with improved recognition accuracy for the light-transmitting body 30.

[0041] In the first embodiment, visible range imaging data Dv and infrared range imaging data Di are acquired by separately capturing images of the common imaging area Ac using a visible range camera 40 and an infrared range camera 42, which have different pixel counts. Therefore, of the imaging data Dv and Di in the first embodiment, the infrared range imaging data Di, which is low-resolution imaging data for the common imaging area Ac, is matched by downsampling with the visible range imaging data Dv, which is high-resolution imaging data for the same area Ac. This prevents pixel groups on the brightness boundaries Bi and Bv, which are actually aligned, from being erroneously determined to be outside the allowable range due to differences in resolution. Therefore, it becomes possible to generate recognition data Dr that ensures recognition accuracy for the light-transmitting body 30.

[0042] Second Embodiment The second embodiment is a modification of the first embodiment. As shown in Fig. 16 , in the second embodiment, an autonomously traveling robot equipped with a visible range camera 40 and an infrared range camera 42, which enables baggage transport or information collection by autonomous traveling or remote traveling, serves as the host vehicle 2. The translucent body 30 assumed in this second embodiment may be, for example, a glass window or a show window on the wall of a building, or a glass partition or a transparent resin partition inside the building.

[0043] In the second embodiment, in addition to the scenes shown in Figures 10 and 13 described in the first embodiment, another scene shown in Figure 17 is assumed as a scene in which the degree of coincidence between the brightness boundaries Bv and Bi determined by S50 of the recognition flow falls outside the allowable range. Note that, hereinafter, the scene in Figure 10 will be referred to as the first assumed scene, the scene in Figure 13 as the second assumed scene, and the scene in Figure 17 as the third assumed scene.

[0044] Specifically, the third assumed scene is a case where, among the targets 3 present in the external world of the host vehicle 2, at least one heat-generating object 37 is included within the field of view Av of the camera 40, including the common imaging area Ac, and is difficult to capture because it is hidden by a shading sheet 36, such as a fabric curtain, from the visible range camera 40, as shown in FIGS. 17 and 18 . In particular, the third assumed scene is a case where the shading sheet 36 captured by the visible range camera 40 is present in the field of view Av of the camera 40, including the common imaging area Ac. In this third assumed scene, the heat-generating object 37, whose temperature is rising, is easily captured by the infrared range camera 42, as shown in FIGS. 17 and 19 . As a result, the brightness boundary Bv is difficult to recognize as an edge in the pixel group matching the brightness boundary Bi in the common pixel area Pcv of the visible range imaging data Dv shown in FIG. 18, compared to the pixel group of the brightness boundary Bi that represents the outline of the heat-generating object 37 in the common pixel area Pci of the infrared range imaging data Di shown in FIG. 19 . From the above, even in the third assumed scene in which the light-transmitting body 30 does not exist in the common shooting area Ac, there is a concern that the degree of coincidence between the luminance boundaries Bv and Bi may be determined to be outside the allowable range in S50 and thus be positive.

[0045] Therefore, in the recognition flow of the second embodiment, in order to distinguish between the first to third assumed scenes, if a positive determination is made in S50 as shown in Fig. 20, the flow proceeds to S250. In S250, the recognition processing block 120 determines whether a visible image of an object, such as a stationary object 32, as shown in Fig. 11 is recognized in the common pixel area Pcv obtained by capturing the common capturing area Ac of the visible range capturing data Dv. As a result, in response to a positive determination in S250, the recognition flow proceeds to S270. In S270, the recognition processing block 120 generates and outputs recognition data Dr to indicate the presence of a light-transmitting body 30 in the common capturing area Ac under the first assumed scene shown in Fig. 10.

[0046] On the other hand, as shown in Fig. 20, if a negative determination is made in S250, the recognition flow proceeds to S251. In S251, the recognition processing block 120 determines whether an infrared image of the temperature-rising portion 300 due to radiant heat from the host vehicle 2 is recognized in the common pixel area Pci obtained by capturing the common capturing area Ac of the infrared range capturing data Di, in accordance with Fig. 15. As a result, in response to a positive determination in S251, the recognition flow proceeds to S271. In S271, the recognition processing block 120 generates and outputs recognition data Dr to notify the presence of the light-transmitting body 30 in the common capturing area Ac under the second assumed scene shown in Fig. 13.

[0047] 20, if a negative determination is made in S251, the recognition flow proceeds to S60. In this case, in S60 according to the second embodiment, recognition data Dr, mainly based on the visible range image data Dv, is generated and output under the third assumed scene. Note that the current recognition flow ends when any of S270, S271, and S60, which is selectively executed in the current recognition flow, is completed.

[0048] As described above, the second embodiment focuses on a case where not only the degree of coincidence between the luminance boundaries Bv, Bi in the respective shooting data Dv, Di drops outside the permissible range, but also a visible image of an object is recognized in the common pixel area Pcv that captures the common shooting area Ac of the visible range shooting data Dv. In this case, in response to the recognition of the visible image in the common pixel area Pcv, it is possible to realize a notification that accurately targets a scene in which a light-transmitting body 30 is present in the common shooting area Ac (specifically, the first assumed scene described above). In other words, it becomes possible to generate recognition data Dr that ensures high recognition accuracy for the light-transmitting body 30.

[0049] Furthermore, in the second embodiment, attention is paid not only to the case where the degree of coincidence between the brightness boundaries Bv, Bi in the respective image data Dv, Di falls outside the allowable range, but also to the case where an infrared image due to radiant heat from the host vehicle 2 is recognized in the common pixel area Pci that captures the common image area Ac of the infrared range image data Di. In this case, in response to the recognition of the infrared image in the common pixel area Pci, it is possible to realize a notification that accurately targets a scene in which a light-transmitting body 30 is present in the common image area Ac (specifically, the second assumed scene described above). In other words, it is possible to generate recognition data Dr that ensures high recognition accuracy for the light-transmitting body 30.

[0050] (Other embodiments) Although multiple embodiments have been described above, the present disclosure should not be construed as being limited to those embodiments, and can be applied to various embodiments and combinations within the scope that does not deviate from the gist of the present disclosure.

[0051] In the modifications of the first and second embodiments, the dedicated computer constituting the recognition system 1 may have at least one of a digital circuit and an analog circuit as a processor. Here, the digital circuit is at least one of an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a system on a chip (SOC), a programmable gate array (PGA), and a complex programmable logic device (CPLD). Such a digital circuit may also have a memory that stores a program.

[0052] In the modified examples of the first and second embodiments, the number of pixels in the infrared range image sensor 420 that captures the common imaging area Ac may be different from the number of pixels in the visible range image sensor 400 that captures the same area Ac, so that the number of pixels in the infrared range image sensor 420 is smaller than the number of pixels in the visible range image sensor 400 that captures the same area Ac. In this case, the infrared range image data Di and the visible range image data Dv correspond to low-resolution image data with a lower resolution than the common imaging area Ac, while the visible range image data Dv corresponds to high-resolution image data with a higher resolution than the low-resolution image data. Furthermore, in this case, in S30, the common pixel area Pci of the infrared range image data Di, which is the high-resolution image data, may be matched by downsampling to the common pixel area Pcv of the visible range image data Dv, which is the low-resolution image data.

[0053] In the modification of the first embodiment, the host vehicle 2 to which the recognition system 1 is applied may be, for example, an autonomous robot capable of autonomously or remotely traveling to transport luggage or collect information, as in the second embodiment. In this case, the translucent body 30 may be, for example, a glass window or a show window on the wall of a building, as well as a glass partition or a transparent resin partition inside the building, as in the second embodiment.

[0054] In addition to the forms described so far, the above-mentioned embodiments and variations may be implemented in the form of a processing circuit (e.g., a processing ECU, etc.) or a semiconductor device (e.g., a semiconductor chip, etc.) as a recognition device configured to be mountable on a host vehicle 2 and having at least one processor 12 and one memory 10.

[0055] (Additional remarks) This specification discloses the following technical ideas and their combinations. Note that the reference symbols in parentheses in the appended remarks indicate the correspondence with the specific means described in the above detailed embodiments, and do not limit the technical scope of the present disclosure.

[0056] (Technical thought 1) A recognition system having a processor (12), which recognizes the external environment of a host vehicle (2) equipped with a visible range camera (40) and an infrared range camera (42) and generates recognition data (Dr), The processor: Obtaining visible range photography data (Dv) photographed by the visible range camera with respect to the external world, and infrared range photography data (Di) photographed by the infrared range camera with respect to the external world including a common photography area (Ac) with the visible range camera; and outputting the recognition data notifying the presence of a translucent body (30) in the common shooting area in response to a decrease in the degree of coincidence between brightness boundaries (Bv, Bi) whose edges are recognized in each of the visible range shooting data and the infrared range shooting data falling outside an acceptable range.

[0057] (Technical thought 2) The acquisition of the visible range photography data and the infrared range photography data includes: The recognition system described in Technical Idea 1 includes acquiring the infrared range imaging data captured by the infrared range camera of a stationary object (32) photographed by the visible range camera through the transparent body against the external world included within the field of view (Av) of the visible range camera, including the common imaging area.

[0058] (Technical Thought 3) The acquisition of the visible range photography data and the infrared range photography data includes: The recognition system according to Technical Idea 1 or 2 includes acquiring the infrared range photographed data of the wall (34) photographed by the visible range camera through the transparent body against the outside world extending within the field of view (Av) of the visible range camera, including the common photographing area, by the infrared range camera.

[0059] (Technical Thought 4) The acquisition of the visible range photography data and the infrared range photography data includes: acquiring the visible range imaging data and the infrared range imaging data, which are individually photographed by the visible range camera and the infrared range camera, each having a different number of pixels for photographing the common imaging area; The output of the recognition data is A recognition system according to any one of Technical Ideas 1 to 3, which includes matching low-resolution photography data, which is one of the visible range photography data and the infrared range photography data and has a low resolution for the common photography area, with high-resolution photography data, which is the other of the visible range photography data and the infrared range photography data and has a higher resolution than the low-resolution photography data for the common photography area, by downsampling, to determine the degree of match.

[0060] (Technical Thought 5) The acquisition of the visible range photography data and the infrared range photography data includes: The recognition system according to Technical Idea 4 includes acquiring the infrared range imaging data that becomes the low-resolution imaging data.

[0061] (Technical Thought 6) The output of the recognition data is The recognition system according to any one of technical ideas 1 to 5 includes outputting the recognition data notifying the presence of the translucent body in the common shooting area in response to the degree of coincidence decreasing to outside the allowable range and a visible image of the object being recognized in a common pixel area (Pcv) that photographed the common shooting area of ​​the visible range shooting data.

[0062] (Technical Thought 7) The output of the recognition data is The recognition system described in any one of technical ideas 1 to 6 includes outputting the recognition data notifying the presence of the translucent body in the common shooting area in response to the degree of coincidence decreasing to outside the allowable range and an infrared image caused by radiant heat from the host vehicle being recognized in a common pixel area (Pci) that photographed the common shooting area of ​​the infrared range shooting data.

[0063] The above-mentioned technical concepts 1 to 7 may be understood as the respective technical concepts of an apparatus, a method, and a program. [Explanation of symbols]

[0064] 1: Recognition system, 2: Host vehicle, 10: Memory, 12: Processor, 30: Transparent body, 32: Stationary object, 34: Wall, 40: Visible range camera, 42: Infrared range camera, Ac: Common shooting area, Av, Ai: Field of view, Bv, Bi: Brightness boundary, Di: Infrared range shooting data, Dr: Recognition data, Dv: Visible range shooting data, Pcv, Pci: Common pixel area

Claims

1. A recognition system having a processor (12) and configured to recognize the external environment of a host vehicle (2) equipped with a visible range camera (40) and an infrared range camera (42) and generate recognition data (Dr), The processor: Obtaining visible range photography data (Dv) photographed by the visible range camera with respect to the external world, and infrared range photography data (Di) photographed by the infrared range camera with respect to the external world including a common photography area (Ac) with the visible range camera; and outputting the recognition data notifying the presence of a translucent body (30) in the common shooting area in response to a decrease in the degree of coincidence between brightness boundaries (Bv, Bi) whose edges are recognized in each of the visible range shooting data and the infrared range shooting data falling outside an acceptable range.

2. The acquisition of the visible range photography data and the infrared range photography data includes:

2. The recognition system according to claim 1, further comprising: acquiring the infrared range imaging data captured by the infrared range camera of a stationary object (32) photographed by the visible range camera through the transparent body against the external world included within a field of view (Av) of the visible range camera including the common imaging area.

3. The acquisition of the visible range photography data and the infrared range photography data includes:

2. The recognition system according to claim 1, further comprising: acquiring the infrared range imaging data captured by the infrared range camera of the wall (34) photographed by the visible range camera through the transparent body, the wall (34) being photographed by the visible range camera against the outside world extending within a field of view (Av) of the visible range camera including the common imaging area.

4. The acquisition of the visible range photography data and the infrared range photography data includes: acquiring the visible range imaging data and the infrared range imaging data, which are individually photographed by the visible range camera and the infrared range camera, each having a different number of pixels for photographing the common imaging area; The output of the recognition data is The recognition system according to any one of claims 1 to 3, further comprising: matching, by downsampling, low-resolution photography data, which is one of the visible range photography data and the infrared range photography data and has a low resolution for the common photography area, with high-resolution photography data, which is the other of the visible range photography data and the infrared range photography data and has a higher resolution than the low-resolution photography data for the common photography area.

5. The acquisition of the visible range photography data and the infrared range photography data includes: The recognition system according to claim 4 , further comprising acquiring the infrared range imaging data that becomes the low-resolution imaging data.

6. The output of the recognition data is The recognition system according to any one of claims 1 to 3, further comprising: outputting the recognition data notifying the presence of the light-transmitting body in the common shooting area in response to the degree of coincidence decreasing to outside the allowable range and a visible image of an object being recognized in a common pixel area (PCV) that photographs the common shooting area of ​​the visible range shooting data.

7. The output of the recognition data is The recognition system according to any one of claims 1 to 3, further comprising: outputting the recognition data notifying the presence of the translucent body in the common photographing area in response to the degree of coincidence decreasing to outside the allowable range and an infrared image due to radiant heat from the host vehicle being recognized in a common pixel area (Pci) that photographed the common photographing area of ​​the infrared photographing data.

8. A recognition device configured to be mountable on a host vehicle (2) together with a visible range camera (40) and an infrared range camera (42), the recognition device having a processor (12) for recognizing an external environment of the host vehicle and generating recognition data (Dr), The processor: Obtaining visible range photography data (Dv) photographed by the visible range camera with respect to the external world, and infrared range photography data (Di) photographed by the infrared range camera with respect to the external world including a common photography area (Ac) with the visible range camera; and outputting the recognition data notifying the presence of a translucent body (30) in the common shooting area in response to a decrease in the degree of coincidence between brightness boundaries (Bv, Bi) whose edges are recognized in each of the visible range shooting data and the infrared range shooting data falling outside an acceptable range.

9. A recognition method executed by a processor (12) for recognizing an external environment of a host vehicle (2) equipped with a visible range camera (40) and an infrared range camera (42) and generating recognition data (Dr), comprising: Obtaining visible range photography data (Dv) photographed by the visible range camera with respect to the external world, and infrared range photography data (Di) photographed by the infrared range camera with respect to the external world including a common photography area (Ac) with the visible range camera; and outputting the recognition data notifying the presence of a translucent body (30) in the common shooting area in response to a decrease in the degree of coincidence between brightness boundaries (Bv, Bi) whose edges are recognized in each of the visible range shooting data and the infrared range shooting data falling outside an acceptable range.

10. A recognition program is stored in a storage medium (10) and includes instructions to be executed by a processor (12) for recognizing the external environment of a host vehicle (2) equipped with a visible range camera (40) and an infrared range camera (42) and generating recognition data (Dr), the recognition program comprising: Obtaining visible range photography data (Dv) photographed by the visible range camera with respect to the external world, and infrared range photography data (Di) photographed by the infrared range camera with respect to the external world including a common photography area (Ac) with the visible range camera; and outputting the recognition data that indicates the presence of a translucent body (30) in the common shooting area in response to a decrease in the degree of coincidence between brightness boundaries (Bv, Bi) that are edge-recognized in each of the visible range shooting data and the infrared range shooting data falling outside an acceptable range.

Images