Image generation device for learning, method for generating images for learning, and program

The learning image generation device generates fisheye images with virtual road markings to train models for trailer state estimation, addressing the need for actual image capture and reducing data preparation load in existing technologies.

JP7948285B2Active Publication Date: 2026-11-11TOYOTA JIDOSHA KK
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2024093676
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2024-06-10
Publication Date
2026-11-11
Estimated Expiration
2044-06-10

AI Technical Summary

Technical Problem

Existing technologies for detecting the connected state between a tow vehicle and a towed vehicle using image processing do not utilize learning models and require capturing fisheye images with road surface paint, which increases the data preparation load.

Method used

A learning image generation device and method that generates learning fisheye images by converting fisheye images to planar normalized images, adding virtual road surface paint, and then reversing the conversion to include road markings, allowing training without actual capture of fisheye images with road markings.

Benefits of technology

Enables training of models to estimate trailer states using fisheye images with virtual road markings, reducing the need for actual image capture and minimizing data preparation load.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007948285000001
    Figure 0007948285000001
  • Figure 0007948285000002
    Figure 0007948285000002
  • Figure 0007948285000003
    Figure 0007948285000003
Patent Text Reader

Abstract

To make it possible, without requiring actually photographing a fisheye image containing road surface paint, to learn a model by using the fisheye image as learning data.SOLUTION: A learning image generation device 1 which generates a learning fisheye image used for learning of a model used for estimation of a condition of a trailer on the basis of a fisheye image photographed by a camera mounted on a vehicle towing the trailer through a towing bar includes: an acquisition unit 3A which acquires a learning fisheye original image photographed by a learning camera mounted on a learning vehicle towing the learning trailer through a learning towing bar; an image conversion unit 3B which generates a plane normal conversion image by applying plane normal image conversion to the learning fish eye original image; an adding unit 3C which adds virtual road surface paint to the plane normal conversion image; and a learning image generation unit 3D which generates a learning fisheye image by applying a conversion opposite to the plane normal conversion to the plane normal conversion (plane image with the virtual road surface paint) after addition of the virtual paint.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to a learning image generation device, a learning image generation method, and a program.

Background Art

[0002] Patent Document 1 describes a tow vehicle in which an imaging unit having a wide-angle lens or a fisheye lens is provided on a wall portion below a rear hatch. Further, Patent Document 1 states that captured image data captured by the imaging unit can be used to detect the connected state (e.g., connection angle, presence or absence of connection, etc.) between the tow vehicle and the towed vehicle.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problem to be Solved by the Invention

[0004] By the way, although Patent Document 1 describes that the detection of the connected state between the tow vehicle and the towed vehicle is performed by image processing, it does not describe whether a model requiring learning is used for such image processing. When a model is used to detect the connected state between the tow vehicle and the towed vehicle, it is necessary to suppress an increase in the load of preparing learning data used for learning the model.

[0005] In view of the above points, the present disclosure aims to provide a learning image generation device, a learning image generation method, and a program that can enable learning of a model using fisheye images including road surface paint as learning data, without the need to actually capture fisheye images including road surface paint with a learning camera. is as follows.

Means for Solving the Problem

[0006] (1) One aspect of the present disclosure is a learning image generation device that generates a learning fisheye image as learning data used to train a model used to train a model that estimates the state of a trailer based on a fisheye image taken by a camera mounted on a vehicle that tows a trailer via a tow bar, the learning image generation device comprising: an acquisition unit that acquires a learning fisheye image taken by a learning camera mounted on a learning vehicle that tows a learning trailer via a learning tow bar; an image conversion unit that generates a planar normalized image converted image by performing a planar normalization conversion, which is a conversion from a fisheye image to a planar image, on the learning fisheye image acquired by the acquisition unit; an addition unit that adds a virtual road surface paint to the planar normalized image converted image generated by the image conversion unit; and a learning image generation unit that generates the learning fisheye image by performing a conversion in the reverse of the planar normalization conversion performed by the image conversion unit on the planar image with virtual road surface paint, which is the planar normalized image converted image after the virtual road surface paint has been added by the addition unit.

[0007] (2) One aspect of the present disclosure is a learning image generation step in which a learning image generation device generates a learning fisheye image as learning data used to train a model used to train a model used to estimate the state of the trailer based on a fisheye image captured by a camera mounted on a vehicle that tows the trailer via a tow bar; an acquisition step in which the learning image generation device acquires a learning fisheye image captured by a learning camera mounted on a learning vehicle that tows the learning trailer via a learning tow bar; and a learning image generation device converts the learning fisheye image acquired in the acquisition step to a planar image from a fisheye image. The learning image generation method comprises an image conversion step of generating a planar normalized image image by performing a planar normalized image conversion, which is a type of transformation, and an additional step of the learning image generation device adding virtual road surface paint to the planar normalized image image generated in the image conversion step, wherein in the learning image generation step, the learning fisheye image is generated by performing a transformation in the reverse of the planar normalized image conversion performed in the image conversion step on the planar image with virtual road surface paint, which is the planar normalized image image after the virtual road surface paint has been added in the additional step.

[0008] (3) One aspect of the present disclosure is a program for causing a processor to perform the following steps: a training image generation step of generating training fisheye images as training data used for training a model used for estimating the state of a trailer based on fisheye images captured by a camera mounted on a vehicle towing a trailer via a tow bar; an acquisition step of acquiring a training fisheye source image captured by a training camera mounted on a training vehicle towing a training trailer via a training tow bar; an image conversion step of generating a planar normalized image by performing a planar normalization conversion, which is a conversion from a fisheye image to a planar image, on the training fisheye source image acquired in the acquisition step; and an addition step of adding virtual road surface paint to the planar normalized image generated in the image conversion step, wherein in the training image generation step, the training fisheye image is generated by performing a conversion in the reverse of the planar normalization conversion performed in the image conversion step on the planar image with virtual road surface paint, which is the planar normalized image after the virtual road surface paint has been added in the addition step. [Effects of the Invention]

[0009] According to this disclosure, it is possible to train a model using fisheye images containing road paint as training data, without actually needing to capture fisheye images containing road paint with a training camera. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows an example of a learning image generation device 1 according to the first embodiment. [Figure 2] This diagram shows an example of a learning vehicle L1 equipped with a learning camera L11. [Figure 3] This figure shows an example of a planar normalized image IM2, etc., generated by the image conversion unit 3B. [Figure 4] This figure shows an example of a training fisheye image IM4 generated by the training image generation unit 3D. [Figure 5] This figure illustrates the state of trailer R2 (hitch angle Φ of trailer R2) estimated using a model trained with training data that uses a training fisheye image IM4 containing virtual road surface paint VP (lane markings) generated by the training image generation unit 3D of the training image generation device 1. [Figure 6] This is a flowchart illustrating an example of the processing performed in the learning image generation device 1 of the first embodiment. [Modes for carrying out the invention]

[0011] Hereinafter, embodiments of the learning image generation apparatus, learning image generation method, and program of this disclosure will be described with reference to the drawings.

[0012] <First Embodiment> Figure 1 shows an example of a learning image generation device 1 according to the first embodiment. In the example shown in Figure 1, the learning image generation device 1 is comprised of a microcomputer that includes a communication interface (I / F) 11, a memory 12, and a processor 13. The communication interface 11 has an interface circuit for connecting the learning image generation device 1 to an external device (for example, a storage device (not shown) that stores the learning fisheye source image IM1 (see Figure 2(B)) captured by the learning camera L11 (see Figure 2(A))).

[0013] Figure 2 shows an example of a learning vehicle L1 equipped with a learning camera L11. Specifically, Figure 2(A) shows an example of a learning vehicle L1 equipped with a learning camera L11, and Figure 2(B) shows an example of a learning fisheye image IM1 captured by the learning camera L11. In the example shown in Figure 2, a learning camera L11 is positioned at the rear end L1R of the learning vehicle L1. The learning camera L11 photographs the area behind the learning vehicle L1 (the right side in Figure 2(A)). As shown in Figure 2(A), the learning vehicle L1 tows a learning trailer L2 via a learning tow bar L3. The learning trailer L2 is rotatably connected to the learning vehicle L1 around a hitch ball (not shown). As shown in Figure 2(B), the learning fisheye source image IM1 includes a portion of the learning vehicle L1, the learning trailer L2, and the learning tow bar L3. On the other hand, in the example shown in Figure 2, as shown in Figure 2(A), there are no lane markings painted on the road surface while the learning vehicle L1 and learning trailer L2 are traveling, and as shown in Figure 2(B), the lane markings painted on the road surface are not included in the learning fisheye source image IM1.

[0014] In the example shown in Figure 1, memory 12 stores programs and various data used in processing performed by processor 13. Processor 13 has the functions of an acquisition unit 3A, an image conversion unit 3B, an addition unit 3C, and a learning image generation unit 3D. The acquisition unit 3A acquires the learning fisheye image IM1 captured by the learning camera L11. Specifically, the acquisition unit 3A acquires the learning fisheye image IM1 that does not include road markings or other road surface paint, for example, as shown in the example in Figure 2(B). The image conversion unit 3B generates a planar normalized image IM2 (see Figure 3(A)) by performing a planar normalization conversion, which is a conversion from a fisheye image to a planar image, on the training fisheye source image IM1 acquired by the acquisition unit 3A. The additional unit 3C generates a planar image IM3 (see Figure 3(C)) with virtual road surface paint VP (lane markings) (see Figure 3(B)) by adding virtual road surface paint to the planar normalized image IM2 generated by the image conversion unit 3B.

[0015] FIG. 3 is a diagram showing an example of a planar normal image conversion image IM2 and the like generated by an image conversion unit 3B. Specifically, FIG. 3(A) shows an example of the planar normal image conversion image IM2 generated by the image conversion unit 3B, FIG. 3(B) shows an example of virtual road surface paint VP (lane marking) added by an adding unit 3C to the planar normal image conversion image IM2 shown in FIG. 3(A), and FIG. 3(C) shows an example of a planar image IM3 with virtual road surface paint, which is a planar normal image conversion image after the virtual road surface paint VP (lane marking) is added by the adding unit 3C. In the example shown in FIG. 3, the adding unit 3C combines the planar normal image conversion image IM2 shown in FIG. 3(A) and the virtual road surface paint VP (lane marking) shown in FIG. 3(B), to generate the planar image IM3 with virtual road surface paint shown in FIG. 3(C).

[0016] In the example shown in FIG. 1, a learning image generation unit 3D performs a conversion inverse to the planar normal image conversion executed by the image conversion unit 3B on the planar image IM3 with virtual road surface paint, so as to generate a learning fisheye image IM4 (see FIG. 4) including the virtual road surface paint VP (lane marking).

[0017] FIG. 4 is a diagram showing an example of the learning fisheye image IM4 generated by the learning image generation unit 3D. In the example shown in FIG. 4, the learning image generation unit 3D performs a conversion inverse to the planar normal image conversion on the planar image IM3 with virtual road surface paint shown in FIG. 3(C), to generate the learning fisheye image IM4 including the virtual road surface paint VP (lane marking).

[0018] As described above, in the examples shown in Figures 1 to 4, the training vehicle L1 and training trailer L2 do not need to actually drive on a road surface with road markings (lane markings) drawn on it, yet they can obtain training fisheye images IM4 that include lane markings (virtual road markings VP) in the same way as if they were actually driving on a road surface with road markings (lane markings) drawn on it. In other words, in the examples shown in Figures 1 to 4, it is possible to train a model using fisheye images that include road markings (lane markings) as training data without actually needing to capture fisheye images that include road markings (lane markings) with the training camera L11.

[0019] In one application example of the learning image generation device 1 of the first embodiment, a learning fisheye image IM4, which includes virtual road surface paint VP (lane markings) generated by the learning image generation unit 3D of the learning image generation device 1, is used as training data for training a model used to estimate the state of trailer R2 (for example, the hitch angle Φ of trailer R2) based on a fisheye image taken by a camera R11 (see Figure 5) mounted on a vehicle R1 (see Figure 5) that tows trailer R2 (see Figure 5) via a towing bar R3 (see Figure 5).

[0020] Figure 5 is a diagram illustrating the state of trailer R2 (hitch angle Φ of trailer R2) estimated using a model trained with training data that uses the training fisheye image IM4, which includes virtual road surface paint VP (lane markings) generated by the training image generation unit 3D of the training image generation device 1, as training data. In the example shown in Figure 5, camera R11 is positioned at the rear end R1R of vehicle R1. Camera R11 photographs the area behind vehicle R1 (right side in Figure 5). Vehicle R1 tows trailer R2 via tow bar R3. Trailer R2 is rotatably connected to vehicle R1 around a hitch ball (not shown). The fisheye image captured by camera R11 includes a portion of vehicle R1, trailer R2, and tow bar R3. In one application example of the learning image generation device 1 of the first embodiment (the example shown in Figures 1 to 5), the hitch angle Φ of trailer R2 is estimated based on the fisheye image (image including trailer R2, etc.) captured by camera R11 (see Figure 5). This is achieved by using a model obtained by training with training data which is a dataset of training fisheye image IM4 including virtual road surface paint VP (lane markings) generated by the learning image generation unit 3D of the learning image generation device 1, and labels indicating the hitch angle θ (see Figure 2(A)) of the training trailer L2 at the time of capturing the training fisheye image IM1 (see Figure 2(B)) corresponding to the training fisheye image IM4. In detail, the model training uses a dataset of training fisheye images IM4 containing virtual road markings (VP), and labels indicating the hitch angle θ of the training trailer L2 at the time the corresponding training fisheye image IM1 was taken (see Figure 2(A)). In addition, a dataset of training fisheye image IM1 without virtual road markings (VP), and labels indicating the hitch angle θ of the training trailer L2 at the time the training fisheye image IM1 was taken (see Figure 2(A)) is also used as training data.

[0021] In one application example of the learning image generation device 1 of the first embodiment (the example shown in Figures 1 to 5), the learning fisheye image IM4, which includes virtual road surface paint VP (lane markings) generated by the learning image generation unit 3D of the learning image generation device 1, is used to train a model used to estimate the hitch angle Φ of the trailer R2. However, in other application examples, the learning fisheye image IM4, which includes virtual road surface paint VP (lane markings), may be used to train a model used to estimate the state of the trailer R2 other than the hitch angle Φ, such as estimating the position (coordinates) of the ends (left end and right end) of the trailer R2.

[0022] Figure 6 is a flowchart illustrating an example of the process performed in the learning image generation device 1 of the first embodiment. In the example shown in Figure 6, in step S10, the acquisition unit 3A acquires a learning fisheye image IM1 that does not include road markings (lane lines) and is captured by the learning camera L11. In step S11, the image conversion unit 3B generates a planar normalized image IM2 by performing a planar normalized image conversion on the learning fisheye source image IM1 acquired in step S10. In step S12, the addition unit 3C generates a planar image IM3 with virtual road markings by adding virtual road markings VP (lane lines) to the planar normalized image IM2 generated in step S11. In step S13, the training image generation unit 3D generates a training fisheye image IM4 that includes virtual road markings VP (lane lines) by performing a transformation on the planar image IM3 with virtual road markings that is the reverse of the planar normalization transformation performed in step S11. Learning can be performed.

[0023] <Second Embodiment> As described above, in the first embodiment (the example shown in Figures 1 to 6), the acquisition unit 3A acquires a learning fisheye image IM1 that does not include road markings or other road surface paint. On the other hand, in the second embodiment, the acquisition unit 3A acquires a learning fisheye image IM1 that does not include road markings as road paint (in detail, for example, road markings such as speed limit, no U-turn, left turn arrow, straight arrow, right turn arrow, etc.).

[0024] As described above, in the first embodiment (an example shown in Figures 1 to 6), the addition unit 3C generates a planar image IM3 with virtual road surface paint by adding virtual road surface paint VP (lane markings) to the planar normalized image IM2 generated by the image conversion unit 3B. On the other hand, in the second embodiment, the additional unit 3C generates a planar image IM3 with virtual road markings by adding virtual road markings VP (road markings) to the planar normalized image IM2 generated by the image conversion unit 3B.

[0025] As described above, in the first embodiment (the example shown in Figures 1 to 6), the learning image generation unit 3D generates a learning fisheye image IM4 including virtual road surface paint VP (lane markings) by performing a transformation on the planar image IM3 with virtual road surface paint that is the reverse of the planar normalization transformation performed by the image transformation unit 3B. On the other hand, in the second embodiment, the learning image generation unit 3D generates a learning fisheye image IM4 including virtual road markings VP (road markings) by performing a transformation on the planar image IM3 with virtual road markings that is the reverse of the planar normalization transformation performed by the image transformation unit 3B. In the second embodiment, the learning vehicle L1 and the learning trailer L2 can obtain a learning fisheye image IM4 including road markings (virtual road markings VP) in the same way as if they were actually driving on a road with road markings, without the learning vehicle L1 and the learning trailer L2 having to actually drive on a road with road markings. In other words, in the second embodiment, it is possible to train a model using a fisheye image including road markings as training data without the need to actually capture a fisheye image including road markings with the learning camera L11.

[0026] As described above, embodiments of the learning image generation device, learning image generation method, and program of this disclosure have been explained with reference to the drawings. However, the learning image generation device, learning image generation method, and program of this disclosure are not limited to the embodiments described above, and can be modified as appropriate without departing from the spirit of this disclosure. The configurations of each example of the embodiments described above may be combined as appropriate. In each example of the embodiments described above, the processing performed in the learning image generation device 1 was described as software processing performed by executing a program. However, the processing performed in the learning image generation device 1 may be hardware processing. Alternatively, the processing performed in the learning image generation device 1 may be a combination of both software and hardware processing. Furthermore, the program stored in the memory 12 of the learning image generation device 1 (a program that realizes the functions of the processor 13 of the learning image generation device 1) may be recorded on a computer-readable storage medium such as a semiconductor memory, magnetic recording medium, or optical recording medium and provided and distributed. [Explanation of Symbols]

[0027] 1... Learning image generation device, 11... Communication interface, 12... Memory, 13... Processor, 3A... Acquisition unit, 3B... Image conversion unit, 3C... Addition unit, 3D... Learning image generation unit, IM1... Learning fisheye original image, IM2... Planar normalized image conversion image, IM3... Planar image with virtual road surface paint, IM4... Learning fisheye image, VP... Virtual road surface paint, L1... Learning vehicle, L1R... Rear end, L11... Learning camera, L2... Learning trailer, L3... Learning tow bar

Claims

1. A learning image generation device that generates learning fisheye images as learning data used to train a model used for estimating the state of a trailer based on fisheye images taken by a camera mounted on a vehicle that tows the trailer via a tow bar, An acquisition unit that acquires the original fisheye image of a learning device captured by a learning camera mounted on a learning vehicle that tows a learning trailer via a learning tow bar, An image conversion unit generates a planar normalized image by performing a planar normalized image conversion, which is a conversion from a fisheye image to a planar image, on the learning fisheye source image acquired by the acquisition unit. An additional unit adds a virtual road surface paint to the planar normalized image converted by the image conversion unit, A learning image generation device comprising: a learning image generation unit that generates the learning fisheye image by performing a transformation in the reverse of the plane normalization transformation performed by the image transformation unit on the planar image with virtual road surface paint, which is the planar normalization transformation image after the virtual road surface paint has been added by the addition unit.

2. A learning image generation step in which a learning image generation device generates learning fisheye images as learning data used to train a model used to train a model that estimates the state of the trailer based on fisheye images taken by a camera mounted on a vehicle that tows the trailer via a tow bar, The learning image generation device acquires a learning fisheye image captured by a learning camera mounted on a learning vehicle that tows a learning trailer via a learning tow bar, and The learning image generation device performs a planar normalization transformation, which is a transformation from a fisheye image to a planar image, on the learning fisheye source image acquired in the acquisition step, thereby generating a planar normalization image; The learning image generation device includes an additional step of adding virtual road surface paint to the planar normalized image generated in the image conversion step, In the training image generation step, the training fisheye image is generated by performing a transformation in the reverse of the plane normalization transformation performed in the image transformation step on the plane image with virtual road surface paint, which is the plane normalization transformation image obtained after the virtual road surface paint has been added in the addition step. Method for generating training images.

3. In the processor, A training image generation step of generating training fisheye images as training data used to train a model used to train a model that estimates the state of the trailer based on fisheye images taken by a camera mounted on a vehicle that tows the trailer via a tow bar, The acquisition step involves obtaining the original fisheye image of the learning device, which is captured by a learning camera mounted on a learning vehicle that is towing a learning trailer via a learning tow bar. An image transformation step is performed on the fisheye image for training acquired in the acquisition step to generate a plane image normalization image by performing a plane image normalization transformation, which is a transformation from a fisheye image to a plane image. A program for causing an additional step to be performed in which a virtual road surface paint is added to the planar normalized image generated in the image conversion step, The program generates the training fisheye image in the training image generation step by performing a transformation in the reverse of the plane normalization transformation performed in the image transformation step on the plane image with virtual road surface paint, which is the plane normalization transformation image after the virtual road surface paint has been added in the addition step.

Citation Information

Patent Citations

  • Traction support device

    JP2018176788A

  • Information processing device

    JP2019120985A

  • Estimating and removing lens distortion from scenes

    US20070098296A1

  • Method and apparatus for detecting a trailer, tow-ball, and coupler for trailer hitch assistance and jackknife prevention

    US20220084212A1

  • Vehicle neural network

    US20220092356A1