Processing device, processing method, and program

Abstract

To enable appropriate ortho-rectification irrespective of an installation direction or an angle of a camera, for example.SOLUTION: A processing device 10 comprises an acquisition unit 120 and a calculation unit 140. The acquisition unit 120 acquires a reference image. The calculation unit 140 calculates a parameter for projective transformation on the basis of the shape and size of an object in the reference image. The object included in the reference image is, for example, a mark drawn on a road surface. The calculation unit 140 extracts a subregion including a mark from the reference image, for example, to calculate a parameter on the basis of the mark included in the subregion.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a processing device, a processing method, and a program. [Background technology]

[0002] Orthoimages of road surfaces are used to create maps for autonomous driving and navigation systems, as well as for road management.

[0003] Patent Document 1 discloses an apparatus for generating road surface orthoimages based on data collected by a mobile measurement vehicle. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-90591 Summary of the Invention [Problem to be solved by the invention]

[0005] For example, when generating an orthoimage using images acquired by various vehicle-mounted cameras, the conversion parameters for orthogonalization differ depending on the mounting direction and angle of the camera. Even if the mounting direction and angle of the camera are adjusted, they may still change due to vibration or impact.

[0006] One example of a problem that the present invention aims to solve is to enable appropriate ortho-conversion regardless of the mounting direction or angle of the camera. [Means for solving the problem]

[0007] The invention described in claim 1 is an acquisition unit that acquires a reference image; a calculation unit that calculates parameters for projective transformation based on the shape and size of the object in the reference image. It is a processing device.

[0008] The invention described in claim 14 is an acquisition step of acquiring a reference image; and calculating parameters for projective transformation based on the shape and size of the object in the reference image. It is a processing method.

[0009] The invention described in claim 15 is A program that causes a computer to execute each step of the processing method according to claim 14. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram illustrating a configuration of a processing device according to a first embodiment. [Figure 2] FIG. 10 is a diagram illustrating an example of a reference image. [Figure 3] 10A and 10B are diagrams illustrating an example of an edge-detected image obtained by performing edge detection on a reference image. [Figure 4] 10A is a diagram showing the vertex coordinates of the outer edge of a partial region in a reference image, and FIG. 10B is a diagram showing the vertex coordinates of the outer edge of a partial region after ortho-transformation. [Figure 5] 3 is a flowchart illustrating the flow of a processing method according to the first embodiment. [Figure 6] FIG. 1 is a diagram illustrating an example of a usage environment of a processing apparatus according to a first embodiment. [Figure 7] FIG. 10 is a diagram illustrating a computer for realizing the processing device. [Figure 8] FIG. 10 is a block diagram illustrating the configuration of a processing device according to a second embodiment. [Figure 9] 10 is a flowchart illustrating a processing method executed by a processing device according to a second embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of edges detected for an object. [Figure 11] FIG. 10 is a diagram for explaining a method for deriving a shape score. [Figure 12]10 is a flowchart illustrating a method for a calculation unit according to a third embodiment to derive the size of an object. [Figure 13] 10A and 10B are diagrams illustrating a method in which a calculation unit according to the fourth embodiment derives the size of an object. [Figure 14] FIG. 10 is a diagram illustrating the configuration of a processing apparatus according to a fifth embodiment. [Figure 15] 10A and 10B are diagrams for explaining a method in which a transformation unit determines a target image and performs projective transformation on the target image. [Figure 16] 10 is a flowchart illustrating the flow of processing performed by a processing device according to a fifth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, like components are designated by like reference numerals, and the description thereof will be omitted as appropriate.

[0012] In the following description, unless otherwise specified, each component of the processing device 10 is represented as a functional block rather than a hardware configuration. Each component of the processing device 10 is realized by any combination of hardware and software, centered around the CPU of any computer, memory, a program loaded into the memory, a storage medium such as a hard disk for storing the program, and a network connection interface. There are many variations in the realization method and device.

[0013] (First embodiment) Fig. 1 is a block diagram illustrating the configuration of a processing device 10 according to the first embodiment. Fig. 2 is a diagram illustrating an example of a reference image 20. The processing device 10 includes an acquisition unit 120 and a calculation unit 140. The acquisition unit 120 acquires the reference image 20. The calculation unit 140 calculates parameters for projective transformation based on the shape and size of an object 200 in the reference image 20. This will be explained in detail below.

[0014] For example, while a moving object such as a car or motorcycle is traveling on a road, a camera captures images of the surrounding area. The road is captured in the images acquired in this way. If images acquired by the camera of any moving object, not just a dedicated vehicle, can be projectively transformed (ortho-transformed), it will be possible to easily collect a large amount of image data required to generate road surface orthoimages.

[0015] However, even if a camera is installed on a moving object, the camera's orientation may change due to vibrations, human contact, or other factors. For example, adjusting the camera's position and measuring its orientation every time the moving object moves is extremely time-consuming. In contrast, the processing device 10 according to this embodiment can calculate the parameters required for projective transformation using a reference image 20 captured of a reference object 200. The calculated parameters can then be used to appropriately perform projective transformation, thereby reducing labor.

[0016] As will be described in detail later, the calculation unit 140 extracts a partial region 22 including a mark from the reference image 20 and calculates parameters based on the mark included in the partial region 22. The object 200 included in the reference image 20 is, for example, a mark on the ground. Note that the ground may be, for example, an underground, elevated, or a surface within a structure, as long as a surface on which a mobile object can move. Examples of the object 200 include a dividing line drawn on a road or a crosswalk. The object 200 may also be a feature on the floor of a garage. The object 200 may also be a reference object installed for a specific purpose or for other purposes. In this case, the reference object can be managed to prevent unexpected changes. The object 200 is, for example, rectangular in plan view in real space.

[0017] 2 to 4(b), a method for calculating parameters by the calculation unit 140 will be described. FIG. 2 is a diagram illustrating an example of a reference image 20. The reference image 20 shows a road and an object 200 on the road. In this example, the object 200 is a white line on the road. The white line is a dashed line separated by a predetermined length. The length and width of each white line are determined in advance. The calculation unit 140 performs edge detection on the reference image 20. FIG. 3 is a diagram illustrating an edge-detected image obtained by performing edge detection on the reference image 20. Next, the calculation unit 140 extracts a partial region 22 that includes the object 200 in the edge-detected image. The calculation unit 140 can detect the object 200 from the reference image 20 using an existing image processing method. In the example shown in this figure, the partial region 22 includes two objects 200 and a region sandwiched between the two objects 200. Furthermore, at least a portion of the outer edge of the partial region 22 overlaps the outer edge of the object 200. At least a part of the outer edge of the partial region 22 connects the edges of the two objects 200. However, the partial region 22 may include only one object 200, and the outer edge of the object 200 and the outer edge of the partial region 22 may entirely coincide. However, by including the two objects 200 and the area sandwiched between the two objects 200 in the partial region 22, highly accurate parameter calculation is possible using the wide partial region 22.

[0018] Next, the calculation unit 140 acquires the vertex coordinates of the outer edge of the partial region 22 in the reference image 20. The coordinate system here is one in which the reference image 20 is placed on the xy plane. Fig. 4(a) is a diagram showing the vertex coordinates of the outer edge of the partial region 22 in the reference image 20. In this diagram, the coordinates of the four vertices of the partial region 22 are shown: (x1, y1), (x2, y2), (x3, y3), and (x4, y4).

[0019] Furthermore, the calculation unit 140 derives the vertex coordinates of the outer edge of the partial region 22 after orthorectification based on the thickness (width) and length of the object 200 in real space. The coordinate system here is assumed to be an XY plane on which the image after orthorectification is placed. FIG. 4(b) is a diagram showing the vertex coordinates of the outer edge of the partial region 22 after orthorectification. In this diagram, the four vertex coordinates (X1, Y1), (X2, Y2), (X3, Y3), and (X4, Y4) of the partial region 22 are shown. For example, the distance d1 between (X1, Y1) and (X4, Y4) is calculated by multiplying the actual length L of the object 200 by the distance d1. t The distance d2 between (X1, Y1) and (X2, Y2) is determined based on the width W of the actual object 200. t and the actual distance d between the two objects 200 t That is, d1 / d2=L t / (2×W t +d t ) holds. The calculation unit 140 calculates the vertex coordinates of the partial region 22 before and after the projective transformation in this way, thereby obtaining the transformation parameters used for the projective transformation.

[0020] In addition, the distance d t If it is unknown, the ratio of the distance between two objects 200 in the reference image 20 to the width of the object 200 may be calculated, and the distance d2 may be determined based on the ratio. That is, if the distance between two objects 200 in the reference image 20 is α times the width of the object 200, then d1 / d2=L t / (2×W t +α×W t ) can be set so that the coordinates are

[0021] The calculated parameters can be used when performing projective transformation on images captured by the same camera of the same moving object. For example, parameter calculation is performed for each series of videos. Specifically, one of the images constituting the series of videos is used as the reference image 20, and parameters are calculated. The calculated parameters can be used to perform projective transformation on the other images constituting the video and the reference image 20. Note that parameter calculation does not necessarily have to be performed for each series of videos. However, because the camera's posture may change after a certain amount of time has passed, it is preferable to perform parameter calculation for the same camera at least every predetermined period of time.

[0022] FIG. 5 is a flowchart illustrating the flow of a processing method according to this embodiment. The processing method according to this embodiment includes an acquisition step S10 and a calculation step S30. In the acquisition step S10, a reference image 20 is acquired. In the calculation step S30, parameters for projective transformation are calculated based on the shape and size of the object 200 in the reference image 20. The processing method according to this embodiment is executed by a processing device 10. The processing performed by the processing device 10 according to this embodiment will be described in detail below.

[0023] In the acquisition step S10, the acquisition unit 120 acquires the reference image 20. The acquisition unit 120 can acquire the reference image 20, for example, from a camera (image capture device) mounted on a moving body. The camera is, for example, an in-vehicle camera. The reference image 20 is, for example, an image including a road. The reference image 20 may also be one of multiple images that make up a video. The reference image 20 may be stored in a storage device in advance, and the acquisition unit 120 may read and acquire the reference image 20 from a storage device accessible by the acquisition unit 120. The reference image 20 may also be an image taken by a camera other than the camera mounted on the moving body. However, the reference image 20 includes the object 200.

[0024] 6 is a diagram illustrating an example of a usage environment of the processing device 10 according to this embodiment. For example, a server machine 50 may collect images from cameras 42 mounted on multiple mobile objects 40, and an acquisition unit 120 of the processing device 10 may acquire the reference image 20 from the server machine 50.

[0025] The calculation unit 140 acquires size information indicating the size of the object 200 in real space and calculates parameters using the size information. Specifically, the calculation unit 140 reads and acquires the size information stored in a storage unit accessible from the calculation unit 140. In this embodiment, the shape of the object 200, the size of the object 200, and the distance between two objects 200 are known. For example, if the object 200 is a white line painted on a road, the size information is determined based on road marking standards. Also, if the object 200 is an installed reference object, the size information is determined based on the size of the reference object. The size information may also include information indicating the distance between multiple objects 200. Note that size information may differ depending on the location, such as between a highway and other roads. In this case, the storage unit stores size information for each location, for example. The calculation unit 140 can then select and read size information from the storage unit according to the capture position of the reference image 20.

[0026] In calculation step S30, the calculation unit 140 calculates parameters necessary for projective transformation using the reference image 20 and the size information, as described above with reference to FIGS. 2 to 4(b). That is, the calculation unit 140 extracts the object 200 from the reference image 20, and calculates the parameters using at least the size of the object 200 in the reference image 20 and the size of the object 200 in real space based on the size information. For example, if the object 200 is rectangular in plan view in real space, the calculation unit 140 calculates the parameters so that the ratio of the long side to the short side of the object 200 is the same between the object 200 in real space and the object 200 in an orthoimage obtained using the parameters. Note that if the partial region 22 includes multiple objects 200, the calculation unit 140 may calculate the parameters by further using the distance between the multiple objects 200.

[0027] In more detail, for example, the calculation unit 140 performs a process to correct lens distortion on the reference image 20, and then performs edge detection. Then, the object 200 is detected based on the detected edges. Then, a partial region 22 including the object 200 is extracted. The partial region 22 is, for example, a region that becomes rectangular when projectively transformed. The calculation unit 140 calculates x and y coordinates (for example, coordinates of the four vertices of a rectangle) indicating the outer edge of the partial region 22 in the reference image 20. Furthermore, the calculation unit 140 calculates x and y coordinates (for example, coordinates of the four vertices of a rectangle) indicating the outer edge of the partial region 22 in the orthoimage after projective transformation based on the acquired size information. Based on the calculated x and y coordinates and x and y coordinates, the calculation unit 140 derives transformation parameters. As a method for deriving the transformation parameters based on the calculated x and y coordinates and x and y coordinates, an existing method can be used.

[0028] As described above, the processing device 10 according to this embodiment calculates appropriate parameters using the acquired reference image 20. Therefore, regardless of the orientation or angle of the camera or the like that captured the image, it is possible to perform projective transformation and obtain a good orthoimage.

[0029] The hardware configuration of the processing device 10 will be described below. Each functional component of the processing device 10 may be realized by hardware that realizes the functional component (e.g., a hardwired electronic circuit, etc.), or by a combination of hardware and software (e.g., a combination of an electronic circuit and a program that controls it). Below, a case where each functional component of the processing device 10 is realized by a combination of hardware and software will be further described.

[0030] 7 is a diagram illustrating a computer 1000 for realizing the processing device 10. The computer 1000 is any computer. For example, the computer 1000 is a system on chip (SoC), a personal computer (PC), a server machine, a tablet terminal, a smartphone, or the like. The computer 1000 may be a dedicated computer designed to realize the processing device 10, or may be a general-purpose computer.

[0031] The computer 1000 includes a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input / output interface 1100, and a network interface 1120. The bus 1020 is a data transmission path through which the processor 1040, the memory 1060, the storage device 1080, the input / output interface 1100, and the network interface 1120 transmit and receive data to and from each other. However, the method of interconnecting the processor 1040 and other components is not limited to bus connection. The processor 1040 may be any of various processors, such as a central processing unit (CPU), a graphics processing unit (GPU), or a field-programmable gate array (FPGA). The memory 1060 is a main storage device implemented using a random access memory (RAM) or the like. The storage device 1080 is an auxiliary storage device implemented using a hard disk, a solid state drive (SSD), a memory card, a read-only memory (ROM), or the like.

[0032] The input / output interface 1100 is an interface for connecting the computer 1000 to an input / output device. For example, the input / output interface 1100 is connected to an input device such as a keyboard and an output device such as a display device.

[0033] The network interface 1120 is an interface for connecting the computer 1000 to a network. This communication network is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network). The network interface 1120 may be connected to the network wirelessly or by wire.

[0034] The storage device 1080 stores program modules that realize the various functional components of the processing device 10. The processor 1040 reads these program modules into the memory 1060 and executes them to realize the functions corresponding to the respective program modules.

[0035] As described above, according to this embodiment, the calculation unit 140 calculates parameters for projective transformation based on the shape and size of the object 200 in the reference image 20. Therefore, appropriate ortho transformation is possible regardless of the mounting direction or angle of the camera.

[0036] (Second embodiment) 8 is a block diagram illustrating the configuration of a processing device 10 according to the second embodiment. The processing device 10 according to this embodiment is the same as the processing device 10 according to the first embodiment, except for the points described below. In the processing device 10 according to this embodiment, an acquisition unit 120 acquires a plurality of images including a reference image 20. The processing device 10 according to this embodiment further includes a selection unit 130 that selects the reference image 20 from the plurality of images acquired by the acquisition unit 120. This will be described in detail below.

[0037] The processing device 10 calculates the parameters using the reference image 20. Here, the accuracy of the parameters depends on the quality of the reference image 20. Therefore, it is preferable to select and use a reference image 20 that is suitable for calculating the parameters. In the processing device 10 according to this embodiment, the acquisition unit 120 selects multiple images, and the selection unit 130 selects one of these images that is suitable as the reference image 20. Then, the calculation unit 140 calculates the parameters using the selected reference image 20.

[0038] 9 is a flowchart illustrating a processing method executed by the processing device 10 according to this embodiment. In the processing method according to this embodiment, a selection step S20 is performed between the acquisition step S10 and the calculation step S30. In the processing method according to this embodiment, a plurality of images including a reference image 20 are acquired in the acquisition step S10. Then, in the selection step S20, the reference image 20 is selected from the plurality of images.

[0039] The acquisition unit 120 acquires, for example, a video captured by a camera mounted on a moving object. The video consists of multiple images. However, the acquisition unit 120 may acquire multiple images that do not constitute a video. The multiple images may be stored in a storage device in advance, and the acquisition unit 120 may read and acquire the reference image 20 from a storage device accessible by the acquisition unit 120. For example, as shown in FIG. 6, a server machine 50 may collect multiple images from cameras 42 mounted on multiple moving objects 40, and the acquisition unit 120 of the processing device 10 may acquire the multiple images from the server machine 50. The multiple images acquired by the acquisition unit 120 may include images that do not include the target object 200.

[0040] The method by which the selection unit 130 selects the reference image 20 will be described below. As will be described below, there are multiple perspectives for selecting a preferred reference image 20. The selection unit 130 may select the reference image 20 based on any one of the perspectives, or may select the reference image 20 based on multiple perspectives. When selecting the reference image 20 based on multiple perspectives, for example, the selection unit 130 can calculate a score for each perspective, calculate an overall score by combining these scores, and select the reference image 20 based on the overall score.

[0041] The selection unit 130 first extracts images that include the entire object 200 from the multiple images. Specifically, the selection unit 130 performs processing to correct lens distortion on the multiple images and then performs edge detection. Next, the selection unit 130 detects the object 200 in the images based on the detected edges. Then, from the multiple images, the selection unit 130 extracts images that include the entire object 200, and selects the reference image 20 from the extracted images.

[0042] <Method based on the position of the object 200> The selection unit 130 can select the reference image 20 based on the position of the object 200 in the image. Specifically, it is preferable to select an image in which the object 200 is located in a region closer to the camera, i.e., lower in the image, as the reference image 20. This increases the accuracy of edge detection and improves the image quality of the orthoimage generated when that region is projectively transformed. Note that the lower side of the image is the side closer to the moving object in real space, and the upper side of the image is the side farther from the moving object in real space.

[0043] For example, the selection unit 130 selects the image in which the object 200 is positioned lowest among the images as the reference image 20. When calculating the score, the selection unit 130 assigns a higher score to an image in which the object 200 is positioned lower among the images.

[0044] <Method based on the state of the moving object> If the multiple images are images obtained by a camera (image capture device) attached to a moving object, the selection unit 130 can select the reference image 20 based on the situation of the moving object when the image was obtained. The situation of the moving object is, for example, the speed and attitude of the moving object at the time of image capture, and more specifically, at least one of the direction, speed, roll, and pitch of the moving object at the time of image capture. It is preferable to evaluate whether the situation of the moving object at the time of image capture was stable, and select an image captured in a stable situation as the reference image 20. The processing device 10 can acquire sensor information indicating the situation of the moving object from a sensor provided on the moving object.

[0045] For example, the selection unit 130 acquires sensor information of the moving object for a predetermined period including the timing at which each image was captured, based on the capture time of the image. The sensor information includes, for example, sensor values ​​from one or more sensors selected from a speed sensor, a direction sensor, a roll sensor, and a pitch sensor. The selection unit 130 calculates the standard deviation for each sensor value included in the acquired sensor information for the predetermined period. Then, the selection unit 130 sums the standard deviations of the one or more sensors. For example, the selection unit 130 selects the image with the smallest sum of standard deviations as the reference image 20. Furthermore, when calculating the score, the selection unit 130 assigns a higher score to an image with a smaller sum of standard deviations.

[0046] <Method based on edge detection results> The selection unit 130 can select the reference image 20 based on the results of edge detection performed on the image. If the object 200 is a mark such as a white line painted on a road or a crosswalk, the paint may have faded or the like. Therefore, it is preferable to select as the reference image 20 an image in which edges can be clearly detected based on the strength at the time of edge detection, whether the positional relationship of the detected edges is close to the outline of the object 200, and the like.

[0047] Specifically, the selection unit 130 calculates the straightness score, shape score, and strength score of the edge as follows. The sum of the calculated straightness score, shape score, and strength score is then set as the score based on edge detection. The selection unit 130 can select the image with the highest score based on edge detection as the reference image 20.

[0048] <<Straightness score>> The selection unit 130 derives the length of the edge detected for the object 200. Then, the longer the derived edge length is within the range of an upper limit, the higher the straightness score is assigned to the image. On the other hand, the lowest score is assigned to the image whose derived edge length exceeds the upper limit.

[0049] 10(a) to 10(c) are diagrams illustrating edges detected for the object 200. FIG. 10(a) shows the detection result for a good object 200. A long edge 210 is detected according to the length of the object 200. FIG. 10(b) shows the detection result when the object 200 is blurred. Part of the edge 210 is broken. FIG. 10(c) is an example in which noise or something other than the object 200 is detected. The edge 210 is excessively long.

[0050] <<Shape score>> FIG. 11 is a diagram illustrating a method for deriving a shape score. The selection unit 130 derives the inclination of the edge 210 detected for the object 200 for all images and calculates the average value of the derived inclinations. Then, the average value of the edge inclinations for all images (the inclination indicated by line 220 in this figure) is compared with the inclination of the edge 210 detected for the object 200 in each image, and a higher shape score is assigned to an image in which an edge with an inclination closer to the average value is detected. Note that when multiple objects 200 with different inclinations are included in each image, the selection unit 130 groups the objects 200 in the multiple images that have similar inclinations as collinear objects 200. The edges of the grouped objects 200 are then used to calculate the average value for each group. The inclination of the edge 210 of the object 200 in each image is then compared with the average value for the group to which that object 200 belongs.

[0051] <<Intensity score>> The selection unit 130 derives the intensity at the time of detection for the edge detected for the object 200. Then, the higher the intensity of the image, the higher the intensity score is assigned.

[0052] <Method based on shooting position> The selection unit 130 can select an image whose shooting position satisfies a predetermined condition as the reference image 20. In this case, information indicating the shooting position of each image is associated with the multiple images. The acquisition unit 120 then acquires the multiple images along with information indicating the shooting position of each image.

[0053] For example, if the object 200 is a white line painted on a road, the spacing between the white lines is not set to a specific value on ordinary roads. Also, depending on the slope of the road surface and the curvature of the road, there may be cases where the white lines are not rectangular. Therefore, it is preferable to check the shape of the object 200 at each location in advance using high-precision map data, aerial photographs, etc., determine the object 200 to be used as a reference, and select an image of that object 200 as the reference image 20.

[0054] A storage unit accessible from the selection unit 130 stores in advance recommended area information indicating an area on a map where a reference object 200 can be photographed. The selection unit 130 acquires the recommended area information from the storage unit. The selection unit 130 can then select an image whose photographing position is within the area indicated in the recommended area information as the reference image 20. When calculating the score, the selection unit 130 assigns a higher score to an image whose photographing position is within the area indicated in the recommended area information than to an image that is not within the area.

[0055] <Method based on overall score> The selection unit 130 can select the reference image 20 by combining the evaluation results based on the above-mentioned aspects, i.e., the position of the object 200, the state of the moving object, the edge detection result, and the shooting position. Specifically, the selection unit 130 calculates a weighted sum of the scores derived from each aspect as a total score. The weight of each score in the weighted sum is determined in advance. The selection unit 130 selects the image with the highest total score as the reference image 20.

[0056] Although examples of the method by which the selection unit 130 selects the reference image 20 have been described above, the selection unit 130 may select the reference image 20 by a method other than these examples.

[0057] The calculation unit 140 calculates the parameters in the same manner as in the first embodiment, using the reference image 20 selected by the selection unit 130. Note that the calculation unit 140 only needs to calculate the parameters using the results of processing such as correction, edge detection, and detection of the object 200 performed by the selection unit 130, and does not need to perform these processes again.

[0058] The hardware configuration of a computer that realizes the processing device 10 according to this embodiment is, for example, shown in Fig. 7, similar to the first embodiment. However, a program module that realizes the function of the selection unit 130 is further stored in the storage device 1080 of the computer 1000 that realizes the processing device 10 according to this embodiment.

[0059] As described above, according to this embodiment, the same actions and effects as those of the first embodiment can be obtained. In addition, according to this embodiment, the processing device 10 further includes a selection unit 130 that selects a reference image 20 from the multiple images acquired by the acquisition unit 120. Therefore, it is possible to calculate highly accurate parameters using a reference image 20 that is suitable for parameter calculation.

[0060] (Third embodiment) FIG. 12 is a flowchart illustrating a method by which the calculation unit 140 according to the third embodiment derives the size of the object 200. The processing device 10 according to this embodiment is the same as the processing device 10 according to the first or second embodiment, except for the points described below. In this embodiment, the acquisition unit 120 acquires a plurality of images, including a reference image 20, constituting a video. This video is video acquired by a camera (image capture device) attached to a moving object. The calculation unit 140 then calculates the size of the object 200 in real space using the video and the speed of the moving object when the video was acquired. Furthermore, the calculation unit 140 calculates parameters using the size of the object 200 in real space. This will be described in detail below.

[0061] In the first and second embodiments, the size of the object 200 is known, and the calculation unit 140 calculates the parameters using size information prepared in advance. In contrast, in the processing device 10 according to this embodiment, the calculation unit 140 identifies the size of the object 200 (size in real space) using multiple images. Then, the calculation unit 140 calculates the parameters using the identified size and the reference image 20. Therefore, the parameters can be calculated even if the size of the object 200 is unknown.

[0062] In the processing device 10 according to this embodiment, the acquisition unit 120 acquires multiple images in the same manner as in the second embodiment. In this embodiment, the partial region 22 is, for example, rectangular in plan view in real space. The calculation unit 140 then assumes multiple lengths L and widths W of the partial region 22 and identifies appropriate lengths L and widths W from among them. Specifically, the calculation unit 140 calculates the estimated movement distance of the moving object between multiple consecutive images using the assumed values ​​of length L and width W, and compares this with the movement distance calculated from the detection value of the sensor of the moving object. In this way, the most likely length L and width W are identified.

[0063] The multiple images acquired by the acquisition unit 120 constitute a series of moving images. Then, the calculation unit 140 performs the following process on the multiple images in the chronological order in which they were taken. That is, in S303 described later, the multiple images are read in order in the chronological order in which they were taken.

[0064] With reference to FIG. 12 , a method by which the calculation unit 140 determines the size of the partial region 22 will be described. When the acquisition unit 120 acquires multiple images, the calculation unit 140 performs the following processing to determine the size of the partial region 22. S301 to S312 constitute loop processing A. In loop processing A, repeated processing is performed while changing the assumed value of L for each loop. Multiple assumed values ​​of L are determined in advance, and loop processing A ends when processing has been performed for all of these assumed values. S302 to S311 constitute loop processing B. In loop processing B, repeated processing is performed while changing the assumed value of W for each loop. Multiple assumed values ​​of W are determined in advance, and loop processing B ends when processing has been performed for all of these assumed values. The assumed values ​​of L and W are set within the range of possible sizes of the partial region 22.

[0065] S303 to S310 constitute loop processing C. In loop processing C, multiple images acquired by acquisition unit 120 are repeatedly processed while being read in order for each loop. When processing has been performed on all of the multiple images, loop processing C ends.

[0066] In S304, detection of the object 200 and extraction of the partial region 22 are attempted for the loaded image. Here, image correction and edge detection may be performed as appropriate prior to detection. When the partial region 22 is extracted, the calculation unit 140 reads the position of the object 200 on the image. Note that even if the partial region 22 includes multiple objects 200, it is sufficient for the calculation unit 140 to read the position of at least one object 200.

[0067] Next, in S305, the calculation unit 140 determines whether or not the reading of the position of the object 200 in S304 was successful. If the reading was not successful, the process proceeds to S310. On the other hand, if the reading was successful, in S306 the calculation unit 140 calculates the distance between the camera that captured the image and the object 200 based on the assumed values ​​of L and W and the position of the object 200 on the image. Note that an existing method can be used to calculate the distance between the camera and the object 200 based on the coordinates of the four vertices of the actual rectangular object 200, the coordinates of the four vertices of the object 200 on the image, and the camera characteristics. Here, the information indicating the camera characteristics can be acquired by the processing device 10 from the camera that captured the image.

[0068] Next, in S307, the calculation unit 140 compares the distance between the camera and the object 200 obtained in this S306 with the distance between the camera and the object 200 obtained in the image of the previous loop processing, and calculates the movement distance Dc of the moving object.

[0069] In S308, the calculation unit 140 acquires information indicating the speed of the moving object when the image was captured from the sensor of the moving object. Then, the calculation unit 140 calculates the distance Da traveled by the moving object from when the previous image was captured to when the current image was captured, based on the speed when the image was captured.

[0070] Then, in S309, the calculation unit 140 calculates the score so that the smaller the difference between the movement distance Dc and the movement distance Da, the higher the score becomes.

[0071] When loop process A ends in S312, that is, when processing is completed for all assumed values ​​of L, all assumed values ​​of W, and all images, the calculation unit 140 calculates the average score for each combination of assumed values ​​of L and W in S313. Then, the assumed values ​​of L and W for the combination with the highest calculated average value are identified as the size of the object 200 to be used in parameter calculation. Note that instead of calculating the average score for each combination of assumed values ​​of L and W, the calculation unit 140 may calculate the average score for each assumed value of L. In that case, the assumed value of L with the highest average value is identified as the L of the object 200. Alternatively, the calculation unit 140 may calculate the average score for each assumed value of W, and identify the assumed value of W with the highest average value as the W of the object 200.

[0072] In this way, the calculation unit 140 specifies the size of the partial region 22, and uses the specified size instead of size information to calculate the parameters in the same manner as described in the first embodiment.

[0073] Note that, in the above-described method, instead of calculating the distance between the camera that captured the image and the object 200, the calculation unit 140 may calculate the distance between the camera and an area on the actual road surface that corresponds to the partial area 22. Even in this case, the size of the partial area 22 can be determined in the same way.

[0074] In the example shown in FIGS. 2 to 4(b), the length L of the object 200 t is the same as the length L of the partial region 22. Therefore, by specifying the length L of the partial region 22, the length L of the object 200 can be calculated. t In the example shown in FIGS. 2 to 4(b), W=2×W t +d t The calculation unit 140 only needs to determine the size of the object 200 in at least one direction. For example, the calculation unit 140 determines the length L of the object 200 by determining the size of the partial region 22. t and the width W of object 200 t Both of these may be specified, or only one of them may be specified.

[0075] As described above, according to this embodiment, the same actions and effects as those of the first embodiment can be obtained. In addition, according to this embodiment, the calculation unit 140 identifies the size of the object 200 using multiple images. Therefore, even if the size of the object 200 is unknown, the parameters can be calculated.

[0076] (Fourth embodiment) In the processing device 10 according to the fourth embodiment, the calculation unit 140 calculates the length L of the object 200. t The processing apparatus 10 according to the third embodiment is the same as that according to the third embodiment except for the method of deriving the parameter .

[0077] In the processing device 10 according to this embodiment, the acquisition unit 120 acquires a plurality of images that make up a moving image. The calculation unit 140 performs correction and edge detection processing on the plurality of images as necessary. The calculation unit 140 extracts the following first and second images from the plurality of images. The first and second images contain the same object 200. Then, the length L of the object 200 in the first image is calculated. t The position of one end of the direction and the length L of the object 200 in the second image t The positions of the other ends of the directions are the same. The calculation unit 140 can extract the first image and the second image from the multiple images using an existing method. The object 200 is, for example, a white line painted on a road.

[0078] The calculation unit 140 calculates the difference between the extracted photographing timing of the first image and the photographing timing of the second image. The calculation unit 140 also acquires the speed of the moving object between these photographing timings. Then, the calculation unit 140 calculates the movement amount of the moving object between the photographing timing of the first image and the photographing timing of the second image using the calculated difference (time) in timing and the speed of the moving object. The calculation unit 140 multiplies the calculated movement amount by the length L of the object 200. t The length L of the object 200 is specified as t is the length in a direction substantially parallel to the direction of travel of the moving body.

[0079] 13 is a diagram illustrating a method by which the calculation unit 140 according to this embodiment derives the size of the partial region 22. In the example of this figure, steps S302 to S311 are as described in the third embodiment. In the example of this figure, in step S320, the calculation unit 140 derives the length L of the object 200 by the above-described method. t Then, the specified length L t The processes of S302 to S311 are performed using the length L of the object 200. t and the length L of the partial region 22 are the same. In this example, it is not necessary to perform loop processing in which L is changed. Then, when loop processing B ends in S311, the calculation unit 140 calculates the average score for each assumed value of W in S321. Then, the assumed value of W with the highest calculated average value is specified as the width W of the object 200 to be used in parameter calculation. The calculation unit 140 performs parameter calculation using the length L and width W specified in this way.

[0080] The width W may be known. That is, the calculation unit 140 may read and acquire information indicating the width W stored in advance in a storage unit, and use the information together with the value of the length L derived as described above to calculate the parameters.

[0081] As described above, according to this embodiment, the same actions and effects as those of the first embodiment can be obtained. In addition, according to this embodiment, the calculation unit 140 identifies the size of the object 200 using multiple images. Therefore, even if the size of the object 200 is unknown, the parameters can be calculated.

[0082] (Fifth embodiment) 14 is a diagram illustrating the configuration of a processing device 10 according to a fifth embodiment. The processing device 10 according to this embodiment is the same as the processing device 10 according to at least one of the first to fourth embodiments, except that it further includes a transformation unit 160 that performs projective transformation on a target image using calculated parameters. This will be described in detail below.

[0083] In this embodiment, the transformation unit 160 may determine a target image to be projectively transformed. For example, the transformation unit 160 may perform projective transformation on the partial region 22, expand the projectively transformed partial region 22, perform inverse projective transformation on the expanded partial region 22, and determine the target image using the result of the inverse transformation.

[0084] For example, when generating an orthoimage of a road surface from an image taken with an in-vehicle camera, it is necessary to determine the range within the captured image to apply projective transformation. However, the appropriate range for projective transformation varies depending on the camera's mounting position, angle, and road surface inclination, etc. By appropriately specifying the transformation range depending on the image, a highly accurate orthoimage can be obtained.

[0085] FIG. 15 is a diagram illustrating a method by which the transformation unit 160 determines the target image 30 and performs projective transformation on the target image 30. The transformation unit 160 performs projective transformation on the partial region 22 using the parameters calculated by the calculation unit 140. The projectively transformed partial region 22 is then expanded by a predetermined magnification. In the example shown in this figure, the transformation unit 160 expands the projectively transformed partial region 22 in the upward and left-right directions of the image. The transformation unit 160 identifies an outer boundary line 220 of the expanded partial region 22. If the partial region 22 is a polygon, the outer boundary line 220 of the expanded partial region 22 can be identified by the coordinates indicating the vertices of the line 220. Next, the transformation unit 160 inversely transforms the expanded partial region 22. That is, the inversely transformed line 220 is identified by calculating the coordinates of its vertices. Next, the transformation unit 160 determines the portion of the reference image 20 that is inside the inversely transformed line 220 as the target image 30. The transformation unit 160 then performs projective transformation on the determined target image 30 to obtain an orthoimage. If the line 220 after inverse transformation extends beyond the reference image 20, a predetermined value is set for the pixels of the extending portion. In this way, the transformation unit 160 can generate an orthoimage.

[0086] The expansion magnification when expanding the partial region 22 that has been subjected to projective transformation can be determined in advance based on, for example, the number of lanes on the photographed road.

[0087] Note that the target image 30 is not limited to the above example. The transformation unit 160 can perform projective transformation on any image as the target image 30. However, the target image 30 may include, for example, a road. It is preferable that the target image 30 and the reference image 20 are images obtained using the same camera of the same moving object. It is also preferable that the reference image 20 and the target image 30 constitute the same video.

[0088] 15, the target image 30 is a part of the reference image 20. In other words, the reference image 20 and the target image 30 are different images. However, the reference image 20 and the target image 30 may be the same image. Furthermore, the conversion unit 160 may use the partial region 22 as the target image 30 without expanding it.

[0089] The transformation unit 160 can perform projective transformation of the target image 30 using an existing method, using the parameters calculated by the calculation unit 140. The transformation unit 160 may further generate a plurality of orthoimages from a plurality of target images 30, align the generated orthoimages based on sensor information acquired from a moving object, and integrate them into a single image.

[0090] The hardware configuration of a computer that realizes the processing device 10 according to this embodiment is, for example, shown in Fig. 7, similar to the first embodiment. However, a program module that realizes the function of a conversion unit 160 is further stored in the storage device 1080 of the computer 1000 that realizes the processing device 10 of this embodiment.

[0091] 16 is a flowchart illustrating the flow of processing performed by the processing device 10 according to this embodiment. The processing method according to this embodiment includes an acquisition step S10, a calculation step S30, and a transformation step S40. In the transformation step S40, the target image is subjected to projective transformation using the calculated parameters.

[0092] As described above, according to this embodiment, the same actions and effects as those of the first embodiment can be obtained. In addition, according to this embodiment, the processing device 10 further includes a transformation unit 160 that performs projective transformation on the target image using the calculated parameters. Therefore, a highly accurate orthoimage can be obtained regardless of the mounting direction or angle of the camera.

[0093] Although the embodiments and examples have been described above with reference to the drawings, these are merely examples of the present invention, and various configurations other than those described above can also be adopted. Below, examples of reference forms are added. 1. An acquisition unit that acquires a reference image; a calculation unit that calculates parameters for projective transformation based on the shape and size of the object in the reference image. Processing equipment. 2. In the processing device described in 1., the object is a mark on the ground, The calculation unit extracting a partial area including the mark from the reference image; The parameter is calculated based on the mark included in the partial area. Processing equipment. 3. In the processing device according to 1. or 2., The apparatus further includes a transformation unit that performs projective transformation on a target image using the calculated parameters. Processing equipment. 4. In the processing device described in 2., a transformation unit that performs projective transformation on a target image using the calculated parameters; The conversion unit projectively transforming the subregion; Expanding the projectively transformed subregion; The expanded subregion is subjected to an inverse projective transformation; The result of the inverse transformation is used to determine the target image. Processing equipment. 5. In the processing device according to 3. or 4., The reference image and the target image are different images. Processing equipment. 6. In the processing apparatus according to any one of 1. to 5., The object is a dividing line painted on the road. Processing equipment. 7. In the processing apparatus according to any one of 1. to 6., the acquisition unit acquires a plurality of images including the reference image, a selection unit that selects the reference image from the plurality of images acquired by the acquisition unit Processing equipment. 8. In the processing device according to 7., The selection unit selects the reference image based on a position of the object in the image. Processing equipment. 9. In the processing device according to 7. or 8., the plurality of images are images obtained by an imaging device attached to a moving object, The selection unit selects the reference image based on the situation of the moving object when the image was obtained. Processing equipment. 10. In the processing apparatus according to any one of 7. to 9., The selection unit selects the reference image based on a result of edge detection performed on the image. Processing equipment. 11. In the processing apparatus according to any one of items 7 to 10, The plurality of images are associated with information indicating the photographing positions of the images, The selection unit selects an image whose photographing position satisfies a predetermined condition as the reference image. Processing equipment. 12. In the processing apparatus according to any one of 1. to 11., The calculation unit acquires size information indicating the size of the object in real space, and calculates the parameter using the size information. Processing equipment. 13. In the processing apparatus according to any one of 1. to 11., the acquisition unit acquires a plurality of images that constitute a moving image, including the reference image; the video is a video obtained by an imaging device attached to a moving object, The calculation unit calculating a size of the object in real space using the moving image and the speed of the moving object when the moving image was obtained; The parameters are calculated using the size of the object in real space. Processing equipment. 14. An acquisition step of acquiring a reference image; and calculating parameters for projective transformation based on the shape and size of the object in the reference image. Processing method. 15. A program that causes a computer to execute each step of the processing method described in 14. [Explanation of symbols]

[0094] 10 Processing equipment 20 Reference Images 22 Partial area 30 Target Images 40 Mobile 42 Camera 50 server machines 120 Acquisition Department 130 Selection Section 140 Calculation Unit 160 Conversion Unit 200 objects 210 Edge 1000 calculator

Claims

1. An acquisition unit that acquires a reference image; a calculation unit that calculates parameters for projective transformation based on the shape and size of the object in the reference image; a transformation unit that performs projective transformation on a target image using the calculated parameters, The calculation unit extracting a partial region including the object from the reference image; Calculating vertex coordinates of the partial region before and after projective transformation based on the size of the object in real space; The parameters are calculated using the vertex coordinates before and after the projective transformation. Processing equipment.

2. an acquisition unit that acquires a reference image; a calculation unit that calculates parameters for projective transformation based on the shape and size of the object in the reference image; a transformation unit that performs projective transformation on a target image using the calculated parameters, the calculation unit extracts a partial region including the object from the reference image; The conversion unit projectively transforming the subregion; Expanding the projectively transformed subregion; The expanded subregion is subjected to an inverse projective transformation; The result of the inverse transformation is used to determine the target image. Processing equipment.

3. 3. The processing apparatus according to claim 1, The reference image and the target image are different images. Processing equipment.

4. an acquisition unit that acquires a plurality of images that constitute a moving image, including a reference image; a calculation unit that calculates parameters for projective transformation based on the shape and size of the object in the reference image, the video is a video obtained by an imaging device attached to a moving object, The calculation unit calculating a size of the object in real space using the moving image and the speed of the moving object when the moving image was obtained; The parameters are calculated using the size of the object in real space. Processing equipment.

5. An acquisition unit that acquires a plurality of images including a reference image; a selection unit that selects the reference image from the plurality of images acquired by the acquisition unit; a calculation unit that calculates parameters for projective transformation based on the shape and size of the object in the reference image, The calculation unit extracting a partial region including the object from the reference image; Calculating vertex coordinates of the partial region before and after projective transformation based on the size of the object in real space; The parameters are calculated using the vertex coordinates before and after the projective transformation. Processing equipment.

6. 6. The processing apparatus according to claim 5, The plurality of images are associated with information indicating the photographing positions of the images, The selection unit selects an image whose photographing position satisfies a predetermined condition as the reference image. Processing equipment.

7. an acquisition unit that acquires a plurality of images including a reference image; a selection unit that selects the reference image from the plurality of images acquired by the acquisition unit; a calculation unit that calculates parameters for projective transformation based on the shape and size of the object in the reference image, The selection unit selects the reference image based on a position of the object in the image. Processing equipment.

8. an acquisition unit that acquires a plurality of images including a reference image; a selection unit that selects the reference image from the plurality of images acquired by the acquisition unit; a calculation unit that calculates parameters for projective transformation based on the shape and size of the object in the reference image, the plurality of images are images obtained by an imaging device attached to a moving object, The selection unit selects the reference image based on the situation of the moving object when the image was obtained. Processing equipment.

9. an acquisition unit that acquires a plurality of images including a reference image; a selection unit that selects the reference image from the plurality of images acquired by the acquisition unit; a calculation unit that calculates parameters for projective transformation based on the shape and size of the object in the reference image, The selection unit selects the reference image based on a result of edge detection performed on the image. Processing equipment.

10. In the processing apparatus according to any one of claims 1 to 3 and 7 to 9, The calculation unit acquires size information indicating the size of the object in real space, and calculates the parameter using the size information. Processing equipment.

11. The processing apparatus according to any one of claims 1 to 10, The object is a mark on the ground. Processing equipment.

12. The processing apparatus according to any one of claims 1 to 11, The object is a dividing line painted on the road. Processing equipment.

13. an acquisition step of acquiring a reference image; a calculation step of calculating parameters for projective transformation based on the shape and size of the object in the reference image; a transformation step of projectively transforming a target image using the calculated parameters, In the calculation step, extracting a partial region including the object from the reference image; Calculating vertex coordinates of the partial region before and after projective transformation based on the size of the object in real space; The parameters are calculated using the vertex coordinates before and after the projective transformation. Processing method.

14. an acquisition step of acquiring a reference image; a calculation step of calculating parameters for projective transformation based on the shape and size of the object in the reference image; a transformation step of projectively transforming a target image using the calculated parameters, In the calculation step, a partial region including the object is extracted from the reference image; In the converting step, projectively transforming the subregion; Expanding the projectively transformed subregion; The expanded subregion is subjected to an inverse projective transformation; The result of the inverse transformation is used to determine the target image. Processing method.

15. an acquisition step of acquiring a plurality of images including a reference image and constituting a moving image; a calculation step of calculating parameters for projective transformation based on the shape and size of the object in the reference image, the video is a video obtained by an imaging device attached to a moving object, In the calculation step, calculating a size of the object in real space using the moving image and the speed of the moving object when the moving image was obtained; The parameters are calculated using the size of the object in real space. Processing method.

16. acquiring a plurality of images including a reference image; a selection step of selecting the reference image from the plurality of images acquired in the acquisition step; a calculation step of calculating parameters for projective transformation based on the shape and size of the object in the reference image, In the selecting step, the reference image is selected based on the position of the object in the image. Processing method.

17. acquiring a plurality of images including a reference image; a selection step of selecting the reference image from the plurality of images acquired in the acquisition step; a calculation step of calculating parameters for projective transformation based on the shape and size of the object in the reference image, the plurality of images are images obtained by an imaging device attached to a moving object, In the selection step, the reference image is selected based on the situation of the moving object when the image was acquired. Processing method.

18. acquiring a plurality of images including a reference image; a selection step of selecting the reference image from the plurality of images acquired in the acquisition step; a calculation step of calculating parameters for projective transformation based on the shape and size of the object in the reference image, In the selection step, the reference image is selected based on the result of edge detection performed on the image. Processing method.

19. A program that causes a computer to execute each step of the processing method according to any one of claims 13 to 18.

Images