Processing device, processing method, and program

Abstract

To suppress deterioration in object detection accuracy while reducing a calculation amount required for object detection.SOLUTION: An image processing device 10 includes: a correspondence information acquisition unit that acquires correspondence information indicating a correspondence relationship between an image capturing surface of an image in which a first area is captured and a plane that includes the first area; a calibration information acquisition unit that acquires a plurality of coordinates in the image; a three-dimensional coordinate transformation unit that obtains a plurality of coordinates after a first transformation by transforming the plurality of coordinates in the image to coordinates respectively corresponding to the plurality of coordinates in the plane based on the correspondence information; a coordinate changing unit that calculates a plurality of new coordinates obtained by changing the plurality of coordinates after the transformation; a two-dimensional coordinate inverse transformation unit that transforms the plurality of new coordinates into coordinates respectively corresponding to the plurality of new coordinates on the image capturing surface based on the correspondence information and obtains a plurality of coordinates after a second transformation; a detection area setting unit that generates a second area including the plurality of coordinates after the transformation as a detection area for detecting an object; and a display control unit that controls display of the detection area onto a display unit.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a processing device, a processing method, and a program. [Background technology]

[0002] In recent years, a technology has been known that detects an object present in a predetermined area based on an image of the predetermined area captured by a camera (hereinafter, also referred to as a "camera image"). For example, the predetermined area may be a road, and the object may be a vehicle moving on the road. As an example, a technology has been disclosed that detects a vehicle present on the road from a camera image of the road and estimates the size of the detected vehicle (see, for example, Patent Document 1).

[0003] In this technology, road plane parameters are estimated by a calibration process based on road information such as the distance between markers (feature points) on the road, and the size of the vehicle is estimated based on the estimated road plane parameters and camera information. For example, the road plane parameters include correspondence information that indicates the three-dimensional correspondence between the imaging surface of the camera and the road.

[0004] Also disclosed is a technique for assisting and correcting input of distances between markers (feature points) on a road used in the calibration process (see, for example, Patent Document 2). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 5899944 [Patent Document 2] Patent No. 5846140 Summary of the Invention [Problem to be solved by the invention]

[0006] In order to reduce the amount of calculation required for object detection, a target area is set in advance in a part of the camera image, and it is sometimes determined whether an object exists within the set target area. However, since a situation may arise in which only a part of the object to be detected falls within the target area, the accuracy of detecting an object within the target area does not necessarily improve.

[0007] Therefore, the present invention has been made in consideration of the above problems, and an object of the present invention is to provide a technology that can suppress a decrease in object detection accuracy while reducing the amount of calculation required for object detection. [Means for solving the problem]

[0008] In order to solve the above problem, according to one aspect of the present invention, a display device includes an image acquisition unit that acquires an image in which a first area is captured, a correspondence information acquisition unit that acquires correspondence information indicating a correspondence relationship between an imaging surface of the image and a plane including the first area, a coordinate information acquisition unit that acquires a plurality of coordinates in the image, a first conversion unit that converts the plurality of coordinates in the image into coordinates corresponding to each of the plurality of coordinates in the plane based on the correspondence information to obtain a plurality of coordinates after a first conversion, a coordinate change unit that calculates new a plurality of coordinates by changing the plurality of coordinates after the first conversion, a second conversion unit that converts the new a plurality of coordinates into coordinates corresponding to each of the new a plurality of coordinates on the imaging surface based on the correspondence information to obtain a plurality of coordinates after a second conversion, an area setting unit that generates a second area including the plurality of coordinates after the second conversion as a detection area for detecting an object, and a display control unit that controls display of the detection area on a display unit. the coordinate information acquisition unit recognizes a predetermined area through which the object may pass based on the image, and acquires a plurality of coordinates in the image based on the predetermined area; A processing device is provided. 。

[0009] According to another aspect of the present invention, there is provided a processing method of a processing device, the method comprising the steps of: acquiring an image in which a first area is captured; acquiring correspondence information indicating a correspondence relationship between an imaging surface of the image and a plane including the first area; acquiring a plurality of coordinates in the image; converting the plurality of coordinates in the image into coordinates corresponding to each of the plurality of coordinates in the plane based on the correspondence information to obtain a plurality of coordinates after first conversion; calculating new a plurality of coordinates by changing the plurality of coordinates after first conversion; converting the new a plurality of coordinates into coordinates corresponding to each of the new a plurality of coordinates on the imaging surface based on the correspondence information to obtain a plurality of coordinates after second conversion; generating a second area including the plurality of coordinates after second conversion as a detection area for detecting an object; and controlling display of the detection area on a display unit. the step of acquiring a plurality of coordinates in the image includes: recognizing a predetermined area through which the object may pass based on the image; and acquiring a plurality of coordinates in the image based on the predetermined area. A processing method is provided. 。

[0010] According to another aspect of the present invention, a computer is caused to function as an image acquisition unit that acquires an image in which a first area is captured, a correspondence information acquisition unit that acquires correspondence information indicating a correspondence relationship between an imaging surface of the image and a plane including the first area, a coordinate information acquisition unit that acquires a plurality of coordinates in the image, a first conversion unit that converts a plurality of coordinates in the image into coordinates corresponding to each of the plurality of coordinates in the plane based on the correspondence information to obtain a plurality of coordinates after a first conversion, a coordinate change unit that calculates new a plurality of coordinates by changing the plurality of coordinates after the first conversion, a second conversion unit that converts the new a plurality of coordinates into coordinates corresponding to each of the new a plurality of coordinates on the imaging surface based on the correspondence information to obtain a plurality of coordinates after a second conversion, an area setting unit that generates a second area including the plurality of coordinates after the second conversion as a detection area for detecting an object, and a display control unit that controls display of the detection area on a display unit. the coordinate information acquisition unit recognizes a predetermined area through which the object may pass based on the image, and acquires a plurality of coordinates in the image based on the predetermined area; The program is offered 。 Effect of the Invention

[0011] As described above, according to the present invention, a technique is provided that can suppress a decrease in object detection accuracy while reducing the amount of calculation required for object detection. [Brief description of the drawings]

[0012] [Figure 1] 1 is a block diagram showing an example of a functional configuration of an image processing apparatus according to a first embodiment of the present invention. [Diagram 2] 4A and 4B are diagrams illustrating examples of camera images acquired by an image acquisition unit. [Diagram 3] FIG. 2 is a diagram for explaining a coordinate system according to an embodiment of the present invention. [Figure 4] 11 is a diagram for explaining an example of conversion from two-dimensional coordinates in a camera image to three-dimensional coordinates on a road plane. FIG. [Diagram 5] FIG. 11 is a diagram for explaining an example of extension of three-dimensional coordinates. [Figure 6] 10A and 10B are diagrams for explaining an example of inverse transformation from three-dimensional coordinates on a road plane to two-dimensional coordinates in a camera image. [Figure 7] FIG. 13 is a diagram showing an example of two-dimensional coordinates after expansion. [Figure 8] FIG. 13 is a diagram illustrating an example of a detection area. [Figure 9] 11 is a flowchart showing an example of the operation of the image processing device according to the embodiment. [Figure 10] FIG. 11 is a block diagram showing an example of a functional configuration of an image processing apparatus according to a second embodiment of the present invention. [Figure 11] 4A and 4B are diagrams illustrating examples of camera images acquired by an image acquisition unit. [Figure 12] 11 is a diagram showing an example of a plurality of two-dimensional coordinates in a camera image acquired by a coordinate information acquisition unit. FIG. [Figure 13] 11 is a flowchart showing an example of the operation of the image processing device according to the embodiment. [Figure 14] FIG. 13 is a diagram for explaining the extension of three-dimensional coordinates in the first modified example. [Figure 15]FIG. 13 is a diagram showing an example of two-dimensional coordinates after expansion according to the first modified example. [Figure 16] FIG. 13 is a diagram showing an example of a detection area according to a first modified example. [Figure 17] FIG. 13 is a diagram for explaining the extension of three-dimensional coordinates in the second modified example. [Figure 18] 1 is a diagram illustrating a hardware configuration of an information processing apparatus as an example of an image processing apparatus according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are designated by the same reference numerals, and duplicated explanations will be omitted.

[0014] In this specification and drawings, multiple components having substantially the same functional configuration are distinguished by adding different numbers after the same reference numeral. Similar components in different embodiments are distinguished by adding different alphabets after the same reference numeral. However, when there is no particular need to distinguish between multiple components having substantially the same functional configuration, only the same reference numerals are used.

[0015] <0.Background> In recent years, a technology has been known that detects an object present in a predetermined area (first area) based on a camera image of the predetermined area captured by a camera. Here, in order to reduce the amount of calculation required for object detection, a target area may be set in advance in a part of the camera image, and it may be determined whether an object exists in the set target area.

[0016] As an example, it is assumed that object detection is performed by a neural network with a limited input size. In such a case, the image input to the neural network may be limited to only a target area that is a part of the camera image, rather than the entire camera image. For example, by setting only a part of the camera image as the target area, an area in which the size of an object captured in the camera image is reduced and the object is crushed, making object detection difficult, may be excluded from the target area.

[0017] However, since a situation may occur in which only a part of an object to be detected falls within the target area, the accuracy of detecting an object within the target area does not necessarily improve. Therefore, in an embodiment of the present invention, a technique is mainly proposed that can suppress a decrease in the accuracy of detecting an object while reducing the amount of calculation required for object detection.

[0018] In the following, it is mainly assumed that the area captured by the camera is a road and the detected object is a vehicle moving on the road. However, the detected object is not limited to a vehicle. For example, the detected object may be an animal other than a vehicle (e.g., a person, etc.). Furthermore, the area captured by the camera is not limited to a road. For example, the area captured by the camera may be appropriately determined according to the range in which the detected object can move.

[0019] More specifically, when a target area is set in advance, it may be assumed that the set target area does not correspond to the movement of the vehicle. Specifically, when the road surface on which the vehicle moves is set as the target area, the size of the vehicle (e.g., the height, width, and length of the vehicle) is not taken into consideration when setting the target area. Therefore, a part of the vehicle may extend outside the target area, and the vehicle may not be detected normally.

[0020] In the following, the height of the vehicle will be simply referred to as "vehicle height", the width of the vehicle will be simply referred to as "vehicle width", and the length of the vehicle will be simply referred to as "vehicle length".

[0021] Furthermore, there are cases where advanced processing is performed, such as obtaining features according to the detection position of the vehicle from each of a plurality of captured images obtained continuously in a time series, and linking identical or similar features to calculate the vehicle's trajectory. Even in such cases, since a part of the vehicle may extend outside the target area, the features obtained from the target area may not be similar even though the images show the same vehicle. This may cause a situation in which the calculation result of the vehicle's trajectory deviates from the actual vehicle's trajectory.

[0022] Therefore, in an embodiment of the present invention, a technology is mainly proposed in which multiple coordinates defining a target area are expanded based on calibration information (e.g., road plane parameters) obtained by a calibration process, and a detection area is set based on the expanded multiple coordinates.

[0023] According to this technology, only a part of the entire camera image is set as the detection area, and the possibility that a part of the vehicle will protrude from the detection area is reduced. Therefore, according to this technology, it is possible to suppress a decrease in the detection accuracy of the vehicle while reducing the amount of calculation required for vehicle detection. For example, when expanding the coordinates, the size of the vehicle (e.g., the height, width, and length of the vehicle), the traveling direction of the vehicle, and the like can be taken into consideration. This is expected to further suppress a decrease in the detection accuracy of the object.

[0024] The background of the embodiments of the present invention has been described above.

[0025] <1. First embodiment> First, a first embodiment of the present invention will be described.

[0026] [1-1. Overview of image processing device] First, an example of the functional configuration of an image processing device (processing device) according to a first embodiment of the present invention will be described. FIG. 1 is a block diagram showing an example of the functional configuration of an image processing device according to a first embodiment of the present invention. As shown in FIG. 1, an image processing device 10 according to the first embodiment of the present invention includes an imaging unit 210, an input unit 220, an image acquisition unit 110, an area coordinate acquisition unit 120, a calibration information acquisition unit 130, a display processing unit 140, and a display unit 230. The image processing device 10 is realized by a computer.

[0027] The imaging unit 210 is configured by a camera having an image sensor that captures an image. The imaging unit 210 obtains a camera image by capturing an image of an area (first area) included in a road plane as an example of a plane. In the example shown in FIG. 1, the imaging unit 210 exists inside the image processing device 10. However, the imaging unit 210 may exist outside the image processing device 10. The camera image captured by the imaging unit 210 is provided to the image acquisition unit 110.

[0028] The input unit 220 is configured by an input device that accepts a user's operation. For example, the input unit 220 accepts an operation (click operation) of designating coordinates in a camera image as an example of a user's operation. Here, a case where the input unit 220 is realized by a mouse is mainly assumed. However, the input unit 220 may be realized by an input device other than a mouse (for example, a touch panel, etc.). In the example shown in FIG. 1, the input unit 220 exists inside the image processing device 10. However, the input unit 220 may exist outside the image processing device 10. The user's operation accepted by the input unit 220 is provided to the area coordinate acquisition unit 120.

[0029] The image acquisition unit 110, the area coordinate acquisition unit 120, the calibration information acquisition unit 130, and the display processing unit 140 may be realized by a control unit (not shown). The display processing unit 140 includes a three-dimensional coordinate conversion unit 141, a coordinate change unit 142, a two-dimensional coordinate inverse conversion unit 143, a detection area setting unit 144, and a display control unit 145. The detailed functions of these blocks will be described later.

[0030] The control unit (not shown) includes a CPU (Central Processing Unit) and the like, and its functions can be realized by the CPU expanding a program stored in a non-volatile storage device into a RAM (Random Access Memory) and executing it. At this time, a computer-readable recording medium on which the program is recorded can also be provided. Alternatively, the control unit (not shown) may be configured with dedicated hardware, or may be configured with a combination of multiple hardware components.

[0031] The display unit 230 is configured with a display. For example, the display unit 230 displays various information under the control of the display processing unit 140. The various information displayed by the display unit 230 is visually recognized by a user. In the example shown in FIG. 1, the display unit 230 exists inside the image processing device 10. However, the display unit 230 may exist outside the image processing device 10.

[0032] Additionally, the image processing device 10 includes a storage unit (not shown). The storage unit (not shown) is a storage device capable of storing programs and data for operating a control unit (not shown). The storage unit (not shown) can also temporarily store various data required in the course of operation of the control unit (not shown). For example, the storage device may be a non-volatile storage device.

[0033] For example, as described above, when the input unit 220 and the display unit 230 are present outside the image processing device 10, the input unit 220 and the display unit 230 may be provided in a terminal device connected to the image processing device 10 wirelessly or by wire. In this case, the type of the terminal device is not limited. For example, the type of the terminal device may be a tablet terminal, a smartphone, or another terminal device (for example, a PC (Personal Computer)).

[0034] An example of the functional configuration of the image processing device 10 according to the first embodiment of the present invention has been described above.

[0035] [1-2. Details of each function] Next, a detailed description will be given of each function of the image processing device 10. In particular, a detailed description will be given of each function of the image acquisition unit 110, the area coordinate acquisition unit 120, the calibration information acquisition unit 130, and the display processing unit 140 (three-dimensional coordinate conversion unit 141, coordinate change unit 142, two-dimensional coordinate inverse conversion unit 143, detection area setting unit 144, and display control unit 145).

[0036] (Image acquisition unit 110) The image acquisition unit 110 acquires, from the imaging unit 210, a camera image obtained by imaging an area (first area) included in a road plane, which is an example of a plane, by the imaging unit 210. Fig. 2 is a diagram showing an example of a camera image acquired by the image acquisition unit 110. With reference to Fig. 2, a camera image G10 is shown as an example of a camera image acquired by the image acquisition unit 110.

[0037] (Calibration information acquisition unit 130) The calibration information acquisition unit 130 acquires correspondence information indicating the correspondence relationship between the imaging surface of the camera image G10 and a road plane including an area imaged by the imaging unit 210. In the following description, parameters (hereinafter also referred to as "road plane parameters") constituting an equation (hereinafter also referred to as "road plane equation") indicating the three-dimensional position of the road plane based on the imaging surface in real space will be taken as an example of such correspondence information.

[0038] The road plane parameters may be calibration information obtained by a calibration process executed when the imaging unit 210 is installed. The calibration process is described in detail in the above-mentioned Patent Document 1, so a detailed description of the calibration process will be omitted here. Additionally, the calibration information acquisition unit 130 acquires camera information from the imaging unit 210. The camera information may be the focal length of the imaging unit 210.

[0039] FIG. 3 is a diagram for explaining a coordinate system according to an embodiment of the present invention. Referring to FIG. 3, an origin O of the coordinate system is shown. Also, a projection plane J is shown. The projection plane J corresponds to the above-mentioned imaging surface. The projection plane J is away from the origin O by a focal length f and is perpendicular to the y-axis in the real space. Also, the left-right direction of the camera image G10 corresponds to the X-axis direction in the camera image G10 and coincides with the x-axis direction in the real space. Furthermore, the up-down direction of the camera image G10 corresponds to the Y-axis direction in the camera image G10 and coincides with the z-axis direction in the real space.

[0040] For example, the origin of the coordinate system of the camera image G10 may be the upper left of the camera image G10, the downward direction of the camera image G10 may be the positive direction of the Y axis, and the rightward direction of the camera image G10 may be the positive direction of the X axis. The length of one graduation of the two-dimensional coordinates (X, Y) in the camera image G10 may be one dot. Meanwhile, the length of one graduation of the three-dimensional coordinates (x, y, z) in the real space may be 1 cm. The correspondence between the two-dimensional coordinates in the camera image and the three-dimensional coordinates of the projection plane J is grasped by the image processing device 10.

[0041] Referring to Figure 3, a road plane existing in real space is shown, and the road plane equation is R1x+R2y+R3z+R4=0. R1, R2, R3, and R4 correspond to road plane parameters. The normal vector of the road plane is expressed as (R1, R2, R3).

[0042] (Area coordinate acquisition unit 120) The area coordinate acquisition unit 120 acquires a plurality of two-dimensional coordinates in the camera image G10. The method of acquiring the plurality of two-dimensional coordinates by the area coordinate acquisition unit 120 is not limited. Here, it is mainly assumed that the display control unit 145 controls the display of the camera image G10 by the display unit 230, the user performs an operation on the camera image G10 displayed on the display unit 230, and the area coordinate acquisition unit 120 acquires a plurality of two-dimensional coordinates in the camera image G10 based on the operation by the user.

[0043] 2, a case is mainly assumed in which a user performs a designation operation (e.g., a click operation) to designate two-dimensional coordinates P11 to P24 on a camera image G10 displayed on a display unit 230, and the area coordinate acquisition unit 120 acquires the two-dimensional coordinates P11 to P24 designated by the user's designation operation as a plurality of two-dimensional coordinates in the camera image G10. An area surrounded by the two-dimensional coordinates P11 to P24 is set as the target area A10. Note that the number of two-dimensional coordinates that the user can designate is not limited as long as it is a plurality of coordinates.

[0044] At this time, it is assumed that the contour of the road surface on which the vehicle moves is specified by the user as two-dimensional coordinates P11 to P24, as shown in Fig. 2. Therefore, it is assumed that the size of the vehicle (e.g., vehicle height, vehicle width, vehicle length, etc.) is not taken into consideration when setting the target area A10 based on the two-dimensional coordinates P11 to P24. Therefore, a part of the vehicle may extend outside the target area A10, and the vehicle may not be detected normally.

[0045] (Three-dimensional coordinate conversion unit 141) The three-dimensional coordinate conversion unit 141 functions as a first conversion unit that converts the two-dimensional coordinates P11 to P24 in the camera image G10 into three-dimensional coordinates on the road plane corresponding to the two-dimensional coordinates P11 to P24, respectively, based on the road plane parameters R1, R2, R3, and R4. As a result, the three-dimensional coordinate conversion unit 141 obtains a plurality of three-dimensional coordinates after conversion (a plurality of three-dimensional coordinates after the first conversion). More specifically, the three-dimensional coordinate conversion unit 141 converts the two-dimensional coordinates P11 to P24 into three-dimensional coordinates on the road plane corresponding to the two-dimensional coordinates P11 to P24, respectively, based on the road plane parameters R1, R2, R3, and R4 and the focal length f.

[0046] FIG. 4 is a diagram for explaining an example of conversion from two-dimensional coordinates in a camera image to three-dimensional coordinates on a road plane. Referring to FIG. 4, a point U(x, y, z) on the road plane is shown. Here, the point U(x, y, z) on the road plane is an intersection point between the road plane and a straight line connecting the origin O and a point U'(x', f, z') on the camera image G10. In other words, based on the road plane equation, the focal length, and the three-dimensional coordinates of the point U on the camera image G10, the point U' The three-dimensional coordinates of a point U on the road plane corresponding to the point U can be calculated.

[0047] More specifically, since point U(x, y, z) is a point on the line connecting origin O and point U'(x', f, z'), U=KU' holds. That is, the following formula (1) holds.

[0048] x = Kx', y = Kf, z = Kz' (1)

[0049] Furthermore, considering that point U is a point on the road plane, point U (x=Kx', y=Kf, z=Kz') satisfies the road plane equation R1x+R2y+R3z+R4=0, so R1Kx'+R2Kf+R3Kz'+R4=0 holds. Transforming this equation gives the following equation (2).

[0050]

number

[0051] From the above, when a point U' (x', f, z') in the camera image G10 is given, the three-dimensional coordinates (x, y, z) of the point U on the road plane corresponding to this point U' are uniquely determined from equations (1) and (2) based on the focal length f and the road plane parameters R1, R2, R3, and R4. In the example shown in Fig. 2, the three-dimensional coordinate conversion unit 141 converts the two-dimensional coordinates P11 to P24 in the camera image G10 into three-dimensional coordinates on the road plane corresponding to the two-dimensional coordinates P11 to P24, respectively.

[0052] (Coordinate change unit 142) The coordinate modification unit 142 calculates new three-dimensional coordinates by modifying the transformed three-dimensional coordinates. As a result, the transformed three-dimensional coordinates are expanded. More specifically, the coordinate modification unit 142 calculates, as new three-dimensional coordinates, the moved three-dimensional coordinates obtained by moving each of the transformed three-dimensional coordinates in one or more moving directions. Here, as an example, a case where the transformed point U(x, y, z) is moved will be described.

[0053] Fig. 5 is a diagram for explaining an example of the expansion of three-dimensional coordinates. Referring to Fig. 5, a point U(x, y, z) after transformation is shown. The foot of the perpendicular line from the origin O to the projection plane J is shown as point D, and the intersection point of the straight line connecting the origin O and point D with the road plane is shown as point D'.

[0054] At this time, it is considered to move point U(x, y, z). The direction and amount of movement of point U are not limited. Here, it is assumed that the direction of movement of point U(x, y, z) includes a total of four directions, including the normal direction of the road plane (one direction) and directions parallel to the road plane (three directions). However, the direction of movement of point U(x, y, z) may include at least one of the normal direction of the road plane and the directions parallel to the road plane.

[0055] The normal direction to the road plane corresponds to the height direction of the vehicle moving on the road plane. On the other hand, the direction parallel to the road plane corresponds to the horizontal direction (e.g., the width direction or the length direction of the vehicle). For example, the direction parallel to the road plane includes a direction perpendicular to the normal direction to the road plane and a predetermined direction.

[0056] Here, as examples of the predetermined direction, the left-right direction (x-axis direction) and the up-down direction (z-axis direction) of the projection plane J are assumed. That is, it is assumed that the direction parallel to the road plane includes one direction perpendicular to the normal direction of the road plane and the left-right direction (x-axis direction) of the projection plane J (i.e., the depth direction of the road plane as viewed from the origin O), and two directions perpendicular to the normal direction of the road plane and the up-down direction (z-axis direction) of the projection plane J (i.e., the left-right direction of the road plane as viewed from the origin O).

[0057] For example, when a vehicle moves from the back to the front as viewed from the origin O, the depth direction of the road plane as viewed from the origin O corresponds to the length direction of the vehicle, and the left-right direction of the road as viewed from the origin O corresponds to the width direction of the vehicle. However, the predetermined direction is not limited to this example.

[0058] For example, the predetermined direction may include a preset traveling direction of the vehicle. In this case, the traveling direction of the vehicle may be used instead of the up-down direction of the projection plane J, and a direction perpendicular to the traveling direction of the vehicle on the road plane may be used instead of the left-right direction of the projection plane J. The traveling direction of the vehicle may be preset by a user's operation, or may be recognized from an image captured in advance by the imaging unit 210.

[0059] More specifically, the point U(x, y, z) moved by N in the normal direction (R1, R2, R3) is defined as V1(x1, y1, z1). In this case, the point V1(x1, y1, z1) after the movement is expressed as in equation (3).

[0060]

number

[0061] Also, let V2 be the point obtained by moving point U(x,y,z) by M in one direction (depth direction of the road plane as seen from origin O) perpendicular to the normal direction (R1,R2,R3) of the road plane and the left-right direction (x-axis direction) of projection plane J. The cross product of the normal vector (R1,R2,R3) and the vector (1,0,0) in the left-right direction (x-axis direction) of projection plane J is calculated as (0,R3,-R2), so the moved point V2(x2,y2,z2) is expressed as in equation (4).

[0062]

number

[0063] Furthermore, points V3 and V4 are obtained by moving point U(x,y,z) by L in two directions (left and right directions of the road plane as seen from the origin O) perpendicular to the normal direction (R1,R2,R3) of the road plane and the up and down direction (z-axis direction) of the projection plane J. The cross product of the normal vector (R1,R2,R3) and the vector (0,0,1) in the up and down direction (z-axis direction) of the projection plane J is calculated as (R2,-R1,0), so the points V3(x3,y3,z3) and V4(x4,y4,z4) after the movement are expressed as in equations (5) and (6).

[0064]

number

[0065] The specific values ​​of the movement amounts N, M, and L are not limited. However, the movement amount N corresponds to the movement amount in the normal direction (which may correspond to the height direction of the vehicle), the movement amount M corresponds to the movement amount in the depth direction of the road (which may correspond to the length direction of the vehicle), and the movement amount L corresponds to the movement in the left-right direction of the road (which may correspond to the width direction of the vehicle). Generally, it is expected that the dimensions of a vehicle are such that the vehicle length is the largest, the vehicle height is the next largest, and the vehicle width is the smallest, so the movement amount M may be the largest, the movement amount N the next largest, and the movement amount L the smallest. Alternatively, the movement amounts N, M, and L may be changeable by a user operation.

[0066] (2D coordinate inverse conversion unit 143) The two-dimensional coordinate inverse conversion unit 143 functions as a second conversion unit that converts the new multiple three-dimensional coordinates calculated by the coordinate change unit 142 into two-dimensional coordinates in the camera image G10 corresponding to each of the new multiple three-dimensional coordinates based on the road plane parameters R1, R2, R3, and R4. In this way, the two-dimensional coordinate inverse conversion unit 143 obtains multiple two-dimensional coordinates after conversion (multiple two-dimensional coordinates after the second conversion). More specifically, the two-dimensional coordinate inverse conversion unit 143 converts the new multiple three-dimensional coordinates into two-dimensional coordinates in the camera image G10 based on the road plane parameters R1, R2, R3, and R4 and the focal length f.

[0067] Fig. 6 is a diagram for explaining an example of inverse transformation from three-dimensional coordinates on the road plane to two-dimensional coordinates in the camera image. Referring to Fig. 6, a point U(x, y, z) on the road plane is shown. Also shown is a point U'(x', f, z') in the camera image G10 corresponding to the point U(x, y, z). That is, based on the road plane equation, the focal length, and the three-dimensional coordinates of the point U on the road plane, the two-dimensional coordinates C(X, Y) of the point U' in the camera image G10 corresponding to the point U on the road plane can be calculated.

[0068] More specifically, from the above formula (1), U'=U / K holds. That is, based on this formula and the above formula (2), the two-dimensional coordinate inverse conversion unit 143 can calculate the three-dimensional coordinates (x, y, z) of point U on the road plane and point U'(x', f, z') in the camera image G10. In the example shown in Fig. 2, after the two-dimensional coordinates P11 to P24 in the camera image G10 are converted by the three-dimensional coordinate conversion unit 141 into three-dimensional coordinates on the road plane, the two-dimensional coordinate inverse conversion unit 143 converts the converted three-dimensional coordinates into two-dimensional coordinates in the camera image G10.

[0069] Fig. 7 is a diagram showing an example of two-dimensional coordinates after expansion. Referring to Fig. 7, two-dimensional coordinates P11, P12, P20, P22, and P23 acquired by the area coordinate acquisition unit 120 are shown. Note that, for ease of viewing, two-dimensional coordinates P13 to P19, P21, and P24 (Fig. 2) are omitted from Fig. 7. Furthermore, referring to Fig. 7, two-dimensional coordinates W11-1 to W11-4 after inverse transformation corresponding to the two-dimensional coordinate P11 in the camera image G10 are shown.

[0070] More specifically, the two-dimensional coordinates W11-1 to W11-4 are obtained by converting the two-dimensional coordinate P11 in the camera image G10 into a three-dimensional coordinate U(x, y, z) on the road plane, moving the converted three-dimensional coordinate to V1 to V4 (Figure 5), and then inversely converting the moved three-dimensional coordinates V1 to V4 into two-dimensional coordinates in the camera image G10.

[0071] In the same manner, two-dimensional coordinates W12-1 to W12-4 after inverse transformation corresponding to the two-dimensional coordinate P12, two-dimensional coordinates W20-1 to W20-4 after inverse transformation corresponding to the two-dimensional coordinate P20, two-dimensional coordinates W22-1 to W22-4 after inverse transformation corresponding to the two-dimensional coordinate P22, and two-dimensional coordinates W23-1 to W23-4 after inverse transformation corresponding to the two-dimensional coordinate P23 are obtained. Although not shown in the figure, two-dimensional coordinates after inverse transformation corresponding to each of the two-dimensional coordinates P13 to P19, P21, and P24 (FIG. 2) are also obtained.

[0072] (Detection area setting unit 144) The detection area setting unit 144 generates an area (second area) including the multiple two-dimensional coordinates after conversion by the two-dimensional coordinate inverse conversion unit 143 as a detection area for detecting a vehicle. FIG. 8 is a diagram showing an example of a detection area. Referring to FIG. 8, among the multiple two-dimensional coordinates after conversion by the two-dimensional coordinate inverse conversion unit 143, the two-dimensional coordinates W11-1 and W12-1 are located at the top, the two-dimensional coordinate W20-4 is located at the rightmost position, the two-dimensional coordinate W22-3 is located at the bottom, and the two-dimensional coordinate W23-3 is located at the leftmost position. The detection area setting unit 144 sets a detection area B10 in which at least these two-dimensional coordinates are contained.

[0073] (Display control unit 145) The display control unit 145 controls the display of the detection area B10 on the display unit 230. In more detail, when information indicating the position of the detection area B10 (for example, if the detection area B10 is a rectangular area, the coordinates of any vertices of the rectangular area and the length and width of the rectangular area) is output from the detection area setting unit 144, the display control unit 145 controls the display unit 230 to add a predetermined display (a diagonal line display in the example shown in FIG. 8) to the detection area B10 based on the information indicating the position of the detection area B10. For example, the predetermined display may be superimposed on the camera image G10 as shown in FIG. 8.

[0074] As described above, it is also assumed that the display unit 230 is located outside the image processing device 10. For example, it is also assumed that the display unit 230 is provided in a terminal device connected wirelessly or by wire to the image processing device 10. In such a case, the display control unit 145 controls the display of the detection area B10 on the display unit 230 by transmitting information indicating the position of the detection area B10 to the terminal device.

[0075] [1-3. Operation of image processing device 10] Next, an operation example of the image processing device 10 according to the first embodiment of the present invention will be described. Fig. 9 is a flowchart showing an operation example of the image processing device 10 according to the first embodiment of the present invention. As shown in Fig. 9, the image acquisition unit 110 acquires a camera image obtained by capturing an image of an area included in a road plane by the imaging unit 210 from the imaging unit 210 (S11).

[0076] Next, the calibration information acquisition unit 130 acquires the calibration information obtained by the calibration process (S12). More specifically, the calibration information acquisition unit 130 acquires road plane parameters as an example of the calibration information. Furthermore, the calibration information acquisition unit 130 acquires the focal length of the imaging unit 210 as an example of the camera information.

[0077] The area coordinate acquisition unit 120 acquires a plurality of two-dimensional coordinates in the camera image (S13). In the first embodiment of the present invention, the display control unit 145 controls the display of the camera image by the display unit 230, the user performs a designation operation to designate two-dimensional coordinates for the camera image displayed on the display unit 230, and the area coordinate acquisition unit 120 acquires the two-dimensional coordinates designated by the designation operation as a plurality of two-dimensional coordinates in the camera image.

[0078] Next, the three-dimensional coordinate conversion unit 141 converts the two-dimensional coordinates in the camera image into three-dimensional coordinates on the road plane based on the calibration information (S14). More specifically, the three-dimensional coordinate conversion unit 141 converts the two-dimensional coordinates in the camera image into three-dimensional coordinates on the road plane based on the road plane parameters R1, R2, R3, R4 and the focal length f.

[0079] The coordinate conversion unit 142 converts the converted three-dimensional coordinates (S15). This results in the calculation of new three-dimensional coordinates. The two-dimensional coordinate inverse conversion unit 143 then converts the new three-dimensional coordinates calculated by the coordinate conversion unit 142 into two-dimensional coordinates in the camera image corresponding to the new three-dimensional coordinates, based on the road plane parameters R1, R2, R3, and R4 (S16).

[0080] If there are any unaltered three-dimensional coordinates remaining ("NO" in S17), the operation proceeds to S14, but if all three-dimensional coordinates have been changed ("YES" in S17), the operation proceeds to S18.

[0081] The detection area setting unit 144 sets an area (e.g., a rectangular area) including the multiple two-dimensional coordinates after the inverse conversion by the two-dimensional coordinate inverse conversion unit 143 as a detection area in which to detect a vehicle (S18). The display control unit 145 outputs information indicating the position of the detection area to the display unit 230 (S19). The display unit 230 attaches a predetermined display to the position of the detection area. This allows the user to recognize the newly set detection area.

[0082] [1-4. Effects of the first embodiment] As described above, according to the image processing device 10 of the first embodiment of the present invention, a new detection area is generated by expanding the target area based on information indicating the three-dimensional position of the road plane. With this configuration, the possibility that a part of the vehicle will go outside the detection area is reduced, so that a situation in which the vehicle is not detected normally is less likely to occur. Therefore, with this configuration, it is possible to suppress a decrease in the detection accuracy of the vehicle while reducing the amount of calculation required for vehicle detection.

[0083] In addition, according to this configuration, even when advanced processing such as calculating the trajectory of the vehicle is performed, the possibility that a part of the vehicle will go outside the detection area is reduced, so that the deterioration of the calculation accuracy of the vehicle trajectory can be suppressed. For example, when expanding the target area, the size of the vehicle (e.g., the height, width, and length of the vehicle), the traveling direction of the vehicle, etc. can be taken into consideration. This can further suppress the deterioration of the detection accuracy of the vehicle.

[0084] The first embodiment of the present invention has been described above.

[0085] <2. Second embodiment> Next, a second embodiment of the present invention will be described.

[0086] [2-1. Overview of image processing device] First, a functional configuration example of an image processing device (processing device) according to a second embodiment of the present invention will be described. Fig. 10 is a block diagram showing a functional configuration example of an image processing device according to a second embodiment of the present invention. As shown in Fig. 10, an image processing device 20 according to the second embodiment of the present invention includes an imaging unit 210, an input unit 220, an image acquisition unit 110, an area coordinate acquisition unit 120, a calibration information acquisition unit 130, a display processing unit 140, and a display unit 230, similar to the image processing device 10 according to the first embodiment of the present invention.

[0087] Additionally, the image processing device 20 according to the second embodiment of the present invention includes an object position acquisition unit 121. The object position acquisition unit 121 may also be realized by a control unit (not shown). The image processing device 20 is realized by a computer. In the following, the object position acquisition unit 121 will be mainly described.

[0088] Fig. 11 is a diagram showing an example of a camera image acquired by image acquisition unit 110. Referring to Fig. 11, a camera image G20 is shown as an example of a camera image acquired by image acquisition unit 110. The user can perform a placement operation (e.g., a drag-and-drop operation) to place an object in camera image G20. Here, it is assumed that a placement operation is performed to place line segment objects H11 to H21 as an example of objects.

[0089] Here, an area surrounded by the line segment objects H11 to H21 is set as the target area A20. Note that the shape of the object that the user can place is not limited to a line segment. For example, the shape of the object that the user can place may be a rectangle. In this case, an area surrounded by rectangular objects may be set as the target area. Also, the number of objects that the user can place is not limited.

[0090] (Object position acquisition unit 121) The object position acquisition unit 121 acquires the positions of the line segment objects H11 to H21 arranged by the arrangement operation. Here, it is mainly assumed that the object position acquisition unit 121 acquires the position of one end of each of the line segment objects H11 to H21. However, the object position acquisition unit 121 may acquire any position of the line segment objects H11 to H21. For example, the object position acquisition unit 121 may acquire the center position of each of the line segment objects H11 to H21.

[0091] The area coordinate acquisition unit 120 acquires a plurality of two-dimensional coordinates in the camera image G20 based on the positions of the line segment objects H11 to H21 arranged by the arrangement operation. More specifically, the area coordinate acquisition unit 120 acquires the positions of the line segment objects H11 to H21 acquired by the object position acquisition unit 121 as a plurality of two-dimensional coordinates in the camera image G20.

[0092] Fig. 12 is a diagram showing an example of a plurality of two-dimensional coordinates in a camera image G20 acquired by the area coordinate acquisition unit 120. In the example shown in Fig. 12, the positions of one ends of line segment objects H11 to H21 are acquired as two-dimensional coordinates P11 to P21. Subsequent operations are executed based on the two-dimensional coordinates P11 to P21 thus acquired by the area coordinate acquisition unit 120. Note that the subsequent operations are executed in the same manner as the corresponding operations in the image processing device 10 according to the first embodiment of the present invention.

[0093] [2-3. Operation of image processing device 20] Next, an operation example of the image processing device 20 according to the second embodiment of the present invention will be described. Fig. 13 is a flowchart showing an operation example of the image processing device 20 according to the second embodiment of the present invention. As shown in Fig. 13, S11 and S12 are executed. These S11 and S12 are executed in the same manner as S11 and S12 (Fig. 9) executed by the image processing device 10 according to the first embodiment of the present invention.

[0094] Next, when the display control unit 145 controls the display unit 230 to display the camera image, the user performs a placement operation to place an object on the camera image displayed on the display unit 230. The object position acquisition unit 121 acquires the position of the object placed by the placement operation (S21). Then, the area coordinate acquisition unit 120 acquires a plurality of two-dimensional coordinates in the camera image based on the position of the object placed by the placement operation (S13).

[0095] Subsequently, steps S14 to S19 are executed in the same manner as steps S14 to S19 (FIG. 9) executed by the image processing device 10 according to the first embodiment of the present invention.

[0096] [2-4. Effects of the second embodiment] As described above, the image processing device 20 according to the second embodiment of the present invention can achieve the same effects as those achieved by the image processing device 10 according to the first embodiment of the present invention. Furthermore, the image processing device 20 according to the second embodiment of the present invention allows a user to place objects along the roadside, and sets an area surrounded by the objects as a target area, making it easier for the user to specify the target area.

[0097] The second embodiment of the present invention has been described above.

[0098] <3. Various modified examples> Next, various modified examples will be described. First, the first modified example will be described. In the above, a case has been mainly described in which the three-dimensional coordinate conversion unit 141 moves the point U(x, y, z) by M in one direction (depth direction of the road plane as viewed from the origin O) perpendicular to the normal direction (R1, R2, R3) of the road plane and the left-right direction (x-axis direction) of the projection plane J, and sets the point V2. However, the three-dimensional coordinate conversion unit 141 may further move the point V2.

[0099] Fig. 14 is a diagram for explaining the extension of three-dimensional coordinates in the first modified example. As shown in Fig. 14, the three-dimensional coordinate conversion unit 141 may obtain a point V2' (x2', y2', z2') by further moving the point V2 in the normal direction (R1, R2, R3) by N. Then, the two-dimensional coordinate inverse conversion unit 143 may convert the point V2' into two-dimensional coordinates instead of the point V2. As in this example, the movement of the three-dimensional coordinates on the road plane is not necessarily limited to the movement in one direction, and may be configured by a combination of movements in multiple directions.

[0100] Fig. 15 is a diagram showing an example of two-dimensional coordinates after expansion according to the first modified example. Referring to Fig. 15, two-dimensional coordinates P11, P12, P20, P22, and P23 acquired by the area coordinate acquisition unit 120 are shown, similar to the example shown in Fig. 7. Furthermore, referring to Fig. 15, unlike the example shown in Fig. 7, two-dimensional coordinates W11-5 after inverse transformation corresponding to the two-dimensional coordinate P11 in the camera image G10 are shown. The two-dimensional coordinates W11-5 are two-dimensional coordinates after point V2', which is obtained by further moving point V2 in the normal direction, is transformed.

[0101] Similarly, two-dimensional coordinates W12-5 after inverse transformation corresponding to the two-dimensional coordinate P12, two-dimensional coordinates W20-5 after inverse transformation corresponding to the two-dimensional coordinate P20, two-dimensional coordinates W22-5 after inverse transformation corresponding to the two-dimensional coordinate P22, and two-dimensional coordinates W23-5 after inverse transformation corresponding to the two-dimensional coordinate P23 are obtained. Although not shown, two-dimensional coordinates after inverse transformation corresponding to each of the two-dimensional coordinates P13 to P19, P21, and P24 (FIG. 2) are also obtained.

[0102] Fig. 16 is a diagram showing an example of a detection area according to the first modified example. In the example shown in Fig. 8, the two-dimensional coordinates W11-1 and W12-1 are located at the top, but in the example shown in Fig. 16, the two-dimensional coordinates W11-5 and W12-5 are located at the top. The detection area setting unit 144 sets a detection area B20 in which the two-dimensional coordinates W11-5 and W12-5 fit, instead of the two-dimensional coordinates W11-1 and W12-1.

[0103] The two-dimensional coordinates W11-5 and W12-5 are coordinates that take into consideration both the depth direction of the road plane and the normal direction of the road plane. As described above, when the vehicle moves from the back to the front, the depth direction of the road plane corresponds to the length direction of the vehicle, and the normal direction of the road plane corresponds to the height direction of the vehicle. That is, when the vehicle moves from the back to the front, the detection area B20 is set so that the lowest point of the front of the vehicle is point U, and a three-dimensional position moved in the height direction and the width direction of the vehicle (the highest point of the back of the vehicle) is also included.

[0104] Next, a second modified example will be described. In the above, a case has been mainly described in which the three-dimensional coordinate conversion unit 141 converts the point U(x, y, z) into two-dimensional coordinates in a camera image by moving the point U(x, y, z) in four directions to points V1 to V4. However, the three-dimensional coordinate conversion unit 141 may use the point of contact between a predetermined figure based on a circle centered on the point U(x, y, z) and a straight line passing through the origin O instead of the points V1 to V4 after the movement.

[0105] Fig. 17 is a diagram for explaining the extension of three-dimensional coordinates in the second modified example. As shown in Fig. 17, the three-dimensional coordinate conversion unit 141 may set a circle of radius r centered on point U(x, y, z) on the road plane, and use the tangent point between a cylinder having the bottom surface as the circle on the road plane and the top surface (the center of the top surface is point V1) moved by N in the normal direction and a straight line passing through the origin O as a substitute for points V1 to V4 after the movement. As a result, the detection area can be set so as to include points V1, V2', V3, and V4, as in the first modified example.

[0106] Next, a third modified example will be described. In the above, the display control unit 145 controls the display unit 230 to display the camera image G10, the user operates the camera image displayed on the display unit 230, and the area coordinate acquisition unit 120 acquires a plurality of two-dimensional coordinates in the camera image based on the user's operation. However, the method of acquiring the two-dimensional coordinates is not limited to this example.

[0107] For example, the area coordinate acquisition unit 120 may recognize a predetermined area through which a vehicle may pass based on a camera image, and may acquire a plurality of two-dimensional coordinates in the camera image based on the recognized area. For example, the area coordinate acquisition unit 120 may acquire two-dimensional coordinates constituting the outline of the recognized area as a plurality of two-dimensional coordinates. The recognition of the area may be realized in any manner.

[0108] For example, when the features of an area (e.g., a road surface, etc.) through which a vehicle may pass are learned in advance and a trained model is generated, the area coordinate acquisition unit 120 may input a camera image to the trained model and recognize the area output from the trained model as an area through which the vehicle may pass. Alternatively, when the features of an area through which a vehicle may pass are registered in advance, the area coordinate acquisition unit 120 may recognize an area in which features recognized from a camera image match or are similar to the features of an area through which a vehicle may move that have been registered in advance as an area through which the vehicle may pass.

[0109] Various modified examples have been described above.

[0110] <4. Hardware configuration example> Next, a hardware configuration example of the image processing device 10 according to the embodiment of the present invention will be described. Below, a hardware configuration example of the information processing device 900 will be described as a hardware configuration example of the image processing device 10 according to the embodiment of the present invention. Note that the hardware configuration example of the information processing device 900 described below is merely one example of the hardware configuration of the image processing device 10. Therefore, the hardware configuration of the image processing device 10 may be such that unnecessary configurations are deleted from the hardware configuration of the information processing device 900 described below, or new configurations are added. Note that the hardware of the image processing device 20 can be realized in a similar manner.

[0111] 18 is a diagram showing a hardware configuration of an information processing device 900 as an example of the image processing device 10 according to an embodiment of the present invention. The information processing device 900 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, a RAM (Random Access Memory) 903, a host bus 904, a bridge 905, an external bus 906, an interface 907, an input device 908, an output device 909, a storage device 910, and a communication device 911.

[0112] The CPU 901 functions as an arithmetic processing device and control device, and controls the overall operation of the information processing device 900 in accordance with various programs. The CPU 901 may also be a microprocessor. The ROM 902 stores programs and arithmetic parameters used by the CPU 901. The RAM 903 temporarily stores programs used in the execution of the CPU 901 and parameters that change appropriately during the execution. These are connected to each other by a host bus 904 consisting of a CPU bus etc.

[0113] The host bus 904 is connected to an external bus 906, such as a PCI (Peripheral Component Interconnect / Interface) bus, via a bridge 905. It is not necessary to configure the host bus 904, bridge 905, and external bus 906 separately, and these functions may be implemented in a single bus.

[0114] The input device 908 is composed of input means such as a mouse, keyboard, touch panel, button, microphone, switch, and lever for the user to input information, and an input control circuit that generates an input signal based on the user's input and outputs it to the CPU 901. A user who operates the information processing device 900 can input various data to the information processing device 900 and instruct the information processing device 900 to perform processing operations by operating this input device 908.

[0115] The output device 909 includes, for example, a display device such as a CRT (Cathode Ray Tube) display device, a Liquid Crystal Display (LCD) device, an OLED (Organic Light Emitting Diode) device, or a lamp, and an audio output device such as a speaker.

[0116] The storage device 910 is a device for storing data. The storage device 910 may include a storage medium, a recording device for recording data in the storage medium, a reading device for reading data from the storage medium, and a deleting device for deleting data recorded in the storage medium. The storage device 910 is configured, for example, with an HDD (Hard Disk Drive). This storage device 910 drives a hard disk and stores programs executed by the CPU 901 and various data.

[0117] The communication unit 911 is, for example, a communication interface configured with a communication device for connecting to a network, etc. Also, the communication unit 911 may be compatible with either wireless communication or wired communication.

[0118] An example of the hardware configuration of the image processing device 10 according to the embodiment of the present invention has been described above.

[0119] <5. Summary>

[0120] Although the preferred embodiment of the present invention has been described in detail above with reference to the accompanying drawings, the present invention is not limited to such an example. It is clear that a person having ordinary knowledge in the technical field to which the present invention pertains can conceive of various modified or altered examples within the scope of the technical ideas described in the claims, and it is understood that these also naturally belong to the technical scope of the present invention. [Explanation of symbols]

[0121] 10,20 Image processing device 110 Image acquisition unit 120 Area coordinate acquisition unit 121 Object position acquisition unit 130 Calibration information acquisition unit 140 Display processing unit 141 Three-dimensional coordinate conversion unit 142 Coordinate change section 143 Two-dimensional coordinate inverse transformation unit 144 Detection area setting section 145 Display control unit 210 Imaging unit 220 Input section 230 Display section

Claims

1. an image acquisition unit that acquires an image of the first area; a correspondence information acquisition unit that acquires correspondence information indicating a correspondence relationship between an imaging surface of the image and a plane including the first area; a coordinate information acquisition unit for acquiring a plurality of coordinates in the image; a first conversion unit that converts a plurality of coordinates in the image into coordinates corresponding to the plurality of coordinates in the plane based on the correspondence information to obtain a plurality of coordinates after a first conversion; a coordinate modification unit that calculates new coordinates by modifying the coordinates after the first transformation; a second conversion unit that converts the new coordinates into coordinates corresponding to the new coordinates on the imaging plane based on the correspondence information to obtain a plurality of coordinates after a second conversion; an area setting unit that generates a second area including the plurality of coordinates after the second transformation as a detection area for detecting an object; a display control unit that controls a display of the detection area on a display unit; Equipped with the coordinate information acquisition unit recognizes a predetermined area through which the object may pass based on the image, and acquires a plurality of coordinates in the image based on the predetermined area. Processing unit.

2. the coordinate change unit calculates, as the new coordinates, a plurality of coordinates after movement obtained by moving each of the plurality of coordinates after the first transformation in one or a plurality of movement directions; The movement direction includes a direction parallel to the plane. The processing device of claim 1 .

3. The movement direction includes a normal direction of the plane. The processing device according to claim 2 .

4. The direction parallel to the plane includes a direction perpendicular to a normal direction of the plane and a predetermined direction. The processing device according to claim 2 .

5. The predetermined direction includes a preset traveling direction of the object. The processing device according to claim 4 .

6. The coordinate information acquisition unit acquires a plurality of coordinates in the image based on an operation by a user on the image. The processing device of claim 1 .

7. the operation includes a designation operation in which the user designates coordinates in the image; the coordinate information acquisition unit acquires coordinates designated by the designation operation as a plurality of coordinates in the image; The processing device of claim 6.

8. the operation includes a placement operation in which the user places an object on the image; the coordinate information acquisition unit acquires a plurality of coordinates in the image based on a position of the object arranged by the arrangement operation; The processing device of claim 6.

9. the correspondence information acquisition unit acquires the correspondence information and camera information related to a camera that captures the image; the first conversion unit converts the plurality of coordinates on the plane into coordinates corresponding to the plurality of coordinates on the plane based on the correspondence information and the camera information; the second conversion unit converts the new coordinates into coordinates corresponding to the new coordinates on the imaging surface, based on the correspondence information and the camera information; The processing device according to any one of claims 1 to 8.

10. The camera information includes a focal length of the camera. The processing device of claim 9.

11. each of the plurality of coordinates after the first transformation is a three-dimensional coordinate in a real space; each of the new coordinates is a three-dimensional coordinate in real space; Each of the plurality of coordinates after the second transformation is a two-dimensional coordinate in a camera image. The processing device according to any one of claims 1 to 10.

12. A processing method for a processing device, comprising: acquiring an image of a first area; acquiring correspondence information indicating a correspondence relationship between an imaging surface of the image and a plane including the first area; obtaining a plurality of coordinates in the image; converting a plurality of coordinates in the image into coordinates corresponding to the plurality of coordinates in the plane based on the correspondence information to obtain a plurality of coordinates after a first conversion; Calculating new coordinates by modifying the first transformed coordinates; converting the new coordinates into coordinates corresponding to the new coordinates on the imaging plane based on the correspondence information to obtain second converted coordinates; generating a second area including the plurality of coordinates after the second transformation as a detection area for detecting an object; Controlling a display of the detection area on a display unit; having The step of acquiring a plurality of coordinates in the image includes: recognizing a predetermined area through which the object may pass based on the image; and acquiring a plurality of coordinates in the image based on the predetermined area. Processing methods.

13. Computer, an image acquisition unit that acquires an image of the first area; a correspondence information acquisition unit that acquires correspondence information indicating a correspondence relationship between an imaging surface of the image and a plane including the first area; a coordinate information acquisition unit for acquiring a plurality of coordinates in the image; a first conversion unit that converts a plurality of coordinates in the image into coordinates corresponding to the plurality of coordinates in the plane based on the correspondence information to obtain a plurality of coordinates after a first conversion; a coordinate modification unit that calculates new coordinates by modifying the coordinates after the first transformation; a second conversion unit that converts the new coordinates into coordinates corresponding to the new coordinates on the imaging plane based on the correspondence information to obtain a plurality of coordinates after a second conversion; an area setting unit that generates a second area including the plurality of coordinates after the second transformation as a detection area for detecting an object; and a display control unit that controls a display of the detection area on a display unit; Functioning as a the coordinate information acquisition unit recognizes a predetermined area through which the object may pass based on the image, and acquires a plurality of coordinates in the image based on the predetermined area. program.

Images