Moving body control device and moving body control system

Abstract

To provide a moving body control device and system capable of accurately reaching a target point with a simple configuration.SOLUTION: A moving body control device controls movement of a moving body that can travel on a road, and includes: a real image acquisition unit for acquiring a real image; a position determination unit which recognizes a moving body in the real image, and determines the position of the moving body on the real image; a virtual space model generation unit which generates a virtual space model having grid lines arranged on a virtual plane; a virtual image generation unit which generates a virtual image by performing perspective transformation to the model using a virtual camera so as to generate a virtual image in which the grid lines are viewed from an extension direction of the grid lines; a real-virtual information linking unit which determines a virtual position of the moving body on a prescribed virtual plane on the basis of a relative position relation between one point in the virtual image having the same coordinate with the real image position coordinate being the position coordinate of the moving body on the real image and the grid lines; and a control information generation unit which generates guiding control information for guiding the moving body to the target position on the basis of the virtual position and a position of the target point of movement of the moving body on a prescribed virtual plane.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The present invention relates to a mobile object control device and a mobile object control system. [Background technology]

[0002] Vehicles that perform automatic driving control are known. For example, Patent Document 1 discloses a vehicle system including a vehicle equipped with a camera, a radar device, a lidar, an object recognition device, an automatic driving control device, a GNSS (Global Navigation Satellite System) receiver, and the like.

[0003] In the vehicle system of Patent Document 1, an object recognition device recognizes the positions of surrounding objects based on the detection results of a camera, a radar device, and a lidar, and an automatic driving control device controls the equipment of the vehicle's drive system based on the recognition results, the vehicle's current position, and a target trajectory that lists the point the vehicle should reach. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-047120 Summary of the Invention [Problem to be solved by the invention]

[0005] For example, when applying a vehicle system such as that disclosed in Patent Document 1 to an autonomous robot capable of autonomously traveling on roads, there is a problem that errors are likely to occur in the estimation of the robot's own position in places where GNSS radio waves are difficult to reach, making it impossible to accurately reach the target point. In addition, the above-mentioned radar devices, lidars, and high-precision GNSS receivers are expensive, and there is also a problem that costs increase when multiple autonomous robots are deployed.

[0006] The present invention has been made in consideration of the above-mentioned problems, and aims to provide a mobile body control device and a mobile body control system that are inexpensively configured and capable of causing a mobile body to reach a target point with high accuracy. [Means for solving the problem]

[0007] The mobile body control device according to the present invention is a mobile body control device that controls the movement of a mobile body that can travel on a road, and is characterized by having: a real image acquisition unit that acquires a real image, which is an image captured by a real camera that captures real space; a mobile body recognition / image position identification unit that recognizes a moving body in the real image and identifies the position of the moving body on the real image; a virtual space model generation unit that generates a virtual space model having grid lines arranged on a specified virtual plane; a virtual image generation unit that performs perspective transformation on the virtual space model using a virtual camera to generate a virtual image in which the grid lines are viewed from the extension direction of the grid lines; a real-virtual information link unit that identifies a virtual position of the moving body on a specified virtual plane based on the relative positional relationship between the grid lines and a point in the virtual image that has the same coordinates as the real image position coordinates, which are the position coordinates of the moving body in the real image; and a control information generation unit that generates guidance control information for guiding the moving body to a target point based on the virtual position and the position of a target point for the movement of the moving body on the specified virtual plane. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating a configuration of a moving object control system according to a first embodiment. [Diagram 2] 1 is a block diagram illustrating an example of a hardware configuration of a mobile object control device according to a first embodiment. [Diagram 3] 1 is a block diagram showing an example of a hardware configuration of a moving body according to a first embodiment. [Figure 4] 4 is a block diagram showing an example of a software configuration of an information processing unit in the mobile object control device according to the first embodiment. FIG. [Diagram 5] 4A to 4C are diagrams illustrating an example of a captured image captured by a camera connected to a mobile body control device and an information superimposed image generated by the mobile body control device according to the first embodiment. [Figure 6] 3 is a diagram showing an example of a virtual space model generated by a moving body control device according to the first embodiment and an example of an imaging mode of the model; FIG. [Figure 7] FIG. 4 is a diagram illustrating an example of a virtual image generated by the moving body control device according to the first embodiment. [Figure 8] 4 is a diagram showing an image when a virtual position of a moving body on a virtual plane is specified by the moving body control device according to the first embodiment. FIG. [Figure 9] 4 is a diagram illustrating an example of a transformation curve that is generated by the moving body control device according to the first embodiment and transforms a Y coordinate on the virtual image into a coordinate in the depth direction of a virtual plane. FIG. [Figure 10] 4 is a diagram illustrating an example of guidance control performed by the mobile object control device according to the first embodiment when guiding the mobile object to a target point. FIG. [Figure 11] 4 is a flowchart showing a control routine of the moving body control device according to the first embodiment. [Figure 12] 5 is a flowchart showing a control subroutine of the mobile object control device according to the first embodiment. [Figure 13] 5 is a flowchart showing a control subroutine of the mobile object control device according to the first embodiment. [Figure 14] FIG. 11 is a diagram illustrating a configuration of a moving object control system according to a second embodiment. [Figure 15] FIG. 11 is a block diagram illustrating an example of a hardware configuration of a mobile object control device according to a second embodiment. [Figure 16] FIG. 11 is a diagram illustrating an example of a virtual grid lane generated by a mobile object control device according to a second embodiment. [Figure 17] FIG. 11 is a diagram illustrating an example of guidance control performed by the mobile object control device according to the second embodiment when guiding the mobile object to a target point. [Figure 18] 10 is a flowchart showing a control routine of a moving object control device according to a second embodiment. [Figure 19] 10 is a flowchart showing a control subroutine of a mobile object control device according to a second embodiment. [Figure 20]10 is a flowchart showing a control subroutine of a mobile object control device according to a second embodiment. [Figure 21] FIG. 11 is a block diagram showing an example of a software configuration of an information processing unit in the mobile object control device according to the second embodiment. [Figure 22] FIG. 13 is a diagram showing an example of a road shape generation mode performed by a mobile object control device according to a modified example of the second embodiment. [Diagram 23] FIG. 11 is a diagram illustrating a configuration of a moving object control system according to a third embodiment. [Figure 24] FIG. 11 is a block diagram showing an example of a hardware configuration of a moving body according to a third embodiment. [Diagram 25] FIG. 11 is a block diagram showing an example of a software configuration of an information processing unit in a moving object according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In the drawings, the same components are designated by the same reference numerals, and the description of the same components will be omitted. EXAMPLES

[0010] [System configuration] First, the overall system configuration will be described with reference to Fig. 1. Fig. 1 is a diagram showing the configuration of a mobile object control system 100 according to the first embodiment. The mobile object control system 100 includes a mobile object control device 10 and a mobile object 30 as shown in Fig. 1.

[0011] The mobile control system 100 can be used, for example, on a two-lane highway, when lane restrictions are imposed on a portion of one of the two lanes for cleaning, construction, or recovery work after an accident, to inform passengers of vehicles traveling behind the restricted section that a lane restriction is in place.

[0012] First, the mobile object control device 10 will be described. The mobile object control device 10 is mounted inside a vehicle V that travels on a road. The vehicle V is a sign vehicle with a sign device SD on the loading platform, the sign device SD having a display on which text such as "Work ahead" or "Watch out for collision" is displayed. In the following description, the left direction in the drawing is the front of the vehicle V. Also, the up-down direction in the drawing is the height direction of the vehicle V.

[0013] In the vehicle V, a camera 12 is provided on the upper part of the sign device SD as a real camera that captures an image of a landscape including a road surface behind the vehicle V. The camera 12 is connected to the mobile body control device 10 by wire or wirelessly, and is configured to be able to transmit the captured image to the mobile body control device 10.

[0014] The camera 12 is a camera that can be swiveled up, down, left and right in response to a control signal from the mobile object control device 10. The camera 12 is, for example, a PTZ camera that can perform pan, tilt and zoom.

[0015] Next, a description will be given of the moving body 30. The moving body 30 is a self-propelled small robot equipped with a traveling device 31 including two wheels that can travel on a road.

[0016] A light emitter LB having a dome shape and capable of continuously or intermittently emitting red or orange light is provided on the upper portion of the moving body 30. The light emitter LB is a warning light (patrol lamp) having a width and depth of about 200 mm and a height of about 100 mm.

[0017] For example, a plurality of moving bodies 30 may be placed at regular intervals (e.g., every several tens of meters) on the road behind the vehicle V. Each of the plurality of moving bodies 30 placed on the road can notify passengers of a vehicle heading toward the restricted section, for example, the driver of the vehicle, that lane regulation is in place ahead, by emitting light from the light emitting body LB. In other words, the moving body 30 is an autonomous robot that warns the driver not to enter the lane regulation section.

[0018] In the mobile body control system 100, the mobile body control device 10 and the mobile body 30 are connected so as to be able to transmit and receive data to and from each other. The connection between the mobile body control device 10 and the mobile body 30 can be established by wireless communication such as Wi-Fi (registered trademark), Wi-SUN (registered trademark), or BLE (Bluetooth Low Energy) (registered trademark).

[0019] In the mobile body control system 100, the mobile body control device 10 recognizes the mobile body 30 in an image captured by the camera 12, and identifies the position of the mobile body 30 on a virtual plane using the position of the mobile body 30 on the captured image and a coordinate system on a virtual plane in virtual space.

[0020] Furthermore, in the mobile body control system 100, the mobile body control device 10 executes guidance control to guide the mobile body 30 to the target point based on the identified position of the mobile body 30 on the virtual plane and the position of the target point of the movement of the mobile body 30.

[0021] [Configuration of mobile control device] The configuration of the mobile object control device 10 in this embodiment will be described below with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the hardware configuration of the mobile object control device 10.

[0022] The mobile object control device 10 is connected to a touch panel display 13 provided in the vehicle V in addition to the camera 12 described above.

[0023] The touch panel display 13 is a display device that combines a display that displays a screen based on the control of the mobile body control device 10 and a touch panel that accepts input operations from an operator of the mobile body control device 10, for example, a passenger of the vehicle V. For example, an operation for establishing wireless communication with the mobile body 30 or a destination point for the movement of the mobile body 30 can be input into the touch panel display 13.

[0024] The mobile object control device 10 is a device in which an information processing unit 17, a communication unit 18, an input unit 19, an output unit 21, and a large-capacity storage device 22 cooperate with each other via a system bus 16. In the mobile object control system 100, the mobile object control device 10 functions as a computer.

[0025] The information processing unit 17 is a processing device including a CPU (Central Processing Unit) 17A, a ROM (Read Only Memory) 17B, and a RAM (Random Access Memory) 17C. In the information processing unit 17, the CPU 17A reads and executes various programs stored in the ROM 17B and a large-capacity storage device 22 described later, thereby realizing various functions.

[0026] The communication unit 18 is a communication device that transmits and receives data to and from external devices in accordance with instructions from the information processing unit 17. The communication unit 18 is, for example, a network interface card (NIC) for connecting to a network. The communication unit 18 transmits and receives various data by connecting to a communication unit of a mobile object 30, which will be described later.

[0027] The input unit 19 is an interface unit that communicatively connects the information processing unit 17 of the mobile object control device 10 with the camera 12 and the touch panel display 13. The information processing unit 17 acquires, for example, an image captured by the camera 12 via the input unit 19.

[0028] The output unit 21 is an interface unit communicatively connected to the touch panel display 13. The information processing unit 17 transmits, for example, a video or image signal to the touch panel display 13 via the output unit 21 for display.

[0029] The large-capacity storage device 22 is a storage device that stores and manages data necessary for processing of the information processing unit 17. The large-capacity storage device 22 is configured, for example, with a hard disk device, an SSD (solid state drive), a flash memory, etc., and stores various programs such as an operating system and software for terminals.

[0030] The above-mentioned various programs may be acquired, for example, from another server device or the like via a network, or may be recorded on a recording medium and read via various drive devices. In other words, the various programs stored in the large-capacity storage device 22 can be transmitted via a network, or can be recorded on a computer-readable recording medium and transferred.

[0031] The recognition model 22A stored in the mass storage device 22 is an AI model that enables the recognition of the moving object 30 in the captured image captured by the camera 12. The recognition model 22A is an AI model that has previously learned about the features of the moving object 30, such as the shape of the moving object 30.

[0032] The recognition model 22A recognizes the moving object 30 in the captured image by inputting the above-mentioned captured image by the information processing unit 17, and outputs as the recognition result, for example, bounding box information indicating the position of the moving object in the captured image and the size of the moving object on the captured image.

[0033] [Mobile configuration] Next, the hardware configuration of the moving body 30 in this embodiment will be described with reference to Fig. 3. The moving body 30 is a device in which an information processing unit 34, a communication unit 35, and an output unit 37 cooperate with each other via a system bus 33. The information processing unit 34 is a processing device including a CPU 34A, a ROM 34B, and a RAM 34C, similar to the information processing unit 17 of the moving body control device 10. The information processing unit 34 is capable of generating a drive control signal for driving the traveling device 31.

[0034] The communication unit 35 is a communication device that transmits and receives data to and from external devices in accordance with instructions from the information processing unit 34. The communication unit 35 is, for example, a NIC for connecting to a network. The communication unit 35 transmits and receives various data by being connected to the communication unit 18 of the mobile body control device 10 described above.

[0035] The output unit 37 is an interface unit communicatively connected to the traveling device 31. The output unit 37 outputs a drive control signal generated in the information processing unit 34 based on a control signal received from, for example, the mobile body control device 10 to the traveling device 31. In response to the drive control signal output from this output unit 37, the rotation speed of the wheels of the traveling device 31 is controlled.

[0036] [Determining the position of a moving object on a virtual plane] Here, the specification of the position of the moving body 30 on the virtual plane by the moving body control device 10 will be described. In the following, it is assumed that the moving body 30 is always imaged by the camera 12 provided on the vehicle V. In other words, it is assumed that the moving body 30 is always within the angle of view of the camera 12.

[0037] Fig. 4 is a block diagram showing an example of the software configuration of the information processing unit 17 in the mobile object control device 10. In Fig. 4, the flow of information output from each block to another block is indicated by solid arrows. Note that the description in parentheses next to the arrow indicates the output information output from the source of the arrow. The software configuration will be described below mainly using Fig. 4 and referring to Figs. 5 to 9 as appropriate.

[0038] The real image acquisition unit 41 sequentially acquires and outputs real images, which are images of the rear of the vehicle V captured by the camera 12. As described above, the moving object 30 is reflected in the real image captured by the camera 12.

[0039] The moving object recognition / image position identification unit 42 inputs the real image acquired by the real image acquisition unit 41 into the recognition model 22A (see Figure 2) stored in the large-capacity storage device 22, and receives as output bounding box information indicating the position of the moving object 30 on the real image and the size of the moving object 30 on the real image as a recognition result of the moving object 30.

[0040] The moving object recognition / image position identification unit 42 identifies the position coordinates of the moving object 30 on the real image from the bounding box information acquired from the recognition model 22A, and uses the bounding box information to generate an information superimposed image in which a bounding box surrounding the moving object 30 is superimposed on the real image.

[0041] 5 is a diagram showing an example of the information superimposed image SI created by the moving object recognition / image position identification unit 42. As described above, when the moving object recognition / image position identification unit 42 acquires the bounding box information from the recognition model 22A, it identifies the position coordinates on the real image RI of the moving object 30 recognized in the real image RI. For example, the moving object recognition / image position identification unit 42 defines an axis along the left-right direction of the real image RI as the X-axis and an axis along the up-down direction as the Y-axis, and identifies the number of pixels in the X-axis and Y-axis directions from the origin (for example, the upper left corner of the real image RI) as the X-coordinate and the Y-coordinate, respectively.

[0042] The moving object recognition / image position specification unit 42 specifies, for example, the coordinates of the center point CP of the moving object 30 as the position coordinates of the moving object 30 on the real image RI. For example, the moving object recognition / image position specification unit 42 specifies the coordinates of the center point CP of the moving object 30 as the position coordinates of the moving object 30 on the real image RI. A (X A , Y A ) is identified as the position coordinates of the moving object 30 on the real image RI.

[0043] In addition, as described above, the moving object recognition / image position identification unit 42 uses the bounding box information obtained from the recognition model 22A to generate an information superimposed image SI in which a bounding box B surrounding the moving object 30 is superimposed on the real image RI, as shown in Figure 5.

[0044] Fig. 6 is a diagram showing an example of a three-dimensional virtual space model VM consisting of an X-axis, a Y-axis, and a Z-axis and a photographing mode of the model. In Fig. 6, a plurality of virtual panels VP each having a rectangular top surface shape are arranged on an XY virtual plane, which is a plane formed by the X-axis and the Y-axis. Each of the virtual panels VP has a length of 1 m along the X-axis and a length of 5 m along the Y-axis, for example. In Fig. 5, each of the virtual panels VP is hatched to make it easier to distinguish the virtual panels VP.

[0045] The virtual space model generation unit 43 arranges each of the virtual panels VP in a checkerboard arrangement pattern on the XY virtual plane, for example, as shown in Fig. 6. By arranging the virtual panels VP in this manner, the virtual space model generation unit 43 can generate a virtual space model in which lattice lines consisting of the outlines (long sides and short sides) of the virtual panels VP and their extensions (dashed lines in the figure) are provided on the XY virtual plane.

[0046] 6, a virtual camera VC that captures an image of the XY virtual plane is disposed on an XZ virtual plane, which is a plane formed by the X-axis and the Z-axis. The virtual camera VC has the same settings as the camera 12 that captures an image of the real space. Specifically, the virtual camera VC has the same camera installation height from the XY virtual plane, the angle of view, and the gaze point as the camera 12.

[0047] The virtual image generating unit 44 performs perspective transformation to convert the three-dimensional virtual space model VM into a two-dimensional image viewed from a certain viewpoint by using the virtual camera VC to capture an image of the virtual space model VM using the virtual camera VC. Specifically, the virtual image generating unit 44 generates a virtual image in which grid lines provided on the XY virtual plane are viewed from the extension direction of the grid lines by perspective transformation using the virtual camera VC having the same settings as the camera 12.

[0048] 7 is a diagram showing a virtual image VI obtained by performing perspective transformation on a virtual space model VM via a virtual camera VC. In the virtual image VI, a grid model GM consisting of grid lines generated based on a plurality of virtual panels VP arranged on an XY virtual plane is displayed. As described above, the grid model GM can be formed by the plurality of virtual panels VP and the extension lines of the outlines of each of the virtual panels VP.

[0049] As described above, the virtual camera VC is set to the same height, angle of view, and gaze point as the camera 12 from the XY virtual plane. Also, the virtual image VI, which is an image obtained by perspective transformation using the virtual camera VC, has the same image size as the real image RI. Therefore, in the virtual image VI, the position coordinates of a point on the virtual image VI can be expressed by the number of pixels on the real image RI. That is, in the virtual image VI, the position coordinates on the virtual image VI can be expressed by the position coordinates on the real image RI.

[0050] In addition, in the mobile body control system 100, a superimposed image may be generated by superimposing the real image RI acquired by the real image acquisition unit 41 and the virtual image VI generated by the virtual image generation unit 44, thereby linking the image coordinate system on the real image RI with the coordinates on the virtual image VI.

[0051] Fig. 8 is a diagram showing an image when the virtual position of the moving body 30 on the XY virtual plane is specified by the moving body control device 10. In Fig. 8, the center of the bottom end of the grid model GM is set as the origin O, and the X-axis and Y-axis passing through the origin O are shown.

[0052] The reality / virtual information link unit 45 calculates the position coordinates of the moving object 30 on the reality image RI, ie, a point P A (X A , Y A ) and the grid lines of the grid model GM, a point P V (X V , Y VThat is, the reality / virtual information link unit 45 has a moving object position specifying function for specifying the virtual position of the moving object 30 on the XY virtual plane.

[0053] First, a virtual width direction coordinate X, which is a coordinate in the width direction (X-axis direction) from the origin O of the virtual position of the moving body 30, is V The calculation method of the Y coordinate (Y A ) and two points P on the same grid line A1 and point P A2 Identify point P A1 and point P A2 Point P for the difference in the number of pixels in the X-axis direction A and point P A1 The virtual width coordinate X is calculated based on the ratio of the difference between the number of pixels in the X-axis direction and V Calculate.

[0054] Here, the Y coordinate (Y A ) and two points P on the same grid line A1 and point P A2 The coordinates of P A1 (X A1 , Y A ) and P A2 (X A2 , Y A ), then the virtual width coordinate X V is calculated together with the length of the grid lines using the following formula 1.

[0055]

number

[0056] Fig. 9 is a diagram showing an example of the above-mentioned conversion curve CC. In Fig. 9, the horizontal axis represents the distance in the depth direction from the origin O on the XY virtual plane, and the vertical axis represents the Y coordinate on the real image RI.

[0057] The conversion curve CC is generated, for example, by plotting the Y coordinates of the position coordinates on the real image corresponding to each length (for example, 5 m, 10 m, 15 m, 20 m) in the depth direction from the origin O in the grid model GM.

[0058] The real / virtual information link unit 45 converts the Y coordinate of the moving object 30 on the real image RI into a virtual depth direction coordinate Y by using this conversion curve CC. V can be sought.

[0059] Therefore, the real-virtual information link unit 45 obtains the virtual width direction coordinate and the virtual depth direction coordinate by the above-mentioned method, and calculates the virtual position point P V (X V , Y V ) can be calculated (hereafter referred to as the virtual position P V That is, the reality / virtual information link unit 45 can specify where on the XY virtual plane the moving object 30 recognized in the real image RI is located.

[0060] [Guidance control of moving objects to their destination] Next, with reference to Fig. 4, the guidance control of the moving body 30 to the target point by the moving body control device 10 will be described using Fig. 10. Fig. 10 is a view of the XY virtual plane in the virtual space model shown in Fig. 6 as viewed from the Z-axis direction. In Fig. 10, the grid lines when the virtual panel VP is arranged are shown by thick solid lines.

[0061] In Fig. 10, the vector origin P0 as the initial position is the starting position of the moving body 30 on the XY virtual plane. In the following, the moving body 30 moves from the vector origin P0 to a virtual position P1 (15, 1), and the moving target of the moving body 30 is a target point P1 15 m in the depth direction from the origin O. t The case where it is set to (0, 15) will be explained.

[0062] The control information generating unit 46 determines the virtual position P of the moving object 30 specified by the reality / virtual information link unit 45. V and the target point P of the moving object 30 on the XY virtual plane t Based on the position of t The guidance control information for guiding the moving body 30 to the target position is generated, and the guidance control information is transmitted to the moving body 30.

[0063] The control information generator 46 determines whether the magnitude of the vector V1 as the first vector extending from the vector origin P0 toward the virtual position P1 is equal to or greater than a predetermined value (for example, 2 m or greater) and whether the magnitude of the vector V1 extending from the virtual position P1 to the target point P t When the magnitude of vector V2 as a second vector toward the target is equal to or greater than a predetermined value (eg, 1 m or greater), the angle θ formed by an extension line of vector V1 (a dashed line in the drawing) and vector V2 is calculated.

[0064] For example, when the angle θ exceeds a predetermined value, that is, when the moving direction of the moving object 30 at the virtual position P1 and the moving direction from the virtual position P1 to the target point P t When the direction in which the moving object 30 is viewed differs greatly, the moving object 30 is controlled to turn so that the angle θ becomes equal to or smaller than a predetermined value. Specifically, the control information generator 46 generates turning control information for controlling the moving object 30 to turn until the angle θ becomes equal to or smaller than the predetermined value, and transmits the information to the moving object 30.

[0065] In addition, for example, when the angle θ is equal to or smaller than a predetermined value, that is, when the moving direction of the moving object 30 at the virtual position P1 and the moving direction from the virtual position P1 to the target point P tare substantially the same, the control information generator 46 executes straight-line control to cause the moving body 30 to continue moving straight.

[0066] The mobile object control device 10 controls the mobile object 30 to turn and go straight based on the turning control information and the straight-line control information transmitted from the mobile object control device 10 as described above, thereby moving the mobile object 30 to the target point P t can be achieved.

[0067] Specific operations of the mobile object control device 10 in this embodiment will be described below using Figures 5 to 10 while referring to Figures 11 to 13. Figure 11 is a flowchart showing a guidance control routine RT1 executed in the information processing unit 17 of the mobile object control device 10 for guiding the mobile object 30 to a target point.

[0068] For example, when wireless communication between the mobile object control device 10 and the mobile object 30 is established, the information processing unit 17 receives a target point P t The guidance control routine RT1 starts when the target point P t This is executed repeatedly every time an update is made to the

[0069] First, when communication is established between the mobile object control device 10 and the mobile object 30, the reality / virtual information link unit 45 of the information processing unit 17 calculates the position coordinates of the mobile object 30 on the real image RI, as shown in FIG. 8 and FIG. A Based on the relative positional relationship between the grid lines of the grid model GM and the XY virtual plane, a virtual position P v (Step S101).

[0070] Next, the control information generating unit 46 of the information processing unit 17 calculates the virtual position P vis set as the vector origin P0 (step S102). That is, the control information generator 46 sets the point P0, which is the starting position of the moving object 30, as a reference point for the movement of the moving object 30.

[0071] After step S102, the control information generator 46 executes start control of the moving body 30 (step S103). Specifically, the control information generator 46 generates straight-line control information for performing straight-line control, for example, and transmits the straight-line control information to the moving body 30.

[0072] After step S103, as shown in FIG. 10, the control information generator 46 identifies a virtual position P1 that is the current virtual position of the moving object 30 after it has moved from the vector origin P0 in the same manner as in step S101 (step S104).

[0073] After step S104, the control information generator 46 acquires a vector V1 that is a vector connecting the vector origin P0 and a virtual position P1 that is the current virtual position, and determines whether or not the magnitude of the vector V1 is equal to or greater than a predetermined value (step S105), as shown in Fig. 10. That is, the control information generator 46 determines whether or not the distance from the vector origin P0 to the current virtual position P1 is equal to or greater than a predetermined value.

[0074] If the magnitude of the vector V1 is not equal to or greater than a predetermined value (step S105: NO), the control information generator 46 executes step S104 again. That is, the control information generator 46 repeatedly executes step S104 until the distance from the vector origin P0 to the current virtual position P1 becomes equal to or greater than a predetermined value.

[0075] If the magnitude of the vector V1 is equal to or greater than a predetermined value (step S105: YES), the control information generator 46 calculates the distance between the virtual position P1, which is the current virtual position, and the target point P t The control information generator 46 obtains a vector V2 that is a vector that connects the current virtual position P1 to the target point P and determines whether the magnitude of the vector V2 is less than a predetermined value (step S106). t It is determined whether the distance to is less than a predetermined value.

[0076] If the magnitude of the vector V2 is not less than a predetermined value (step S106: NO), the control information generator 46 executes straight / turn control to control the moving body 30 to go straight or turn (step S107). After step S107, the control information generator 46 repeats step S104.

[0077] If the magnitude of the vector V2 is less than a predetermined value (step S106: YES), the control information generator 46 executes stop control to transmit a stop signal to the moving body 30 to stop the moving body 30 on the spot (step S108). t Or target point P t When it is determined that the moving object 30 has reached the vicinity of the target object 30, the control information generator 46 executes stop control of the moving object 30. After step S108, the control information generator 46 ends the guidance control routine RT1.

[0078] Next, a description will be given of the virtual position specifying subroutine RT2 of the moving body 30 executed in steps S101 and S104.

[0079] First, the real image acquisition unit 41 of the information processing unit 17 sequentially acquires and outputs a real image RI, which is an image of the rear of the vehicle V captured by the camera 12, which is a real camera (step S201). Note that in the following, it is assumed that the moving body 30 is always captured by the camera 12, as in the above description.

[0080] After step S201, the moving object position / image position identification unit 42 of the information processing unit 17 inputs the real image RI acquired by the real image acquisition unit 41 to the recognition model 22A stored in the large-capacity storage device 22, and acquires bounding box information indicating the position of the moving object on the real image RI and the size of the moving object on the real image RI from the recognition model 22A as output (step S202).

[0081] After step S202, the moving object position / image position identification unit 42 identifies the position coordinates of the moving object 30 on the real image RI from the bounding box information, and generates an information superimposed image SI in which the bounding box is superimposed on the real image RI, as shown in Figure 5 (step S203).

[0082] In step S203, the moving object position / image position identification unit 42 identifies the center coordinates of the moving object 30 as position coordinates on the real image RI, for example, when the number of pixels in the X-axis direction and the Y-axis direction from the origin in the real image RI are defined as the X coordinate and the Y coordinate, respectively, as described above.

[0083] Next, the virtual space model generating unit 43 of the information processing unit 17 generates a virtual space model VM in which grid lines are arranged on a virtual plane in the three-dimensional virtual space (step S204). Specifically, as shown in Fig. 6, the virtual space model generating unit 43 arranges a plurality of virtual panels VP in a checkerboard pattern on the XY virtual plane, and generates the virtual space model VM based on the outlines of each of the arranged virtual panels VP.

[0084] Next, the virtual image generation unit 44 of the information processing unit 17 performs perspective transformation on the virtual space model VM generated by the virtual space model generation unit 43 using a virtual camera VC (see Figure 6) having settings similar to those of the camera 12, thereby generating a virtual image VI in which the XY virtual plane is viewed from the extension direction of the grid lines (step S205).

[0085] Specifically, the virtual image generation unit 44 generates a virtual image VI in which a grid model GM consisting of grid lines having a predetermined length is arranged and an image coordinate system on the real image RI is linked, as shown in FIG.

[0086] Finally, the reality / virtual information link unit 45 of the information processing unit 17 identifies the virtual position of the moving body 30 on the virtual plane based on the virtual image VI generated by the virtual image generation unit 44 and the position coordinates of the moving body 30 on the reality image RI (step S206).

[0087] Specifically, the reality / virtual information link unit 45 calculates the position coordinates of a point P A The X-coordinate of (X A ) and the grid lines of the grid model GM, the virtual width direction coordinate (X V ) is calculated.

[0088] As shown in FIG. 9, the reality / virtual information link unit 45 calculates the position coordinates of a point P A Y coordinate of (Y A ) and the transformation curve CC, the virtual depth direction coordinate (Y V ) is calculated. As a result, the reality / virtual information link unit 45 calculates the virtual position of the moving object 30, that is, the point P V (X V , Y V ) to identify the

[0089] The information processing unit 17 ends the virtual position specifying subroutine RT2 after step S206. Note that, in the above-mentioned virtual position specifying subroutine RT2, step S204 (generation of a virtual space model) may be performed before step S201. That is, the virtual space model generating unit 43 may generate in advance a virtual space model VM having an XY virtual plane on which grid lines are arranged.

[0090] Next, the straight-line / turn control subroutine executed in step S107 will be described below.

[0091] First, the control information generating unit 46 of the information processing unit 17 calculates a vector V1 connecting the vector origin P0 and the current virtual position P1, and a vector V2 connecting the current virtual position P1 and the target point P t An angle θ between the vector V1 and the vector V2 connecting the vectors is calculated (step S301).

[0092] After step S301, the control information generator 46 determines whether the angle θ is equal to or smaller than a predetermined value (step S302). If the control information generator 46 determines that the angle θ is not equal to or smaller than the predetermined value (step S302: NO), the control information generator 46 executes turning control of the moving object 30 (step S303).

[0093] In step S303, the control information generator 46 generates, for example, turning control information for turning the moving body 30 by a predetermined angle, and transmits it to the moving body 30. After step S303, the control information generator 46 ends the straight ahead / turning control subroutine RT3.

[0094] When the control information generator 46 determines that the angle θ is equal to or smaller than a predetermined value (step S302: YES), it sets the current virtual position P1 of the moving object 30 as the vector origin P0 (step S304). That is, the next time the control information generator 46 acquires a virtual position, it obtains vectors V1 and V2 with the virtual position P1 as the vector origin P0.

[0095] After step S304, the control information generator 46 executes straight-line control of the moving body 30 (step S305). For example, the control information generator 46 generates straight-line control information for controlling the moving body 30 to continue moving straight, and transmits the straight-line control information to the moving body 30. After step S305, the control information generator 46 ends the straight-line / turn control subroutine RT3.

[0096] According to the mobile object control system 100 of this embodiment, the above-mentioned guidance control routine RT1 is used to identify the virtual position of the mobile object 30 on the virtual plane, and the mobile object 30 is guided to the target point P t By guiding the moving object 30 to the target point P t can be achieved.

[0097] Therefore, according to the mobile object control system 100 of this embodiment, even when the mobile object 30 is traveling in an environment where it is difficult for GPS radio waves to reach, the mobile object 30 can be placed at a desired position with high accuracy.

[0098] In the mobile object control system 100 of this embodiment, the mobile object control device 10 specifies the virtual position of the mobile object 30 and sets the target point P t In order to guide the moving body 30 to the target position, the moving body 30 itself can be configured simply.

[0099] Therefore, according to the mobile object control system 100 of this embodiment, for example, it is not necessary to provide an expensive GPS receiver or the like in the mobile object 30, and the cost of the entire system can be reduced. t It is possible to reach this with high accuracy.

[0100] In the moving object control system 100 of this embodiment, the real image acquisition unit 41 calculates the virtual depth direction coordinate Y V Based on the above, the camera 12 is zoomed at a predetermined magnification to acquire a real image RI zoomed at the predetermined magnification. For example, the real image acquisition unit 41 zooms the camera 12 when the moving object 30 moves away from the starting point by a predetermined distance or more. This makes it possible to prevent the moving object 30 from appearing unclear in the real image RI.

[0101] Furthermore, in this embodiment, when the real image acquisition unit 41 performs optical zoom on the camera 12 as described above, the virtual image generation unit 44 zooms the virtual camera VC in accordance with the zoom magnification of the camera 12. This makes it possible to obtain the real image RI and the virtual image VI, each of which is zoomed in at a predetermined zoom magnification.

[0102] In this embodiment, the moving body 30 is provided with an illuminant LB, and the illuminant LB is illuminated to alert vehicles heading toward the lane-restricted section, but attention may be drawn in other ways in addition to the illumination of the illuminant LB.

[0103] For example, the moving object 30 may be provided with a speaker that outputs sound under the control of the information processing unit 17. The information processing unit 17 may output, for example, via the speaker, a sound that indicates that lane regulation is being implemented on the road ahead to a vehicle heading toward the lane regulation section together with the light emission of the light emitter LB. EXAMPLES

[0104] Next, a mobile object control system 200 according to a second embodiment will be described with reference to Fig. 14. Fig. 14 is a diagram showing the configuration of the mobile object control system 200, and is a diagram showing a vehicle V and a mobile object 30 located on a road as viewed from above. Fig. 14 shows a state in which the vehicle V has passed a curved road CR and is stopped at the bottom of the figure. In the following description, the downward direction in the figure is the front of the vehicle V.

[0105] [Configuration of mobile control device] First, the configuration of the mobile object control device 10 in this embodiment will be described with reference to Fig. 15. Fig. 15 is a block diagram showing an example of the hardware configuration of the mobile object control device 10. Below, differences from the first embodiment will be mainly described.

[0106] The GPS receiver 14 is connected to the mobile object control device 10 via an input unit 19, and is a receiver that receives signals (GPS signals) from GPS (Global Positioning System) satellites. The GPS receiver 14 is disposed, for example, on a dashboard inside the vehicle V.

[0107] A map information database (hereinafter referred to as map information DB) 22B stored in the large-capacity storage device 22 is a database in which map information is stored. The reality / virtual information link unit 45 of the information processing unit 17 identifies the current position of the vehicle V based on the GPS signal received by the GPS receiver 14 and the map information DB 22B.

[0108] [Guidance control of moving objects using virtual grid lanes] Here, the guidance control of the moving object 30 to the target point using the virtual grid lane performed by the information processing unit 17 in this embodiment will be described with reference to Fig. 16 and Fig. 17. The software configuration of the information processing unit 17 is the same as that of the first embodiment.

[0109] In this embodiment, the reality virtual information link unit 45 identifies the current position of the vehicle V based on the GPS signal received by the GPS receiver 14 and the map information DB 22B, and acquires, from the map information DB 22B, road shape information indicating the shape of the road to a point a predetermined distance behind the current position of the vehicle V. For example, the reality virtual information link unit 45 acquires road shape information including the curved road CR along which the vehicle V has traveled, as shown in FIG.

[0110] The virtual space model generating unit 43 generates a three-dimensional virtual guide lane model (not shown) on an XY virtual plane, which is in accordance with the shape of the curved road CR included in the road shape information acquired by the reality virtual information link unit 45.

[0111] The virtual image generation unit 44 uses a virtual camera VC having the same settings as the camera 12 to perform perspective transformation on a virtual guide lane model set on an XY virtual plane, thereby generating a virtual guide lane VL viewed from the extension direction of the virtual guide lane model.

[0112] Fig. 16 is a diagram showing an example of a virtual grid lane. As shown in Fig. 16, the real-world virtual information link unit 45 generates a virtual grid lane VGL, which is a virtual guide lane VL having information of the grid model GM, by superimposing the virtual guide lane VL on the grid model GM of the virtual image VI. That is, the real-world virtual information link unit 45 has a virtual grid lane generation function that generates a virtual grid lane VGL based on the road shape shown in the road shape information.

[0113] FIG. 17 shows the location of the target point P of the moving object 30 using the virtual grid lane VGL. t17 is a diagram showing an example of guidance control to a target point P1', which is a current virtual position of the moving object 30. FIG. 17 shows a state in which the moving object 30 deviates from the virtual guide lane VL at a point P1', which is a moving target point of the moving object 30. t is set on the virtual grid lane VGL.

[0114] As described above, the control information generating unit 46 controls the moving object 30 to move to the target point P based on the virtual position of the moving object 30 specified by the reality / virtual information linking unit 45. t In this embodiment, the control information generating unit 46 generates guidance control information for guiding the moving object 30 to the target point P. t When guiding the moving object 30 to the virtual grid lane VGL, the control mode of the movement of the moving object 30 is changed based on whether the moving object 30 deviates from the virtual grid lane VGL.

[0115] Specifically, when the moving object 30 does not deviate from the virtual grid lane VGL, the control information generator 46 generates the guidance control information shown in the first embodiment and transmits it to the moving object 30. That is, the control information generator 46 guides the moving object 30 to the target point P t The control unit 30 transmits turning control information, straight-line control information, and the like to the moving body 30 in order to guide the moving body 30 to the target position.

[0116] On the other hand, if the moving body 30 deviates from the virtual grid lane VGL, the control information generator 46 generates return control information to cause the moving body 30 to perform return control to return the moving body 30 to within the virtual grid lane VGL, and transmits the return control information to the moving body 30.

[0117] Specifically, the control information generator 46 may generate a control information for the current traveling direction D i A direction D from the center line CL passing through the center of the width direction of the virtual grid lane VGL c Turning control information for turning the vehicle to face the direction D after the turn c The control unit 30 generates return control information including straight-line control information instructing the moving body 30 to move straight along the straight line, and transmits the return control information to the moving body 30.

[0118] When the moving object 30 receives the return control information and returns to the virtual grid lane VGL, that is, when the moving object 30 returns to the virtual grid lane VGL, the control information generator 46 determines that the moving object 30 has returned to the virtual grid lane VGL, and returns the moving object 30 to the target point P t The guidance control information for guiding the moving object 30 to the target position is generated and transmitted to the moving object 30.

[0119] Specific operations of the mobile object control device 10 in this embodiment will be described below using Fig. 16 and Fig. 17 while also referring to Fig. 18 to Fig. 20. Below, differences from the first embodiment will be mainly described.

[0120] Fig. 18 is a flowchart showing a guidance control routine RT4 for the moving body 30 executed by the information processing unit 17 of the moving body control device 10. The guidance control routine RT4 differs from Fig. 11 in that step S101 in Fig. 11 is changed to step S401 and step S402 is added.

[0121] In step S401, the reality / virtual information link unit 45 of the information processing unit 17 receives the virtual position P v In addition to identifying the vehicular grid lane, the virtual grid lane VGL is generated as shown in FIG.

[0122] In step S106, the control information generator 46 calculates the virtual position P1 and the target point P t If the magnitude of the vector V2 connecting the virtual grid lane VGL and the moving object 30 is not less than a predetermined value (step S106: NO), the lane return control is executed based on whether or not the moving object 30 has deviated from the virtual grid lane VGL (step S402).

[0123] Next, a description will be given of the subroutine RT5 for specifying the virtual position of the moving body 30 executed in step S401.

[0124] After step S205, the reality / virtual information link unit 45 of the information processing unit 17 acquires road shape information indicating the shape of a road from the current position of the vehicle V to a point a predetermined distance behind the current position of the vehicle V based on the current position of the vehicle V and the map information DB22B (step S501). Specifically, the reality / virtual information link unit 45 acquires road shape information indicating the shape of a road including a curved road CR behind the vehicle V as shown in FIG.

[0125] After step S501, the virtual space model generation unit 43 generates a three-dimensional virtual guide lane model on the XY virtual plane that conforms to the shape of the curved road CR included in the road shape information acquired by the reality virtual information link unit 45 (step S502).

[0126] After step S502, the reality / virtual information link unit 45 performs perspective transformation on the virtual guide lane model generated by the virtual space model generation unit 43 to generate a virtual guide lane VL in which the virtual guide lane model is viewed from the extension direction (step S503).

[0127] After step S503, the reality / virtual information link unit 45 generates a virtual grid lane VGL, which is a virtual guide lane VL having information of the grid model GM, by superimposing the virtual guide lane VL on the grid model GM of the virtual image VI, as shown in FIG. 16 (step S504).

[0128] After step S504, the reality / virtual information link unit 45 uses the virtual grid lane VGL generated by the reality / virtual information link unit 45 to specify the virtual position of the moving object 30 as described in the first embodiment (step S206).

[0129] In addition, in the above-mentioned virtual position specification subroutine RT5, steps S501 to S503 (generation of the virtual guide lane) may be performed before step S201. That is, the reality virtual information link unit 45 may generate the virtual guide lane VL in advance by performing perspective transformation of the virtual guide lane model.

[0130] Next, the lane return control subroutine executed in step S402 will be described below.

[0131] First, as shown in FIG. 17, the control information generator 46 determines whether or not the moving object 30 has deviated from the virtual grid lane VGL based on P1' that is the current virtual position of the moving object 30 (step S601).

[0132] When the control information generator 46 determines that the moving object 30 has not deviated from the virtual grid lane VGL (step S601: NO), it ends the lane return control subroutine RT6.

[0133] When the control information generator 46 determines that the moving object 30 deviates from the virtual grid lane VGL (step S601: YES), it executes control for returning the moving object 30 to the virtual grid lane VGL (step S602).

[0134] Specifically, the control information generator 46 determines the current traveling direction of the moving object 30 as shown in FIG. i A direction D from the center line CL passing through the center of the width direction of the virtual grid lane VGL c Turning control information for turning the vehicle to face the direction D after the turn c The control information generator 46 generates return control information including straight-ahead control information instructing the vehicle 30 to travel straight along the lane, and transmits the return control information to the vehicle 30. After step 602, the control information generator 46 ends the lane return control subroutine RT6.

[0135] According to the mobile object control device 10 of this embodiment, the control information generating unit 46 changes the control mode for the mobile object 30 based on whether or not the mobile object 30 has deviated from the virtual grid lane VGL as described above. As a result, even when the mobile object 30 is traveling on a winding road such as the curved road CR shown in FIG. 14, the mobile object 30 can be prevented from deviating from the curved road CR and can be guided to the target point P.t It can be guided to.

[0136] Therefore, according to the mobile object control system 200 of this embodiment, similar to the mobile object control system 100 shown in the first embodiment, it is possible to control the mobile object 30 to the target point P with an inexpensive configuration. t It is possible to reach this with high accuracy.

[0137] [Modification of Example 2] Next, a modified example of the mobile object control system 200 according to the second embodiment will be described with reference to Figs. 21 and 22. In this modified example, the method of generating the virtual guide lane VL is different from that of the second embodiment, and other points, such as the method of identifying the virtual position of the mobile object 30 and the method of identifying the target point P t The method of guidance and control is the same as in the second embodiment.

[0138] In this modification, the recognition model 22A stored in the mass storage device 22 has been trained on the characteristics of vehicles such as four-wheeled motor vehicles and motorcycles in addition to the characteristics of the moving body 30, and recognizes vehicles in images captured by the camera 12. The recognition model 22A receives as output, as the recognition result, for example, bounding box information indicating the position of the vehicle on the real image and the size of the vehicle on the real image.

[0139] 21 is a block diagram showing an example of a software configuration of the information processing section 17 in this modified example. In FIG. 21, the information processing section 17 newly includes a moving object recognition / moving area specification section 47.

[0140] The moving object recognition / moving area identification unit 47 identifies the position coordinates of the vehicle on the real image from the bounding box information acquired from the recognition model 22A, and uses the bounding box information to generate an information superimposed image SI in which a bounding box surrounding the vehicle is superimposed on the real image RI.

[0141] Fig. 22 is a diagram showing an example of an information superimposed image SI. In Fig. 22, vehicles C1 and C2 are traveling in a lane adjacent to a lane where lane restrictions are in place. In the following, the following description will be given on the assumption that vehicles C1 and C2 are constantly being imaged by the camera 12 and traveling from the back to the front in the figure.

[0142] In this modification, the moving object recognition / moving area identification unit 47 identifies the moving area of ​​the vehicle in the information superimposed image SI when it moves on the road for a predetermined time. Specifically, the moving object recognition / moving area identification unit 47 identifies the moving area MA1 of the vehicle C1 by tracing the bottom edge BL1 of the bounding box B1 when it recognizes the vehicle C1 traveling on the road as the vehicle C1 moves, that is, by tracing from the back to the front in the figure.

[0143] In addition, the moving object recognition / movement area identification unit 47, similar to vehicle C1, identifies the movement area MA2 of vehicle C2 by tracing the bottom edge BL2 of the bounding box B2 when vehicle C2 is recognized as vehicle C2 moves.

[0144] The moving object recognition / moving area identification unit 47 superimposes the identified moving area MA1 of vehicle C1 and the moving area MA2 of vehicle C2 to generate a single overlapping moving area SMA, and identifies the front end line of the overlapping moving area SMA as guideline GL1, the central line as guideline GL2, and the rear end line as guideline GL3.

[0145] The moving object recognition / moving area identification unit 47 generates a virtual guide lane VL based on each of the three guidelines GL1-GL3 in the superimposed moving area SMA. Specifically, the moving object recognition / moving area identification unit 47 generates the virtual guide lane VL by offsetting each of the guidelines GL1-GL3 in the real image RI by a predetermined distance in the direction of the arrow in the figure, and connecting the end points of the offset guidelines GL1-GL3 with curved lines.

[0146] In addition, the moving object recognition / moving area identification unit 47 generates a virtual grid lane VGL, which is a virtual guide lane VL having information of the grid model GM, by superimposing the virtual guide lane VL on the grid model GM of the virtual image VI, as in the second embodiment.

[0147] The control information generating unit 46 uses the virtual grid lane VGL generated by the moving object recognizing / moving area identifying unit 47 to determine whether or not the moving object 30 has deviated from the virtual grid lane VGL in the same manner as in the second embodiment, and determines the target point P t The moving object 30 can be guided to the position.

[0148] In this modified example, the moving object recognition / moving area identification unit 47 recognizes a vehicle (automobile) traveling on the road and identifies the moving area, but the recognition target may be a moving object other than an automobile as long as the moving object recognition / moving area identification unit 47 is capable of generating a virtual grid lane VGL.

[0149] For example, the moving object recognition / movement area identification unit 47 may identify the movement area of ​​the road worker by tracing the movement of the road worker as he walks from the front to the back in Figure 22 on a road where lane restrictions are in place. EXAMPLES

[0150] Next, a mobile object control system 300 according to a third embodiment will be described with reference to Fig. 23. Fig. 23 is a diagram showing the configuration of the mobile object control system 300, and is a diagram showing a vehicle V located on a road, a mobile object 50 as a first mobile object, and a mobile object 60 as a second mobile object, as viewed from above. Fig. 23 shows a state in which the vehicle V has passed through a curved road CR and is stopped at the bottom of the figure. In the following description, the downward direction in the figure is the front of the vehicle V.

[0151] In FIG. 23, the moving object 50 detects a target point P t1 The moving object 60 is guided to the target point P t1 Target point P, which is further back thant2 This shows the situation when traveling towards the

[0152] 23, the moving body 60 is hidden from view of the vehicle V by a plurality of vehicles C traveling in a lane adjacent to the lane where the lane regulation is in place, and therefore cannot be imaged by the camera 12. In other words, in FIG. 23, only the moving body 50 can be imaged by the camera 12.

[0153] 23, the mobile body control device 10 and the mobile body 50, and the mobile body 50 and the mobile body 60 are located at mutually communicable distances, while the mobile body control device 10 and the mobile body 60 are located at mutually incommunicable distances. In other words, the mobile body control device 10 does not directly move the mobile body 60 to the target point P t2 It has become impossible to guide and control the missile.

[0154] [Mobile configuration] Here, the configuration of the moving body 50 will be described with reference to Figures 24 and 25. Figure 24 is a block diagram showing an example of the hardware configuration of the moving body 50.

[0155] First, the hardware configuration of the moving body 50 will be described with respect to differences from the moving body 30 of Example 1. In this example, the moving body 50 has an input unit 51 connected to an information processing unit 34 via a system bus 33, and a camera 52 as an imaging unit and a large-capacity storage device 53 connected to the input unit 51, as shown in FIG.

[0156] The camera 52 is a camera that can be swiveled up, down, left and right in response to a control signal from the moving body 50. The camera 52 is, for example, a PTZ camera that can perform Pan, Tilt and Zoom in the same manner as the camera 12. The camera 52 is provided, for example, on the upper part of the moving body 50.

[0157] The large-capacity storage device 53 is a storage device in which a recognition model 53A is stored. The recognition model 53A has learned about the characteristics of roads, and recognizes road portions in captured images captured by the camera 52. The recognition model 53A outputs, as the recognition result, bounding box information indicating the position of the road portion in the captured image and the size of the road portion on the captured image.

[0158] The moving object recognition / image position identification unit 42 of the information processing unit 17 identifies the position coordinates of the road portion on the real image from the bounding box information acquired from the recognition model 53A, and uses the bounding box information to generate an information superimposed image in which, for example, an area indicating the road portion is superimposed on the captured image.

[0159] Next, the software configuration of the information processing unit 34 of the moving object 50 will be described with reference to Fig. 25. Fig. 25 is a block diagram showing an example of the software configuration of the information processing unit 34 in the moving object 50.

[0160] Similar to the first embodiment, the real image acquiring unit 55 sequentially acquires and outputs real images, which are images of the rear of the moving body 50 captured by the camera 52. In the present embodiment, the real image acquiring unit 55 transmits the real images to the moving body control device 10 sequentially or periodically.

[0161] Road recognition / camera direction control unit 56 inputs the captured image acquired by real image acquisition unit 55 to recognition model 53A, and controls the direction in which camera 52 faces based on the road in the information superimposed image generated by moving object recognition / image position identification unit 42. For example, road recognition / camera direction control unit 56 recognizes the edge of the road in the information superimposed image, and controls camera 52 so that the gaze point of camera 52 faces the center of the road.

[0162] Therefore, the moving body 50 can control the camera 52 by the road recognition / camera direction control unit 56 so that the camera 52 always faces the center of the road. t1 This allows the moving object 60 traveling behind the vehicle to be included within the viewing angle of the camera 52.

[0163] In this embodiment, the mobile object control device 10 acquires a real image in which the mobile object 60 is captured, transmitted from the mobile object 50, and specifies the virtual position of the mobile object 60 in the same manner as in the first embodiment. In addition, the mobile object control device 10 determines the specified virtual position and the target point P t2 Based on this, the moving body 60 is moved to the target point P t2 The guidance control information for guiding the moving body 50 to the target position is generated, and the guidance control information is transmitted to the moving body 50.

[0164] When the moving body 50 receives the guidance control information transmitted from the moving body control device 10, it transfers it to the moving body 60. The moving body 60, which has received the guidance control information transferred from the moving body 50, determines the target point P based on the guidance control information. t2 In this way, even if the camera 12 cannot capture an image of the moving object 60, the moving object control device 10 can control the moving object 60 to the target point P t2 It can be guided to.

[0165] Therefore, in this embodiment, the mobile body control device 10 functions as a base station that guides the mobile body 50 and the mobile body 60 to their respective target points, and the mobile body 50 functions as a relay station that transmits captured images including the mobile body 60 to the mobile body control device 10 and transfers the guidance control information for the mobile body 60 transmitted from the mobile body control device 10 to the mobile body 60.

[0166] According to the mobile object control system 300 of this embodiment, even in a case where the camera 12 mounted on the vehicle V cannot directly capture an image of the mobile object, the mobile object can be guided to the target location by functioning as a relay station as described above.

[0167] Furthermore, according to the mobile object control system 300 of this embodiment, for example, in the case where a plurality of mobile objects are arranged one by one at a predetermined interval, even when the mobile object control device 10 is far enough away from the mobile object that it cannot directly communicate with the mobile object, it is possible to guide the mobile object to the target point. Therefore, the mobile object control system 300 of this embodiment can also accurately guide the mobile object to the target point with an inexpensive configuration.

[0168] In this embodiment, the mobile body control device 10 controls the mobile body 60, but the mobile body 50 may have the mobile body recognition model and mobile body control function of the mobile body control device 10. In other words, the mobile body 50 may directly guide and control the mobile body 60 to the target point. [Explanation of symbols]

[0169] 100, 200, 300 Mobile Control System 10 Mobile control device 12, 52 Camera 13 Touch panel display 16, 33 System Bus 17, 34 Control section 18, 35 Communications Department 19, 51 Input section 21, 37 Output section 22, 53 mass storage device 30, 50, 60 Mobile 31 Running gear 41, 55 Reality image acquisition unit 42 Moving object recognition and image position determination unit 43 Virtual space model generation unit 44 Virtual image generation unit 45 Reality / Virtual Information Link Division 46 Control information generation unit 56 Road recognition / camera direction control unit

Claims

1. A mobile object control device that controls movement of a mobile object that can travel on a road, a real image acquisition unit that acquires a real image that is an image captured by a real camera that captures an image of a real space; a moving object recognition / image position specification unit that recognizes the moving object in the real image and specifies a position of the moving object on the real image; a virtual space model generating unit that generates a virtual space model having grid lines arranged on a predetermined virtual plane; a virtual image generating unit that performs perspective transformation on the virtual space model using a virtual camera to generate a virtual image in which the grid lines are viewed from an extension direction of the grid lines; a real-virtual information link unit that specifies a virtual position of the moving object on the predetermined virtual plane based on a relative positional relationship between a point in the virtual image having the same coordinates as real image position coordinates that are position coordinates of the moving object in the real image and the grid lines; a control information generating unit that generates guidance control information for guiding the moving object to the target point based on the virtual position and a position of a target point of movement of the moving object on the predetermined virtual plane; A mobile object control device comprising:

2. The mobile body control device according to claim 1, characterized in that the reality / virtual information link unit generates a transformation curve that converts a virtual image vertical direction coordinate, which is the vertical coordinate in the virtual image of the one point in the virtual image, into a virtual depth direction coordinate, which is the coordinate in the extension direction of the grid lines of the specified virtual plane, calculates a virtual depth direction coordinate of the moving body in the specified virtual plane based on the transformation curve and the virtual image vertical direction coordinate, identifies two points on the grid lines whose virtual image vertical direction coordinates are the same as the coordinate of the one point, and identifies a virtual width direction coordinate of the moving body, which is the coordinate in the direction perpendicular to the grid lines on the specified virtual plane, based on the distance between the one point and each of the two points.

3. The mobile body control device according to claim 1, characterized in that the control information generation unit generates guidance control information that guides the mobile body so that an angle formed by a first vector connecting the initial position and current position of the mobile body on the specified virtual plane and a second vector connecting the current position of the mobile body and the target point is equal to or less than a specified value.

4. the reality / virtual information link unit generates a virtual grid lane in the virtual image for guiding the moving object to the target point; 2. The mobile body control device according to claim 1, wherein the control information generation unit determines whether the moving body has deviated from the virtual grid lane based on a position of the moving body on the virtual image and the virtual grid lane, generates the guidance control information when the moving body is traveling within the virtual grid lane, and generates return control information for returning the moving body to within the virtual grid lane when the moving body has deviated from the virtual grid lane.

5. a moving object recognition / moving area specification unit that recognizes a moving object moving on a road in the real image and specifies a moving area through which the moving object has moved; The mobile object control device according to claim 4 , wherein the reality / virtual information link unit generates the virtual grid lane based on the movement area.

6. the real image acquisition unit acquires a real image by zooming the real camera at a predetermined magnification based on the virtual depth direction coordinate of the moving object; The mobile object control device according to claim 2 , wherein the virtual image generating unit zooms the virtual camera in accordance with a zoom magnification of the real camera.

7. A mobile object control system including the mobile object control device according to claim 1 and a plurality of mobile objects capable of traveling on a road, One of the plurality of moving bodies includes an imaging unit that images other moving bodies, and transmits an image including the other moving body to the moving body control device; The mobile body control system is characterized in that the mobile body control device identifies the virtual position of the other moving body based on the captured image transmitted from the one moving body, and performs guidance control for the other moving body to the target point based on the virtual position and the position of a target point for the movement of the other moving body on the specified virtual plane.

8. the mobile object control device generates guidance control information for guiding the other mobile object to the target point and transmits the guidance control information to the one mobile object; 8. The mobile body control system according to claim 7, wherein the one mobile body transfers the guidance control information transmitted from the mobile body control device to the other mobile body.

Images