Information Processing Systems

Abstract

A technology is provided that can prevent a decrease in the accuracy of estimating the position of a moving body. [Solution] The information processing system comprises a mobile body that can be moved by unmanned operation, a camera that captures images of the mobile body from outside and outputs captured images, a plurality of position estimation units that estimate the position of the mobile body using different logic for a single captured image using camera parameters that are determined according to the position and orientation of the camera and indicate the correspondence between local coordinates and global coordinates, and a parameter control unit that performs a matching process to match the camera parameters used by the plurality of position estimation units when each estimates the position of the mobile body.

Description

[Technical Field]

[0001] The present disclosure relates to an information processing system. [Background technology]

[0002] BACKGROUND ART Conventionally, a technique is known in which a vehicle is driven autonomously by monitoring the vehicle's movement using a camera mounted outside the vehicle (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2017-538619 Summary of the Invention [Problem to be solved by the invention]

[0004] In order to operate a vehicle in an unmanned driving mode, the vehicle's position may be estimated using captured images output from a camera installed outside the vehicle. In this case, the vehicle's position may be estimated using multiple different logics for a single captured image using camera parameters. If the camera parameters used when estimating the vehicle's position using multiple logics do not match, the accuracy of the vehicle's position estimation may decrease. This issue is not limited to vehicles, but is common to all moving objects. [Means for solving the problem]

[0005] The present disclosure can be realized in the following forms.

[0006] (1) According to one aspect of the present disclosure, an information processing system is provided. The information processing system includes a mobile object capable of moving by unmanned operation, a camera that captures an image of the mobile object from outside the mobile object and outputs the captured image, multiple position estimation units that estimate the position of the mobile object using different logics for one captured image using camera parameters determined according to the position and attitude of the camera, the camera parameters indicating a correspondence between local coordinates and global coordinates, and a parameter control unit that performs a matching process to match the camera parameters used by the multiple position estimation units when estimating the position of the mobile object. According to this aspect, the information processing system can match the camera parameters used by the multiple position estimation units when estimating the position of the mobile object. As a result, the information processing system can prevent a decrease in the estimation accuracy of the position of the mobile object when estimating the position of the mobile object using different logics for one captured image using the camera parameters. (2) In the above aspect, the information processing system may further include a memory that stores a database containing the camera parameters, and the matching process may be a process of transmitting instruction information to each of the plurality of position estimation units so that the plurality of position estimation units use, from the camera parameters in the database, camera parameters for the camera that outputs the captured image that are used when estimating the position of the moving object. According to this aspect, the information processing system can match the camera parameters used when estimating the position of the moving object by transmitting instruction information to each of the plurality of position estimation units so that the same camera parameters in the database are used. (3) In the above configuration, the system further includes a matching unit that compares the camera parameters used by each of the multiple position estimation units when estimating the position of the moving body, and if the camera parameters used by each of the multiple position estimation units when estimating the position of the moving body do not match, the parameter control unit may perform at least one of the following processes: matching process, notification process that notifies the user of error information regarding the camera parameters, and stop process that stops the moving body. In this configuration, if the camera parameters used by the multiple position estimation units when estimating the position of the moving body do not match, the information processing system can correct the deficiency of the camera parameters by performing the matching process. Furthermore, if the camera parameters used by the multiple position estimation units when estimating the position of the moving body do not match, the information processing system can notify the user of error information by performing the notification process. This allows the information processing system to prompt the user to take measures to match the camera parameters. In this way, the information processing system can further suppress a decrease in the accuracy of estimating the position of the moving body by performing at least one of the matching process and the notification process. In addition, if the camera parameters used by multiple position estimation units when estimating the position of a moving body do not match, the information processing system can execute a stop process to stop the moving body if there is a risk that the estimation accuracy of the moving body's position will decrease. (4) The above aspect further includes a memory that stores the database including the camera parameters, and a comparison unit that compares, among the camera parameters in the database, the camera parameters for the target camera that output the captured image used by the plurality of position estimation units when estimating the position of the moving body with the camera parameters used by the plurality of position estimation units when estimating the position of the moving body, and when, among the camera parameters in the database, the camera parameters for the target camera do not match with the camera parameters used by at least some of the plurality of position estimation units when estimating the position of the moving body, the parameter control unit may execute at least one of the matching process, a notification process that notifies a user of error information regarding the camera parameters, and a stop process that stops the moving body. According to this aspect, when the camera parameters for the target camera in the database do not match with the camera parameters used by at least one of the plurality of position estimation units when estimating the position of the moving body, the information processing system can correct the deficiency of the camera parameters by executing the matching process. Furthermore, when the camera parameters of the target camera in the database do not match the camera parameters used by at least one of the multiple position estimation units when estimating the position of the moving object, the information processing system can notify the user of error information by executing a notification process. This allows the information processing system to prompt the user to take measures to make the camera parameters match. In this way, by executing at least one of the matching process and the notification process, the information processing system can further suppress a decrease in the estimation accuracy of the moving object's position. Furthermore, when the camera parameters of the target camera in the database do not match the camera parameters used by at least one of the multiple position estimation units when estimating the position of the moving object, the information processing system can execute a stop process to stop the moving object if there is a risk of a decrease in the estimation accuracy of the moving object's position. (5) In the above configuration, the matching unit may identify the camera parameters used by each of the multiple position estimation units when estimating the position of the moving object, based on the correspondence between the local coordinates of predetermined feature points in the captured image used by each of the multiple position estimation units when estimating the position of the moving object, and the global coordinates corresponding to the feature points. According to this configuration, the information processing system can identify the camera parameters used by each of the multiple position estimation units when estimating the position of the moving object, based on the correspondence between the local coordinates of feature points in the captured image and the global coordinates corresponding to the feature points. This disclosure can be implemented in various forms other than the information processing system described above. For example, it can be implemented in the form of a method for manufacturing an information processing system, a method for controlling an information processing system, an information processing method, a computer program that implements these methods, or a non-temporary recording medium on which the computer program is stored. [Brief explanation of the drawings]

[0007] [Figure 1] A conceptual diagram showing the configuration of the information processing system in the first embodiment. [Figure 2] A block diagram showing the configuration of an information processing system. [Figure 3] A flowchart illustrating the processing procedure for vehicle driving control in the first embodiment. [Figure 4] A diagram illustrating the vehicle position estimation method by the first position estimation unit. [Figure 5] A diagram illustrating the vehicle position estimation method used by the second position estimation unit. [Figure 6] A flowchart illustrating an example of an information processing method. [Figure 7] An explanatory diagram showing the schematic configuration of the information processing system in the second embodiment. [Figure 8] A flowchart illustrating the processing procedure for vehicle driving control in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] A. First embodiment: 1 is a conceptual diagram showing the configuration of an information processing system 50 according to the first embodiment. The information processing system 50 includes one or more vehicles 100 as moving objects, a server 200, and one or more external sensors 300.

[0009] In this disclosure, a "mobile body" refers to an object that can move, such as a vehicle or an electric vertical take-off and landing aircraft (a so-called flying car). A vehicle may be a vehicle that runs on wheels or a vehicle that runs on tracks, such as a passenger car, truck, bus, motorcycle, automobile, or construction vehicle. Vehicles include electric vehicles (BEVs: Battery Electric Vehicles), gasoline-powered vehicles, hybrid vehicles, and fuel cell vehicles. When a mobile body is something other than a vehicle, the terms "vehicle" and "car" in this disclosure may be appropriately replaced with "mobile body," and the term "traveling" may be appropriately replaced with "moving."

[0010] The vehicle 100 is configured to be capable of traveling in an unmanned manner. "Unmanned driving" refers to driving without the driver's control. Driving operation refers to operations related to at least one of "running," "turning," and "stopping" of the vehicle 100. Unmanned driving is achieved by automatic or manual remote control using a device located outside the vehicle 100, or by autonomous control of the vehicle 100. A vehicle 100 traveling in an unmanned manner may have a driver on board who does not operate the vehicle. A driver who does not operate the vehicle may, for example, simply be seated in the vehicle 100, or a person who is riding in the vehicle 100 and performing work other than driving operations, such as assembly, inspection, or operating switches. Driving in which a driver controls the vehicle is sometimes called "manned driving."

[0011] In this specification, "remote control" includes "full remote control" in which all of the operations of vehicle 100 are completely determined from outside vehicle 100, and "partial remote control" in which some of the operations of vehicle 100 are determined from outside vehicle 100. Furthermore, "autonomous control" includes "full autonomous control" in which vehicle 100 autonomously controls its own operations without receiving any information from devices external to vehicle 100, and "partial autonomous control" in which vehicle 100 autonomously controls its own operations using information received from devices external to vehicle 100.

[0012] In this embodiment, the information processing system 50 is used in a factory FC where the vehicle 100 is manufactured. The reference coordinate system of the factory FC is a global coordinate system GC, and any position in the factory FC can be expressed by X, Y, and Z coordinates in the global coordinate system GC. The factory FC has a first location PL1 and a second location PL2. The first location PL1 and the second location PL2 are connected by a road TR along which the vehicle 100 can travel. A plurality of external sensors 300 are installed along the road TR in the factory FC. The position of each external sensor 300 in the factory FC is adjusted in advance. The vehicle 100 moves from the first location PL1 to the second location PL2 along the road TR in an unmanned operation.

[0013] The external sensor 300 is a sensor located outside the vehicle 100. In this embodiment, the external sensor 300 is a sensor that captures the vehicle 100 from outside the vehicle 100. The external sensor 300 is equipped with a communication device (not shown) and can communicate with other devices such as the server 200 via wired communication or wireless communication. Specifically, the external sensor 300 is configured by a camera 310. The camera 310 as the external sensor 300 captures an image of the vehicle 100 and outputs a captured image CI as a detection result.

[0014] 2 is a block diagram showing the configuration of the information processing system 50. The vehicle 100 includes a vehicle control device 110 for controlling each part of the vehicle 100, an actuator group 120 including one or more actuators that are driven under the control of the vehicle control device 110, and a communication device 130 for communicating via wireless communication with external devices such as a server 200. The actuator group 120 includes an actuator for a drive device for accelerating the vehicle 100, an actuator for a steering device for changing the traveling direction of the vehicle 100, and an actuator for a braking device for decelerating the vehicle 100.

[0015] The vehicle control device 110 is configured by a computer including a processor 111, a memory 112, an input / output interface 113, and an internal bus 114. The processor 111, the memory 112, and the input / output interface 113 are connected via the internal bus 114 to enable bidirectional communication. The input / output interface 113 is connected to an actuator group 120 and a communication device 130. The processor 111 functions as a vehicle control unit 115 by executing a program PG1 stored in the memory 112.

[0016] The vehicle control unit 115 controls the actuator group 120 to cause the vehicle 100 to run. The vehicle control unit 115 controls the actuator group 120 using a running control signal received from the server 200 to cause the vehicle 100 to run. The running control signal is a control signal for causing the vehicle 100 to run. In this embodiment, the running control signal includes the acceleration and steering angle of the vehicle 100 as parameters. In other embodiments, the running control signal may include the speed of the vehicle 100 as a parameter instead of or in addition to the acceleration of the vehicle 100.

[0017] The server 200 is composed of a computer comprising a processor 201, memory 202, an input / output interface 203, and an internal bus 204. The processor 201, memory 202, and input / output interface 203 are connected via the internal bus 204 to enable bidirectional communication. A communication device 205 for communicating with various devices outside the server 200 is connected to the input / output interface 203. The communication device 205 can communicate with the vehicle 100 via wireless communication and can communicate with each external sensor 300 via wired or wireless communication. The processor 201 functions as a plurality of position estimation units 211, 212, a matching unit 213, a parameter control unit 214, and a remote control unit 215 by executing a program PG2 stored in memory 202.

[0018] Each of the multiple position estimation units 211 and 212 uses camera parameters to estimate the position of the vehicle 100 for a single captured image CI using different logic. Then, each of the multiple position estimation units 211 and 212 outputs the estimated position of the vehicle 100 to the remote control unit 215. As shown in Figure 1, in this embodiment, the position of the vehicle 100 is represented as coordinates indicating the position of a pre-set positioning point 101 for a specific part of the vehicle 100.

[0019] The camera parameters are parameters for each camera 310 determined according to the position and orientation of the camera 310. The camera parameters indicate the correspondence between local coordinates LP represented by a local coordinate system LC and global coordinates GP represented by a global coordinate system GC. The camera parameters are, for example, a transformation matrix for converting the coordinates LP and GP between the local coordinate system LC and the global coordinate system GC. The camera parameters are identified, for example, by capturing a pattern, from the correspondence between the local coordinates LP of predetermined feature points FP1 to FP3 in the captured image CI and the global coordinates GP corresponding to the feature points FP1 to FP3. The camera parameters of each camera 310 are stored in a database DB that indicates the camera parameters for each camera 310. The camera parameters in the database DB are updated, for example, at a predetermined interval. The camera parameters in the database DB may be updated when at least one of the position and orientation of the camera 310 changes.

[0020] Of the multiple position estimation units 211, 212, the first position estimation unit 211 estimates the position of the vehicle 100 from a captured image CI by executing a first application AP1 from a program PG2 stored in the memory 202. The first application AP1 is one of multiple applications AP that are made redundant so that the position of the vehicle 100 can be estimated for one captured image CI using different logics by using camera parameters. Of the multiple position estimation units 211, 212, the second position estimation unit 212 executes a second application AP2 from the program PG2 stored in the memory 202 to estimate the position of the vehicle 100 from the same captured image CI as the first position estimation unit 211 using logic different from that of the first position estimation unit 211. The second application AP2 is one of multiple redundant applications AP and is an application AP different from the first application AP1.

[0021] The camera parameters used by each of the multiple position estimation units 211 and 212 are updated at specific timings. For example, if the camera parameters in the database DB are updated, the camera parameters used by each of the multiple position estimation units 211 and 212 are updated. The camera parameters used by each of the multiple position estimation units 211 and 212 may be updated before the control of unmanned operation is started. If the application AP is started when the control of unmanned operation is started, the camera parameters used by each of the multiple position estimation units 211 and 212 may be updated when the application AP is started. The camera parameters used by each of the multiple position estimation units 211 and 212 may be updated when the application AP is restarted. The camera parameters used by each of the multiple position estimation units 211 and 212 may be updated manually by the user or automatically by machine. If the camera parameters used by each of the multiple position estimation units 211 and 212 are automatically updated by machine, for example, the following process is executed. In this case, each of the multiple position estimation units 211 and 212 refers to the database DB at specific timings to obtain the camera parameters for the target camera 310. Then, each of the position estimation units 211, 212 updates the camera parameters used when estimating the position of the vehicle 100 to the acquired camera parameters. Note that the method of estimating the position of the vehicle 100 by each of the position estimation units 211, 212 will be described later.

[0022] The matching unit 213 identifies the camera parameters used by each of the multiple position estimation units 211 and 212 when estimating the position of the vehicle 100. The matching unit 213 identifies the camera parameters used by each of the multiple position estimation units 211 and 212 when estimating the position of the vehicle 100, for example, from the correspondence between the local coordinates LP and global coordinates GP of feature points FP1 to FP3 in the captured image CI. The matching unit 213 may also identify the camera parameters used by each of the multiple position estimation units 211 and 212 when estimating the position of the vehicle 100, based on the acquisition history when camera parameters were acquired by referring to the database DB. The matching unit 213 then performs a matching process to match the multiple camera parameters, and executes at least one of a first matching process and a second matching process. The first matching process is a process of matching the camera parameters used by each of the multiple position estimation units 211 and 212 when estimating the position of the vehicle 100. The second matching process involves comparing the camera parameters for the target camera 310 from the database DB with the camera parameters used by the multiple position estimation units 211 and 212 when estimating the position of the vehicle 100. The target camera 310 is the camera that outputs the captured image CI used by the multiple position estimation units 211 and 212 when estimating the position of the vehicle 100. The matching unit 213 outputs the matching results of the multiple camera parameters to the parameter control unit 214.

[0023] The parameter control unit 214 performs at least one of the following processes in at least one of the first and second cases: matching process, notification process, and stop process. The first case is when the camera parameters used by the multiple position estimation units 211 and 212 when estimating the position of the vehicle 100 do not match. The second case is when the camera parameters for the target camera 310 in the database DB do not match the camera parameters used by at least one of the multiple position estimation units 211 and 212 when estimating the position of the vehicle 100.

[0024] The matching process is a process of matching camera parameters used by the multiple position estimation units 211, 212 when estimating the position of the vehicle 100. The matching process is a process of transmitting instruction information to each of the multiple position estimation units 211, 212 to use camera parameters for the target camera 310 from among the camera parameters in the database DB, for example. In the matching process, the parameter control unit 214 sets a flag for the camera parameters for the target camera 310 from among the camera parameters in the database DB, for example. Then, the parameter control unit 214 transmits instruction information to each of the multiple position estimation units 211, 212 to use the flagged camera parameters. Note that the parameter control unit 214 may match camera parameters used by the multiple position estimation units 211, 212 when estimating the position of the vehicle 100 by using another method.

[0025] The notification process is a process of notifying the user of error information related to camera parameters. In the notification process, the parameter control unit 214 notifies the user of the error information, for example, by displaying text information as the error information on a display (not shown). In the notification process, the parameter control unit 214 may notify the user of the error information by playing audio information as the error information from a speaker (not shown). In the notification process, the parameter control unit 214 may notify the user of the error information by operating various devices mounted on the vehicle 100, such as lamps and a horn, in a predetermined pattern.

[0026] The stopping process is the process of stopping the vehicle 100. In this embodiment, during the stopping process, the parameter control unit 214 generates a stop control signal to stop the vehicle 100. The parameter control unit 214 then transmits the stop control signal to the vehicle 100, thereby stopping the vehicle 100 by remote control.

[0027] The remote control unit 215 acquires the position estimation result of the vehicle 100 from the position estimation units 211 and 212, and uses the position estimation result of the vehicle 100 to generate a driving control signal for controlling the actuator group 120 of the vehicle 100. Then, the remote control unit 215 transmits the driving control signal to the vehicle 100, thereby driving the vehicle 100 by remote control.

[0028] FIG. 3 is a flowchart showing a processing procedure for driving control of the vehicle 100 in the first embodiment.

[0029] In step S1, the processor 201 of the server 200 acquires vehicle position information using the detection results output from the external sensor 300. The vehicle position information is the position information that forms the basis for generating the driving control signal. In this embodiment, the vehicle position information includes the position and orientation of the vehicle 100 in the global coordinate system GC of the factory FC. Specifically, in step S1, the processor 201 acquires vehicle position information using the captured image CI acquired from the camera 310, which is the external sensor 300.

[0030] In detail, in step S1, the processor 201 detects the outline of the vehicle 100 from the captured image CI, calculates the coordinates of the positioning point 101 of the vehicle 100 in the coordinate system of the captured image CI, i.e., the local coordinate system LC, and obtains the position of the vehicle 100 by converting the calculated coordinates to coordinates in the global coordinate system GC. The outline of the vehicle 100 included in the captured image CI can be detected, for example, by inputting the captured image CI into a detection model DM that utilizes artificial intelligence. The detection model DM is prepared, for example, within or outside the information processing system 50 and stored in the memory 202 of the server 200. Examples of the detection model DM include a pre-trained machine learning model that has been trained to implement either semantic segmentation or instance segmentation. As this machine learning model, for example, a convolutional neural network (CNN) trained by supervised learning using a training dataset can be used. The training dataset includes, for example, multiple training images containing vehicle 100, and labels indicating whether each region in the training images represents vehicle 100 or something other than vehicle 100. During CNN training, it is preferable to update the CNN parameters using backpropagation to reduce the error between the output result of the detection model DM and the labels. The processor 201 can also obtain the orientation of vehicle 100 by, for example, using the optical flow method, estimating the orientation of the vehicle 100's movement vector calculated from the positional changes of the vehicle 100's feature points between frames of the captured image CI.

[0031] In step S2, the processor 201 of the server 200 determines a target position to which the vehicle 100 should next head. In this embodiment, the target position is represented by X, Y, and Z coordinates in the global coordinate system GC. A reference route RR, which is a route to be traveled by the vehicle 100, is stored in advance in the memory 202 of the server 200. The route is represented by nodes indicating the departure point, nodes indicating passing points, nodes indicating the destination, and links connecting the nodes. The processor 201 uses the vehicle position information and the reference route RR to determine a target position to which the vehicle 100 should next head. The processor 201 determines a target position on the reference route RR that is ahead of the current location of the vehicle 100.

[0032] In step S3, the processor 201 of the server 200 generates a travel control signal for causing the vehicle 100 to travel toward the determined target position. The processor 201 calculates the travel speed of the vehicle 100 from the change in the position of the vehicle 100 and compares the calculated travel speed with the target speed. When the travel speed is lower than the target speed, the processor 201 determines an acceleration such that the vehicle 100 accelerates. When the travel speed is higher than the target speed, the processor 201 determines an acceleration such that the vehicle 100 decelerates. Furthermore, when the vehicle 100 is located on the reference route RR, the processor 201 determines a steering angle and acceleration such that the vehicle 100 does not deviate from the reference route RR. When the vehicle 100 is not located on the reference route RR, in other words, when the vehicle 100 has deviated from the reference route RR, the processor 201 determines a steering angle and acceleration such that the vehicle 100 returns to the reference route RR.

[0033] In step S4, the processor 201 of the server 200 transmits the generated driving control signal to the vehicle 100. The processor 201 repeats, at a predetermined cycle, the acquisition of vehicle position information, the determination of a target position, the generation of a driving control signal, and the transmission of the driving control signal.

[0034] In step S5, the processor 111 of the vehicle 100 receives a driving control signal transmitted from the server 200. In step S6, the processor 111 of the vehicle 100 controls the actuator group 120 using the received driving control signal, thereby driving the vehicle 100 at the acceleration and steering angle indicated in the driving control signal. The processor 111 repeats the reception of the driving control signal and the control of the actuator group 120 at predetermined intervals. According to the information processing system 50 in this embodiment, the vehicle 100 can be driven by remote control, and the vehicle 100 can be moved without using transport equipment such as cranes or conveyors.

[0035] Figure 4 is a diagram illustrating the method for estimating the position of vehicle 100 by the first position estimation unit 211. The first position estimation unit 211 calculates the local coordinates LP of the positioning point 101 of vehicle 100 based on the contour of vehicle 100, and estimates the position of vehicle 100 by converting the local coordinates LP to global coordinates GP using camera parameters. Specifically, in step S101, the first position estimation unit 211 acquires the captured image CI. In step S102, the first position estimation unit 211 detects the contour of vehicle 100 from the captured image CI. Then, the first position estimation unit 211 generates a masked image MI by masking the region representing vehicle 100 from each region constituting the captured image CI, thereby adding a masked region MS to the captured image CI. In step S103, the first position estimation unit 211 generates a bird's-eye view image PI from a viewpoint above vehicle 100 that is approximately perpendicular to the road surface RS by performing a perspective transformation on the masked image MI. As a result, the first position estimation unit 211 generates a bird's-eye view image PI represented in image coordinate system IC from a mask image MI represented in camera coordinate system CC. Camera coordinate system CC is a local coordinate system LC with the focal point of camera 310 as the origin. Image coordinate system IC is a local coordinate system LC with a point on the image plane as the origin. In step S104, the first position estimation unit 211 calculates the image coordinate IP corresponding to the positioning point 101 of vehicle 100 in image coordinate system IC from the bird's-eye view image PI as the local coordinate LP of the positioning point 101 of vehicle 100. In step S105, the first position estimation unit 211 converts the image coordinate IP to global coordinate GP using camera parameters. As a result, the first position estimation unit 211 estimates the position of vehicle 100.

[0036] 5 is a diagram illustrating a method for estimating the position of the vehicle 100 by the second position estimation unit 212. The second position estimation unit 212 calculates local coordinates LP of the positioning point 101 of the vehicle 100 based on the contact surface of the vehicle 100 with the road surface RS, and estimates the position of the vehicle 100 by converting the local coordinates LP into global coordinates GP using camera parameters. Specifically, in step S201, the second position estimation unit 212 acquires a captured image CI. In step S202, the second position estimation unit 212 generates a circumscribing rectangle RC that surrounds the area occupied by the vehicle 100 when the vehicle 100 is projected onto the road surface RS. Then, the second position estimation unit 212 calculates local coordinates LP of four vertices VC1 to VC4 of the circumscribing rectangle RC. In step S203, the second position estimation unit 212 calculates local coordinates LP of the positioning point 101 of the vehicle 100 based on the relative positional relationship between each of the vertices VC1 to VC4 of the circumscribing rectangle RC and the positioning point 101 of the vehicle 100. In step S204, the second position estimation unit 212 converts the local coordinates LP of the positioning point 101 of the vehicle 100 into global coordinates GP using the camera parameters. In this way, the second position estimation unit 212 estimates the position of the vehicle 100.

[0037] FIG. 6 is a flowchart showing an example of an information processing method. The flow shown in FIG. 6 is executed repeatedly at a predetermined cycle, for example, during a period when unmanned driving control is being executed. In step S301, the collation unit 213 identifies the camera parameters used by the multiple position estimation units 211, 212 when estimating the position of the vehicle 100. In step S302, the collation unit 213 executes a first collation process. If the camera parameters used by the multiple position estimation units 211, 212 when estimating the position of the vehicle 100 match each other (step S303: Yes), this flow ends. On the other hand, if the camera parameters used by the multiple position estimation units 211, 212 when estimating the position of the vehicle 100 do not match each other (step S303: No), in step S304, the parameter control unit 214 executes at least one of a matching process, a notification process, and a stop process.

[0038] According to the above embodiment, the information processing system 50 can estimate the position of the vehicle 100 using the camera parameters with the multiple position estimation units 211, 212 and multiple different logics for one captured image CI. In this case, if at least one of the position and attitude of the camera 310 changes, it is preferable to use the camera parameters after the position and attitude change when estimating the position of the vehicle 100 using any of the multiple logics. Therefore, the camera parameters used by each of the multiple position estimation units 211, 212 are updated at specific times. However, if the camera parameters used by each of the multiple position estimation units 211, 212 are manually updated by a user when at least one of the position and attitude of the camera 310 changes, the following possibilities arise. In this case, the user may forget to update the camera parameters used by some of the multiple position estimation units 211, 212. Furthermore, if the camera parameters used by each of the multiple position estimation units 211, 212 are automatically updated when at least one of the position and attitude of the camera 310 changes, the following possibilities arise. In this case, due to a communication interruption or the like, some of the position estimation units 211, 212 may be unable to acquire the camera parameters after the position or attitude has changed. In this case, some of the position estimation units 211, 212 may use the camera parameters stored in the cache memory before the position or attitude has changed. Due to these various possibilities, different camera parameters may be used by some of the multiple position estimation units 211, 212 when estimating the position of the vehicle 100. If the camera parameters used to estimate the position of the vehicle 100 using multiple logics do not match, the accuracy of estimating the position of the vehicle 100 may be reduced. In response to this, according to the above embodiment, the information processing system 50 can, by the parameter control unit 214, match the camera parameters used by the multiple position estimation units 211, 212 when estimating the position of the vehicle 100. This allows the information processing system 50 to correct any deficiencies in the camera parameters.Therefore, when the information processing system 50 estimates the position of the vehicle 100 using different logic for a single captured image CI with camera parameters, it can suppress a decrease in the accuracy of estimating the position of the vehicle 100.

[0039] Furthermore, according to the above embodiment, the information processing system 50 can perform the following processing using the parameter control unit 214. The information processing system 50 can send instruction information to each of the multiple position estimation units 211, 212 to use the camera parameters for the target camera 310 from the camera parameters in the database DB. In this way, the information processing system 50 can make the camera parameters used by the multiple position estimation units 211, 212 when estimating the position of the vehicle 100 match by having them refer to the same camera parameters in the database DB.

[0040] Furthermore, according to the above embodiment, if at least one of the position and orientation of the camera 310 changes, it is preferable to use the camera parameters after the change in position and orientation when estimating the position of the vehicle 100 using any of the multiple logics. However, if the old camera parameters from before the change in position and orientation are used when estimating the position of the vehicle 100 using any of the multiple logics, there is a risk that the accuracy of estimating the position of the vehicle 100 will decrease. In contrast, according to the above embodiment, if at least one of the position and orientation of the camera 310 changes, the information processing system 50 can update the camera parameters in the database DB. This allows the information processing system 50 to store the latest camera parameters corresponding to the changed position and orientation in the database DB. In this way, by having the multiple position estimation units 211 and 212 each use the camera parameters for the target camera 310 in the database DB, the information processing system 50 can improve the accuracy of estimating the position of the vehicle 100.

[0041] Furthermore, according to the above embodiment, the information processing system 50 can use the matching unit 213 to compare the camera parameters used by the multiple position estimation units 211 and 212 when estimating the position of the vehicle 100. This allows the information processing system 50 to confirm whether or not the position of the vehicle 100 was estimated using the same camera parameters.

[0042] Furthermore, according to the above embodiment, if the camera parameters used by the multiple position estimation units 211 and 212 when estimating the position of the vehicle 100 do not match, the information processing system 50 can perform a matching process using the parameter control unit 214. In this way, when there is a risk that the accuracy of estimating the position of the vehicle 100 will decrease due to the camera parameters not matching, the information processing system 50 can make the camera parameters used by the multiple position estimation units 211 and 212 match. As a result, the information processing system 50 can further suppress a decrease in the accuracy of estimating the position of the vehicle 100.

[0043] Furthermore, the information processing system 50 may perform a matching process when updating the camera parameters used by each of the multiple position estimation units 211 and 212. In this way, the information processing system 50 can pre-match the camera parameters used by the multiple position estimation units 211 and 212 before estimating the position of the vehicle 100. This further suppresses a decrease in the accuracy of the vehicle 100 position estimation.

[0044] Furthermore, according to the above embodiment, if the camera parameters used by the multiple position estimation units 211 and 212 to estimate the position of the vehicle 100 do not match, the information processing system 50 can perform a notification process using the parameter control unit 214. In this way, the information processing system 50 can notify the user of error information regarding the camera parameters. This allows the information processing system 50 to prompt the user to take measures to match the camera parameters. Therefore, the information processing system 50 can further suppress a decrease in the accuracy of estimating the position of the vehicle 100.

[0045] Furthermore, according to the above embodiment, if the camera parameters used by the multiple position estimation units 211 and 212 to estimate the position of the vehicle 100 do not match, the information processing system 50 can perform a stop process using the parameter control unit 214. In this way, if there is a risk that the accuracy of estimating the position of the vehicle 100 will decrease due to the mismatch of camera parameters, the information processing system 50 can stop the vehicle 100.

[0046] Furthermore, according to the above embodiment, the information processing system 50 can perform the following processing using the matching unit 213. The information processing system 50 can compare the camera parameters for the target camera 310 from the camera parameters in the database DB with the camera parameters used by the multiple position estimation units 211 and 212 when estimating the position of the vehicle 100. In this way, the information processing system 50 can confirm whether or not the position of the vehicle 100 was estimated using the same camera parameters in the database DB.

[0047] Furthermore, according to the above embodiment, the information processing system 50 can store the latest camera parameters corresponding to the changed position and orientation in the database DB. In this way, the information processing system 50 can compare the camera parameters in the database DB with the camera parameters used by the multiple position estimation units 211 and 212 when estimating the position of the vehicle 100, and perform the following: In this case, the information processing system 50 can confirm whether or not the position of the vehicle 100 has been estimated using the latest camera parameters.

[0048] Furthermore, according to the above embodiment, the information processing system 50 can automatically update the camera parameters used by the multiple position estimation units 211, 212 when estimating the position of the vehicle 100 by referencing the database DB to acquire camera parameters. However, for some reason, at least one of the multiple position estimation units 211, 212 may erroneously acquire camera parameters for a camera 310 other than the target camera 310. In this case, the erroneous camera parameters other than those of the target camera 310 may be used when estimating the position of the vehicle 100, which may result in a decrease in the accuracy of estimating the position of the vehicle 100. In contrast, according to the above embodiment, when the camera parameters for the target camera 310 in the database DB do not match the camera parameters used by at least one of the multiple position estimation units 211, 212 when estimating the position of the vehicle 100, the information processing system 50 can do the following. In this case, the information processing system 50 can execute a matching process using the parameter control unit 214. In this way, when there is a risk that the accuracy of estimating the position of the vehicle 100 may decrease because at least one of the multiple position estimation units 211, 212 uses camera parameters other than those of the target camera 310 or uses old camera parameters, the information processing system 50 can match the camera parameters used by the multiple position estimation units 211, 212. Furthermore, the information processing system 50 can execute notification processing and stop processing. Therefore, the information processing system 50 can further suppress a decrease in the accuracy of estimating the position of the vehicle 100.

[0049] Furthermore, according to the above embodiment, the information processing system 50 can perform the following processing using the matching unit 213. The information processing system 50 can identify the camera parameters used by the multiple position estimation units 211 and 212 when estimating the position of the vehicle 100, based on the correspondence between the local coordinates LP of predetermined feature points FP1 to FP3 in the captured image CI and the global coordinates GP corresponding to the feature points FP1 to FP3. In this way, the information processing system 50 can identify the camera parameters used by the multiple position estimation units 211 and 212 when estimating the position of the vehicle 100 each time a match is made, by utilizing pattern imaging. As a result, the information processing system 50 can improve the accuracy of matching multiple camera parameters.

[0050] Furthermore, according to the above embodiment, the information processing system 50 can perform the following processing using the matching unit 213. Based on the acquisition history when camera parameters are obtained by referring to the database DB, the information processing system 50 can identify the camera parameters used by the multiple position estimation units 211 and 212 when estimating the position of the vehicle 100. In this way, the information processing system 50 can easily identify the camera parameters used by the multiple position estimation units 211 and 212 when estimating the position of the vehicle 100, according to the acquisition history, without having to identify them each time a match is made. As a result, the information processing system 50 can reduce the processing load required to identify the camera parameters.

[0051] Furthermore, each of the multiple position estimation units 211, 212 may estimate the position of the vehicle 100 using a method different from that described above. In addition, the information processing system 50 may include three or more position estimation units 211, 212.

[0052] B. Second embodiment: Figure 7 is an explanatory diagram showing the schematic configuration of the information processing system 50v in the second embodiment. In this embodiment, the information processing system 50v differs from the first embodiment in that it does not have a server 200. Also, in this embodiment, the vehicle 100v can be driven by autonomous control of the vehicle 100v. The other configurations are the same as in the first embodiment unless otherwise specified.

[0053] In this embodiment, the processor 111v of the vehicle control device 110v functions as a vehicle control unit 115v, a first position estimation unit 116, a second position estimation unit 117, a matching unit 118, and a parameter control unit 119 by executing a program PG1 stored in memory 112v. The two position estimation units 116 and 117 each estimate the position of vehicle 100v using camera parameters and different logic for one captured image CI. The matching unit 118 matches multiple camera parameters. The parameter control unit 119 performs at least one of the following processes: matching process, notification process, and stop process, in at least one of the first and second cases. The vehicle control unit 115v obtains the position estimation results of vehicle 100v by the position estimation units 116 and 117 and generates a driving control signal using the position estimation results of vehicle 100v. The vehicle control unit 115v then outputs the generated driving control signal to operate the actuator group 120, thereby causing the vehicle 100v to drive under autonomous control. In this embodiment, the memory 112v pre-stores a detection model DM and a reference route RR in addition to a program PG1 including a plurality of applications AP.

[0054] FIG. 8 is a flowchart showing the processing procedure for controlling the traveling of the vehicle 100v in the second embodiment. In step S901, the processor 111v of the vehicle control device 110v acquires vehicle position information using the detection results output from the camera 310, which is the external sensor 300. In step S902, the processor 111v determines a target position to which the vehicle 100v should next head. In step S903, the processor 111v generates a traveling control signal for causing the vehicle 100v to travel toward the determined target position. In step S904, the processor 111v controls the actuator group 120 using the generated traveling control signal, thereby causing the vehicle 100v to travel in accordance with the parameters represented in the traveling control signal. The processor 111v repeatedly acquires vehicle position information, determines the target position, generates the traveling control signal, and controls the actuators at a predetermined cycle. According to the information processing system 50v in this embodiment, the vehicle 100v can be caused to travel by autonomous control of the vehicle 100v without remotely controlling the vehicle 100v using the server 200.

[0055] C. Other Embodiments: (C1) The information processing systems 50, 50v may have only one camera 310 instead of multiple cameras 310. Even in this configuration, a decrease in the accuracy of estimating the position of the vehicle 100 can be suppressed.

[0056] (C2) The information processing system 50, 50v may further include an external sensor 300 other than a camera. In this case, the external sensor 300 may be, for example, a distance measuring device. The distance measuring device may be, for example, a LiDAR (Light Detection and Ranging). In this case, the detection result output by the external sensor 300 may be three-dimensional point cloud data representing the vehicle 100, 100v. In this case, the server 200 or the vehicle 100 may acquire vehicle position information by template matching using the three-dimensional point cloud data as the detection result and reference point cloud data prepared in advance.

[0057] (C3) In the first embodiment described above, the server 200 performs the processing from acquiring vehicle position information to generating a driving control signal. In contrast, the vehicle 100 may perform at least a part of the processing from acquiring vehicle position information to generating a driving control signal. For example, the following forms (1) to (3) may also be used.

[0058] (1) The server 200 may acquire vehicle position information, determine a target position to which the vehicle 100 should next head, and generate a route from the current location of the vehicle 100 indicated in the acquired vehicle position information to the target position. The server 200 may generate a route to the target position between the current location and the destination, or may generate a route to the destination. The server 200 may transmit the generated route to the vehicle 100. The vehicle 100 may generate a driving control signal so that the vehicle 100 drives on the route received from the server 200, and control the actuator group 120 using the generated driving control signal.

[0059] (2) Server 200 may acquire vehicle position information and transmit the acquired vehicle position information to vehicle 100. Vehicle 100 may determine a target position to which vehicle 100 should next head, generate a route from the current location of vehicle 100 indicated in the received vehicle position information to the target position, generate a driving control signal so that vehicle 100 travels on the generated route, and control actuator group 120 using the generated driving control signal.

[0060] (3) In the embodiments of (1) and (2) above, the vehicle 100 is equipped with internal sensors, and the detection results output from the internal sensors may be used in at least one of the generation of a route and the generation of a driving control signal. The internal sensors are sensors mounted on the vehicle 100. The internal sensors may include, for example, sensors that detect the motion state of the vehicle 100, sensors that detect the operating state of each part of the vehicle 100, and sensors that detect the environment around the vehicle 100. Specifically, the internal sensors may include, for example, a camera 310, LiDAR, millimeter-wave radar, ultrasonic sensors, GPS sensors, acceleration sensors, gyro sensors, etc. For example, in the embodiment of (1) above, the server 200 may acquire the detection results of the internal sensors and reflect the detection results of the internal sensors in the route when generating a route. In the embodiment of (1) above, the vehicle 100 may acquire the detection results of the internal sensors and reflect the detection results of the internal sensors in the driving control signal when generating a driving control signal. In the embodiment of (2) above, the vehicle 100 may acquire the detection results of the internal sensors and reflect the detection results of the internal sensors in the route when generating a route. In the above embodiment (2), the vehicle 100 may acquire the detection result of the internal sensor, and when generating the driving control signal, may reflect the detection result of the internal sensor in the driving control signal.

[0061] (C4) In the second embodiment described above, the vehicle 100v is equipped with an internal sensor, and the detection result output from the internal sensor may be used in at least one of the generation of the route and the generation of the driving control signal. For example, the vehicle 100v may acquire the detection result from the internal sensor and reflect the detection result from the internal sensor in the route when generating the route. The vehicle 100v may acquire the detection result from the internal sensor and reflect the detection result from the internal sensor in the driving control signal when generating the driving control signal.

[0062] (C5) In the second embodiment described above, the vehicle 100v acquires vehicle position information using the detection results of the external sensor 300. In contrast, the vehicle 100v may be equipped with an internal sensor, which may acquire vehicle position information using the detection results of the internal sensor, determine the next target location to which the vehicle 100v should go, generate a route from the vehicle 100v's current location to the target location as shown in the acquired vehicle position information, generate a driving control signal for driving along the generated route, and control the actuator group 120 using the generated driving control signal. In this case, the vehicle 100v can drive without using the detection results of the external sensor 300 at all. The vehicle 100v may also acquire the target arrival time and congestion information from outside the vehicle 100v and reflect the target arrival time and congestion information in at least one of the route and the driving control signal.

[0063] (C6) In the first embodiment described above, the server 200 automatically generates a driving control signal to be transmitted to the vehicle 100. Alternatively, the server 200 may generate a driving control signal to be transmitted to the vehicle 100 in accordance with the operation of an external operator located outside the vehicle 100. For example, the external operator may operate a control device that includes a display for displaying captured images CI output from the external sensor 300, a steering wheel for remotely controlling the vehicle 100, an accelerator pedal, a brake pedal, and a communication device for communicating with the server 200 via wired or wireless communication, and the server 200 may generate a driving control signal in accordance with the operation applied to the control device.

[0064] (C7) In each of the above embodiments, the vehicle 100, 100v may have a configuration that allows it to travel by unmanned driving, and may be in the form of a platform having the configuration described below, for example. Specifically, the vehicle 100, 100v may have at least a vehicle control device 110, 110v and an actuator group 120 to perform the three functions of "running," "turning," and "stopping" by unmanned driving. When the vehicle 100, 100v acquires information from the outside for unmanned driving, the vehicle 100, 100v may further have a communication device 130. In other words, the vehicle 100, 100v that can travel by unmanned driving may not be equipped with at least some interior parts such as a driver's seat or a dashboard, may not be equipped with at least some exterior parts such as a bumper or a fender, and may not be equipped with a body shell. In this case, the remaining components, such as the body shell, may be attached to the vehicle 100, 100v before the vehicle 100, 100v is shipped from the factory FC, or the remaining components, such as the body shell, may be attached to the vehicle 100, 100v after the vehicle 100, 100v is shipped from the factory FC without the remaining components, such as the body shell, being attached to the vehicle 100, 100v. Each component may be attached from any direction, such as the top, bottom, front, rear, right side, or left side of the vehicle 100, 100v, and may be attached from the same direction or from different directions. Note that the position of the platform configuration may also be determined in the same manner as for the vehicle 100, 100v in the first embodiment.

[0065] (C8) Vehicles 100, 100v may be manufactured by combining multiple modules. A module means a unit composed of one or more parts grouped together according to the configuration and function of the vehicle 100, 100v. For example, the platform of vehicle 100, 100v may be manufactured by combining a front module that constitutes the front part of the platform, a central module that constitutes the central part of the platform, and a rear module that constitutes the rear part of the platform. The number of modules that constitute the platform is not limited to three, and may be two or fewer, or four or more. In addition to the platform, or in place of the platform, parts of vehicle 100, 100v that are different from the platform may be modularized. Furthermore, various modules may include any exterior parts such as bumpers and grilles, or any interior parts such as seats and consoles. Furthermore, not limited to vehicles 100, 100v, any type of mobile body may be manufactured by combining multiple modules. Such modules may be manufactured, for example, by joining multiple parts by welding or fasteners, or by integrally molding at least a part of the module as a single part by casting. A molding technique that integrally molds at least a portion of a module as a single component is also called Gigacast or Megacast. By using Gigacast, parts of a mobile body that were conventionally formed by joining multiple components can be formed as single components. For example, the forward module, central module, and rear module mentioned above may be manufactured using Gigacast.

[0066] (C9) Transporting the 100,100v vehicles using unmanned operation is also called "autonomous transport." The configuration for realizing autonomous transport is also called a "vehicle remote control autonomous driving transport system." Furthermore, a production method that uses autonomous transport to produce the 100,100v vehicles is also called "autonomous production." In autonomous production, for example, in a factory cluster (FC) that manufactures the 100,100v vehicles, at least a portion of the transport of the 100,100v vehicles is realized by autonomous transport.

[0067] The present disclosure is not limited to the above-described embodiments and can be realized in various configurations without departing from the spirit thereof. For example, the technical features of the embodiments corresponding to the technical features in each aspect described in the Summary of the Invention section can be appropriately replaced or combined to solve some or all of the above-described problems or achieve some or all of the above-described effects. Furthermore, if a technical feature is not described as essential in this specification, it can be appropriately deleted. [Explanation of symbols]

[0068] 50, 50v...information processing system, 100, 100v...vehicle, 101...positioning point, 110, 110v...vehicle control device, 111, 111v...vehicle control device processor, 112, 112v...vehicle control device memory, 113...vehicle control device input / output interface, 114...vehicle control device internal bus, 115, 115v...vehicle control unit, 116, 211...first position estimation unit, 117, 212...second position estimation unit, 118, 213...collation unit, 119, 214...parameter control unit, 120...actuator group, 130...vehicle communication device, 200...server, 201...server processor, 202...server memory, 203...server input / output interface, 204...server Internal bus of the server, 205...communication device of the server, 215...remote control unit, 300...external sensor, 310...camera, AP...application, AP1...first application, AP2...second application, CC...camera coordinate system, CI...captured image, DB...database, DM...detection model, FC...factory, FP1 to FP3...feature points, GC...global coordinate system, GP...global coordinate, IP...image coordinate, LC...local coordinate system, LP...local coordinate, MI...mask image, MS...mask area, PG1, PG2...program, PI...bird's-eye view image, PL1...first location, PL2...second location, RC...circumscribed rectangle, RR...reference route, RS...road surface, TR...track, VC1 to VC4...vertices

Claims

1. An information processing system, A mobile body that can move by unmanned operation; a camera that captures an image of the moving body from outside the moving body and outputs the captured image; Multiple position estimation units that estimate the position of the moving object using different logic for a single captured image, using camera parameters determined according to the position and orientation of the camera, which indicate the correspondence between local coordinates and global coordinates, An information processing system comprising: a parameter control unit that performs a matching process to match the camera parameters used by each of the multiple position estimation units when estimating the position of the moving object.

2. The information processing system according to claim 1, further comprising: a memory for storing a database including the camera parameters; The matching process is an information processing system in which instruction information is sent to each of the multiple position estimation units to use, from among the camera parameters in the database, the camera parameters for the camera that outputs the captured image which each of the multiple position estimation units uses when estimating the position of the moving object.

3. The information processing system according to claim 1, further comprising: a comparison unit that compares the camera parameters used when each of the plurality of position estimation units estimated the position of the moving object with each other; An information processing system in which, when the camera parameters used by each of the multiple position estimation units to estimate the position of the moving body do not match, the parameter control unit performs at least one of the following processes: matching process, notification process to notify the user of error information regarding the camera parameters, and stop process to stop the moving body.

4. The information processing system according to claim 1, further comprising: a memory for storing a database containing the camera parameters; The system includes a comparison unit that compares the camera parameters in the database with the camera parameters for which the captured image was output, which were used by each of the multiple position estimation units when estimating the position of the moving object, and the camera parameters used by each of the multiple position estimation units when estimating the position of the moving object. An information processing system in which, when the camera parameters for the target camera among the camera parameters in the database do not match the camera parameters used by at least some of the multiple position estimation units when estimating the position of the moving body, the parameter control unit executes at least one of the matching process, a notification process that notifies a user of error information regarding the camera parameters, and a stop process that stops the moving body.

5. 5. The information processing system according to claim 3, The matching unit identifies the camera parameters used by the plurality of position estimation units when estimating the position of the moving body from a correspondence between local coordinates of predetermined feature points in the captured image used by the plurality of position estimation units when estimating the position of the moving body and global coordinates corresponding to the feature points.

Images