Route control device and route control method

Abstract

To remotely perform route control for a vehicle with a simpler structure.SOLUTION: A route control device 1 that controls a route of a vehicle 2 from a position in a communication area A1 as an initial point to a position in a communication area An as a destination point in a movement space of the vehicle 2 defined by a plurality of communication areas A1 to An, includes: a first acquisition unit 10 that acquires a position in the communication area A associated with a position registration signal from the vehicle 2, transmitted when it crosses the communication area A, as a current position of the vehicle 2; a determination unit 15 that determines a route to be traveled next by the vehicle 2 from the current position in the communication area A, acquired as the current position of the vehicle 2 by the first acquisition unit 10 on the basis of a strategy of the route to be sequentially travelled by the vehicle 2 from the position of the communication areas A, learned using a learning model; and a route control unit 16 that instructs the route to be traveled next by the vehicle 2, determined by the determination unit 15, through a core network 3 having a predetermined communication standard.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a route control device and a route control method. [Background technology]

[0002] Conventionally, known vehicle position estimation technologies for autonomous driving include signals from positioning satellites such as GPS, a six-axis inertial sensor (Inertial Measurement Unit: IMU) that detects vehicle behavior to complement the signals from positioning satellites, and a Distance Measuring Instrument (DMI) that measures the distance traveled by the vehicle by measuring the number of tire rotations.

[0003] In poor communication conditions, such as when a vehicle is driving in a tunnel, it may not be possible to receive GPS signals from positioning satellites. However, even with IMU and DMI, which are used to complement GPS, the vehicle's position estimation accuracy may not be sufficient depending on the situation. For example, vehicle position estimation using IMU has the disadvantage that errors tend to accumulate. When vehicle position estimation using DMI is used, the measurement accuracy may decrease when the vehicle speed or direction changes.

[0004] As described above, with conventional vehicle position estimation in autonomous driving, when reception of radio waves from GPS positioning satellites was poor, the accuracy of vehicle position estimation decreased, making it difficult for the vehicle's autonomous driving ECU to accurately select a course or route.

[0005] Therefore, Patent Document 1 discloses a technology in which, when a communication delay occurs in controlling autonomous driving by combining GPS and map information, information on the area where the vehicle is located is collected to create current situation map information, and a predictive calculation is performed of the vehicle's driving area to the destination based on the set destination and the current situation map information.

[0006] However, the technology disclosed in Patent Document 1 collects various information such as the vehicle speed and surrounding information, generates a map, and performs predictive calculations of the vehicle's driving area, which makes the predictive calculations complicated and results in a large calculation load. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] JP 2021-111329 A Summary of the Invention [Problem to be solved by the invention]

[0008] As described above, in the conventional technology, it is not possible to remotely control the course of a vehicle using a simpler configuration.

[0009] The present invention has been made to solve the above-mentioned problems, and has an object to remotely control the route of a vehicle using a simpler configuration. [Means for solving the problem]

[0010] In order to solve the above-mentioned problems, the route control device of the present invention is a route control device that controls the route of a vehicle from the position of an initial communication area to the position of a destination communication area in a vehicle movement space defined by a plurality of communication areas, and is equipped with a first acquisition unit configured to acquire the position of the communication area associated with a position registration signal from the vehicle transmitted when crossing a communication area as the current position of the vehicle, a determination unit configured to determine the next route to be taken by the vehicle from the position of the current communication area acquired by the first acquisition unit as the current position of the vehicle based on a route plan for the vehicle to take sequentially from the position of each communication area learned using a learning model, and a route control unit configured to instruct the vehicle of the next route to be taken by the vehicle determined by the determination unit via a core network of a specified communication standard.

[0011] Furthermore, in the route control device of the present invention, a learning unit is configured to apply a reward function to an estimated result of calculating a route that the vehicle should take sequentially from the position of each communication area until the vehicle reaches the position of the communication area of ​​the destination from the position of the communication area of ​​the initial point, and update the route so as to maximize a reward for the vehicle to reach the position of the communication area of ​​the destination, using the learning model, and a memory unit is configured to store the route plan learned by the learning unit, and the determination unit may read out the route plan from the memory unit and determine the next route that the vehicle should take.

[0012] In order to solve the above-mentioned problems, the route control device of the present invention may further include a second acquisition unit configured to acquire the position of a communication area corresponding to an area where congestion occurs in the movement space, and the reward function may include, as variables, a degree of arrival of the vehicle to the position of the communication area related to the destination point, and a degree of arrival of the vehicle to the position of the communication area corresponding to the area where congestion occurs.

[0013] In addition, in the route control device of the present invention, the learning model is a neural network model including an input layer, a hidden layer, and an output layer, and the learning unit provides the position of the current communication area as an input to the neural network model, performs calculations on the neural network model, and outputs a first estimate of an action value function that represents an expected value of the future cumulative value of the reward to be obtained when the vehicle takes each of the actions including a right turn, a left turn, and going straight as a next route to be taken from the position of the current communication area, the learning unit further provides the position of the communication area to be next reached by the vehicle as an input to the neural network model, performs calculations on the neural network model, and outputs a second estimate of the action value function, the learning unit learns weight parameters of the neural network model so that the first estimate becomes a target value calculated from the second estimate, and the memory unit stores the learned weight parameters.

[0014] In addition, in the route control device of the present invention, the first acquisition unit may acquire the location of the current communication area of ​​the vehicle from an integrated data repository that manages subscriber information and is included in the core network, and the route control unit may transmit instructions regarding the next route to the vehicle via a user plane function included in the core network.

[0015] In order to solve the above-mentioned problems, the route control method of the present invention is a route control method for controlling the route of a vehicle from the position of a communication area at an initial point to the position of a communication area at a destination point in a vehicle movement space defined by a plurality of communication areas, and includes a first acquisition step of acquiring the position of the communication area associated with a position registration signal from the vehicle transmitted when crossing a communication area as the current position of the vehicle, a determination step of determining the next route to be taken by the vehicle from the position of the current communication area acquired as the current position of the vehicle in the first acquisition step based on a route plan for the vehicle to take sequentially from the position of each communication area learned using a learning model, and a route control step of instructing the vehicle on the next route to be taken by the vehicle determined in the determination step via a core network of a predetermined communication standard.

[0016] In addition, the route control method of the present invention may further include a learning step configured to apply a reward function to an estimation result obtained by calculating a route that the vehicle should take sequentially from the positions of each communication area until the vehicle reaches the position of the communication area of ​​the destination from the position of the communication area of ​​the initial point, and update the route so as to maximize a reward for the vehicle to reach the position of the communication area of ​​the destination, using the learning model, and a storage step to store the route strategy learned in the learning step in a storage unit, and the determination step may include reading out the route strategy from the storage unit and determining the next route that the vehicle should take.

[0017] In addition, in the route control method of the present invention, a second acquisition step may be further configured to acquire a position of a communication area corresponding to an area where congestion occurs in the movement space, and the reward function may include, as variables, a degree of arrival of the vehicle to a position of the communication area related to the destination point and a degree of arrival of the vehicle to a position of the communication area corresponding to the area where congestion occurs.

[0018] In addition, in the route control method according to the present invention, the learning model may be a neural network model including an input layer, a hidden layer, and an output layer, and the learning step may include providing the position of the current communication area as an input to the neural network model, performing calculations on the neural network model, and outputting a first estimate of an action value function representing an expected value of a cumulative value of the future reward to be obtained when the vehicle takes each of the actions including a right turn, a left turn, and going straight as a next route to be taken from the current communication area, the learning step may further provide the position of a communication area to be next reached by the vehicle as an input to the neural network model, performing calculations on the neural network model, and outputting a second estimate of the action value function, the learning step may learn weight parameters of the neural network model so that the first estimate becomes a target value calculated from the second estimate, and the storage step may store the learned weight parameters in the storage unit.

[0019] In addition, in the route control method of the present invention, the first acquisition step may acquire the location of the current communication area of ​​the vehicle from an integrated data repository that manages subscriber information and is included in the core network, and the route control step may transmit instructions regarding the next route to the vehicle via a user plane function included in the core network. Effect of the Invention

[0020] According to the present invention, the next route to be taken by the vehicle from the current communication area position is determined based on the route plan for the vehicle to take from each communication area position, which is learned using a learning model. Therefore, the route control of the vehicle can be performed remotely with a simpler configuration. [Brief description of the drawings]

[0021] [Figure 1] FIG. 1 is a block diagram showing the configuration of a route control system including a route control device according to an embodiment of the present invention. [Diagram 2]FIG. 2 is a diagram for explaining an overview of the route control system according to the present embodiment. [Diagram 3] FIG. 3 is a diagram for explaining the learning process performed by the learning unit according to the present embodiment. [Figure 4] FIG. 4 is a block diagram showing a configuration of a learning unit according to the present embodiment. [Diagram 5] FIG. 5 is a block diagram showing the hardware configuration of the route control device according to the present embodiment. [Figure 6] FIG. 6 is a flowchart showing the learning process of the route control device according to the present embodiment. [Figure 7] FIG. 7 is a flowchart showing the learning process of the route control device according to the present embodiment. [Figure 8] FIG. 8 is a flowchart showing the route control process of the route control device according to this embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present invention will now be described in detail with reference to FIGS.

[0023] 1 is a block diagram showing the configuration of a route control system including a route control device 1 according to an embodiment of the present invention. The route control system according to this embodiment controls the route of the vehicle 2 from the position of the communication area A1, which is the initial point, to the position of the communication area An, which is the destination point, using a position registration signal sent by the vehicle 2 when the vehicle 2 crosses the communication areas A1 to An in a moving space of the vehicle 2 defined by a plurality of communication areas A1 to An.

[0024] [Route control system configuration] First, an overview of a route control system including a route control device 1 according to an embodiment of the present invention will be described. As shown in Fig. 1, the route control system includes a route control device 1, a vehicle 2, base stations BS1 to BSn, and a core network 3 that are compatible with a 5G wireless communication system using the SA method, for example.

[0025] The base stations BS1 to BSn are composed of wireless base stations compatible with the 5G system, and relay communications between vehicles 2 located in the communication areas A1 to An and the core network 3. In the following, when the base stations BS1 to BSn and the communication areas A1 to An are not distinguished from each other, they may be collectively referred to as the base station BS and the communication area A, respectively.

[0026] As shown in Fig. 1, communication areas A1 to An of base stations BS1 to BSn define a moving space in which vehicle 2 moves. Each of the communication areas A1 to An is arranged to cover a section of a road such as a main road along which vehicle 2 moves from an initial point to a destination point. In this embodiment, it is assumed that communication areas A1 to An have cells of the same size.

[0027] Fig. 2 is a schematic diagram showing a moving space controlled by the route control system. Each dotted circle in Fig. 2 indicates communication areas A1 to A16 covered by each base station BS1 to BS16 arranged in the moving space. The vehicle 2 moves along a route indicated by an arrow from an initial point S to the position of the communication area A16 of the destination point G, using the positions of the communication areas A1 to A16 of the base stations BS1 to BS16 as waypoints.

[0028] In this embodiment, the vehicle 2 turns right, turns left, or goes straight in each of the communication areas A1 to A16 according to the route instructions from the route control system, and reaches the destination point G from the initial point S. The route instructions are assumed to be right or left turns or straight based on the traveling direction of the vehicle 2. For example, if a route instruction to go straight is given at the position of the communication area A1, the vehicle 2 goes straight from the position of the communication area A1 and reaches the next communication area A2. Furthermore, if a route instruction to turn left is given from the position of the communication area A2, the vehicle 2 turns left and reaches the next communication area A6. In this way, the vehicle 2 moves in the movement space according to the route instructions for each of the communication areas A1 to A16.

[0029] A communication terminal 20 is mounted on the vehicle 2. The vehicle 2 includes an automobile, a motor vehicle, a motorcycle, etc. The communication terminal 20 includes a processor, a main memory device, an auxiliary memory device, a communication interface, etc., and is realized as a terminal device mounted on the vehicle 2, or a mobile communication terminal such as a smartphone of a user who uses the vehicle 2, a tablet computer, etc.

[0030] Specifically, the communication terminal 20 includes a SIM 21. The vehicle 2 is uniquely identified by an International Mobile Subscriber Identity (IMSI) of the SIM 21 included in the communication terminal 20.

[0031] When the communication terminal 20 crosses over the communication areas A1 to An as the vehicle 2 moves, the processor of the communication terminal 20 transmits a location registration signal (TAU) to the base station BS1 to BSn of the new communication area A1 to An.

[0032] The vehicle 2 also includes an ECU (Electronic Control Unit) 22, which processes route instructions from the route control device 1 as steering control, drive control, brake control, and automatic driving control of the vehicle 2. The vehicle 2 can include a GPS module having a GPS function (not shown), a car navigation system, and various sensors such as a camera and LiDAR.

[0033] The route control device 1 and the core network 3 are connected via a network NW such as a LAN or a WAN. In addition, the base stations BS1 to BSn constituting a wireless access network and the core network 3 are connected via a network L such as a backhaul link.

[0034] The core network 3 includes an AMF (Access and Mobility Management Function) 30, a UDM (Unified Data Management) 31, and a UDR (Unified Data Repository) 32, which are nodes in the C-plane. The core network 3 also includes a UPF (User Plane Function) 33, which is a node in the U-plane. The route control device 1 acquires location information of the communication area A associated with a location registration signal from the vehicle 2 via a communication interface 32a of the UDR 32. The route control device 1 also transmits instructions for the determined route to the vehicle 2 via a communication interface 33a of the UPF 33.

[0035] For example, as shown in FIG. 1, when vehicle 2 crosses from communication area A1 of base station BS1 to communication area A2 of base station BS2, communication terminal 20 transmits a location registration signal to UDM 31 via base station BS2 and AMF 30 to make a location registration request.

[0036] The AMF 30 transmits the received signal to the UDM 31, and the UDM 31 performs location registration using the terminal identification information of the communication terminal 20 provided in the vehicle 2. Furthermore, the location registration signal and the terminal identification information transmitted by the communication terminal 20 provided in the vehicle 2 are stored in the UDR 32 together with identification information on the serving base station BS and communication area A, and a transmission timestamp (information indicating the date and time) of the location registration signal.

[0037] The route control device 1 according to this embodiment acquires, via the network NW, identification information of each base station BS that has received the location registration signal or identification information of the communication area A, which is associated with the location registration signal stored in the UDR 32. The route control device 1 also stores, as setting information, position information such as GPS coordinates of the longitude and latitude of the base stations BS and communication area A arranged in the moving space in a setting information storage unit 14 described later. In this embodiment, the position of the vehicle 2 is treated as the position of the base station BS of the communication area A in which the vehicle 2 is present. In the following description, the position of the communication area A refers to the position of the corresponding base station BS.

[0038] In this way, by acquiring the position registration signal from the vehicle 2 via the UDR 32, the route control device 1 can acquire the position of the vehicle 2 at the time indicated by the time stamp of the position registration signal.

[0039] [Route control device function block] The route control device 1 includes a first acquisition unit 10, a second acquisition unit 11, a learning unit 12, a learning model storage unit 13, a setting information storage unit 14, a determination unit 15, and a route control unit 16. The route control device 1 determines the next route that the vehicle 2 should take based on a position registration signal from the vehicle 2, and instructs the vehicle 2 on the route.

[0040] The first acquisition unit 10 acquires the position of the communication area A associated with a position registration signal from the vehicle 2 transmitted when the vehicle 2 crosses the communication area A as the current position of the vehicle 2. More specifically, the first acquisition unit 10 acquires the position registration signal transmitted by the communication terminal 20 of the vehicle 2 and information associated with the position registration signal from the communication interface 32a of the UDR 32 via the network NW. The information associated with the position registration signal includes the SIM 21 of the communication terminal 20, identification information of the base station BS that received the position registration signal or identification information of the communication area A, and a timestamp of the time when the position registration signal was transmitted.

[0041] The first acquisition unit 10 can acquire the position of the vehicle 2 at the time the position registration signal was transmitted by referring to the location information of the GPS coordinates of the communication areas A1 to An associated with the identification information of the base stations BS1 to BSn or the identification information of the communication areas A1 to An stored in the setting information storage unit 14 described below.

[0042] The second acquisition unit 11 acquires the position of the communication area A corresponding to an area where congestion occurs in the moving space of the vehicle 2. The second acquisition unit 11 can acquire congestion information and position coordinates where traffic regulations occur in the moving space from an external traffic information server (not shown) via the network NW. The second acquisition unit 11 refers to the position information of the communication areas A1 to An arranged in the moving space stored in the setting information storage unit 14, and acquires the positions of the communication areas A1 to An corresponding to the coordinates of the area where congestion occurs.

[0043] The second acquisition unit 11 acquires the positions of the communication areas A7 and A11 corresponding to the area J where congestion occurs, for example, as shown in Fig. 2. The communication areas A7 and A11 related to congestion or traffic regulations are set to be excluded from route selection when performing route control of the vehicle 2.

[0044] The second acquisition unit 11 can acquire the position of the communication area A related to congestion or traffic regulation at the time when the initial point and the destination point of the vehicle 2 are set and route control is started. Alternatively, the second acquisition unit 11 can acquire the position of the communication area A related to congestion or traffic regulation at a fixed period.

[0045] The learning unit 12 applies a reward function to the estimated results of calculating the route that the vehicle 2 should take from each of the communication areas A1 to An until it reaches the position of the communication area An of the destination point from the position of the communication area A1 of the initial point, and updates the route so as to maximize the reward for the vehicle 2 to reach the position of the communication area An of the destination point, and learns the route strategy that the vehicle 2 should take from each of the communication areas A1 to An using the learning model.

[0046] In this embodiment, a case is illustrated in which the vehicle 2 adopts three actions, a right turn, a left turn, and going straight, as a course measure to be taken by the vehicle 2 to proceed sequentially from the position of each of the communication areas A1 to An. However, the vehicle 2 can learn more detailed actions as course measures according to the area covered by the communication area A, the intervals between the communication areas A arranged according to the shape of the road, and the like.

[0047] In this embodiment, the learning unit 12 uses a neural network model including an input layer, a hidden layer, and an output layer as shown in Fig. 3 as a learning model. In addition, as the neural network model, a state s t , and all action values ​​Q(s t ,straight ahead), Q(s t , turn left), Q(s t We adopt Deep Q-Network (DQN), a neural network that outputs the following behavior: turn right, turn right.

[0048] More specifically, the learning unit 12 provides the current position of the communication area A, which indicates the current position of the vehicle 2, as an input to the neural network model, performs calculations on the neural network model, and outputs a first estimated value Q1 of an action value function that represents the expected value of the cumulative value of future rewards to be obtained when the vehicle 2 takes each action, including a right turn, a left turn, and going straight, as the next course to be taken from the current position of the communication area A.

[0049] The reward is given by a reward function r=r(s, a, s') of a state s indicating the current position of the vehicle 2, an action a of the vehicle 2 turning right, turning left, or going straight, and the next position of the vehicle 2, i.e., the next state s'. In this embodiment, the reward function includes, as variables, the degree of arrival of the vehicle 2 to the position of the communication area A related to the destination point, and the degree of arrival of the vehicle 2 to the position of the communication area A corresponding to an area where congestion is occurring. For example, when the vehicle 2 approaches the destination point or reaches the destination point in the shortest distance by the action of turning right or left or going straight, the reward, which is a scalar quantity, is set to a larger value.

[0050] On the other hand, a negative reward value (e.g., r=-1) can be given when the vehicle 2 moves away from the destination point or reaches the communication areas A7 and A11 related to congestion or traffic restrictions as shown in Fig. 2. In this way, by setting the reward for the communication area A related to congestion or traffic restrictions as a negative value, the vehicle 2 can reach the destination point while avoiding these points.

[0051] Furthermore, the learning unit 12 provides the next position in the communication area A that the vehicle 2 reaches as an input to the neural network model, performs calculations on the neural network model, and outputs a second estimated value Q2 of the action-value function. The learning unit 12 learns weight parameters of the neural network model so that the first estimated value Q1 becomes a target value calculated from the second estimated value Q2.

[0052] If the weight parameter of the neural network model is θ and the action value function is Q(s, a; θ), the minimization loss function for learning is given by the following equation (1). L(θ)=1 / 2{r+γmax a Q(s',a';θ)-Q(s,a;θ)} 2 (1)

[0053] In the above formula (1), r is the reward (immediate reward) and γ is the discount rate. Q(s, a; θ) corresponds to the first estimated value Q1, and Q(s', a'; θ) corresponds to the action value at the state s' one step forward, that is, the second estimated value Q2. The target value is r+γmax a’ It is expressed as Q(s',a';θ).

[0054] The learning unit 12 can update the weight parameters of the neural network model by backpropagating the gradient of the loss function given by the above formula (1).

[0055] More specifically, the learning unit 12 may employ a Fixed Target Q-Network that uses two neural networks, a main QN 121 and a target QN 123, as shown in FIG. 4. The main QN 121 selects an optimal action and updates an action-value function Q. Meanwhile, the target QN 123 estimates and evaluates the value of an action a' to be taken in the next state s' of the result of the action. The main QN 121 and the target QN 123 have neural networks with the same layer structure, but the parameter of the main QN 121 is "θ" and the parameter of the target QN 123 is "θ". - " is given by

[0056] The main QN 121 receives the current position of the vehicle 2 as a state s from the environment 120. The environment 120 is a system of a moving space in which the vehicle 2 is placed, and in this environment 120, the vehicle 2 moves to another communication area A by taking an action a of turning right or left and going straight, and at the same time transitions to the next state s', and acquires a reward r from the environment 120.

[0057] The learning unit 12 inputs the state s relating to the current position of the vehicle 2 to the main QN 121, and obtains the action value function Q(s, a; θ). The learning unit 12 calculates the action a using, for example, the ε-greedy method, or calculates the optimal action argmax a Q(s, a; θ) is calculated. In the environment 120, the vehicle 2 takes the optimal right / left turn / straight forward action argmax a Q(s, a; θ). The environment 120 is a space where vehicle 2 takes the action argmax a As a result of performing Q(s, a; θ), the position of the communication area A to which the robot has moved is observed as the next state s', and a reward r is output. The experience data 124 stores the experience (s, a, r, s') output from the environment 120.

[0058] The learning unit 12 calculates a loss function L in a DQN loss calculation 122, and updates the weight of the main QN 121 with the gradient of the loss function L.

[0059] The learning unit 12 periodically copies the weights of the main QN 121 to the target QN 123 to synchronize them. The synchronization of the target QN 123 is performed less frequently than the update frequency of the weights of the main QN 121. The learning unit 12 extracts experience from the experience data 124, inputs past states to the target QN 123, and calculates the estimated value max a’ Q(s', a';θ - The learning unit 12 outputs the estimated value max a’ Q(s', a';θ - ) based on the target value r+γmax a’ Q(s', a';θ - ) is used to learn the weights of the main QN 121 in the DQN loss calculation 122.

[0060] Returning to FIG. 1, the learning model storage unit 13 stores the weights of the trained neural network model.

[0061] The setting information storage unit 14 stores identification information of the vehicle 2 and the communication terminal 20, position information of the moving space in which the route of the vehicle 2 is controlled, and position information of each communication area A arranged in the moving space. The setting information storage unit 14 also stores position information of the initial point and destination point of the vehicle 2 acquired in advance. The setting information storage unit 14 can store the positions of the communication area A corresponding to the positions of the initial point and the destination point. In addition, the setting information storage unit 14 can store map information of the moving space.

[0062] The determination unit 15 determines the next route that the vehicle 2 should take from the current position of the communication area A acquired by the first acquisition unit 10, based on the course plan for the vehicle 2 to take sequentially from the position of each communication area A, learned using the learning model. The determination unit 15 reads out the learned weights stored in the learning model storage unit 13, provides the current position of the communication area A as an input to the trained neural network model, performs calculations on the trained neural network model, and determines the optimal action of turning right or left, or going straight as the next route to take.

[0063] The route control unit 16 instructs the vehicle 2 on the next route determined by the determination unit 15 via the core network 3. Specifically, the route control unit 16 transmits the route instruction to the vehicle 2 via the UPF 33. The route control unit 16 continues to instruct the vehicle 2 on the route until the vehicle 2 reaches the communication area of ​​the destination point.

[0064] [Hardware configuration of route control device] Next, an example of a hardware configuration for realizing the route control device 1 having the above-mentioned functions will be described with reference to FIG.

[0065] As shown in FIG. 5, the route control device 1 can be realized, for example, by a computer including a processor 102, a main memory device 103, a communication interface 104, an auxiliary memory device 105, and an input / output (I / O) device 106 connected via a bus 101, and a program that controls these hardware resources.

[0066] The main memory device 103 stores in advance programs for the processor 102 to perform various controls and calculations. The processor 102 and the main memory device 103 realize the functions of the route control device 1, such as the first acquisition unit 10, the second acquisition unit 11, the learning unit 12, the determination unit 15, and the route control unit 16 shown in FIG.

[0067] The communication interface 104 is an interface circuit for network connection between the route control device 1 and various external electronic devices.

[0068] The auxiliary storage device 105 is composed of a readable / writable storage medium and a drive for reading and writing various information such as programs and data from and to the storage medium. The auxiliary storage device 105 can use a semiconductor memory such as a hard disk or a flash memory as the storage medium.

[0069] The auxiliary storage device 105 has a program storage area for storing the route control program executed by the route control device 1. The auxiliary storage device 105 also has an area for storing a learning program for learning a neural network model. The learning model storage unit 13 and the setting information storage unit 14 described in Fig. 1 are realized by the auxiliary storage device 105. Furthermore, for example, the auxiliary storage device 105 may have a backup area for backing up the above-mentioned data and programs.

[0070] The input / output I / O 106 is an input / output device that inputs signals from an external device and outputs signals to an external device.

[0071] [Route control device operation] Next, the operation of the route control device 1 having the above-mentioned configuration will be described with reference to the flowcharts of FIG. 6 to FIG.

[0072] First, the learning process by the route control device 1 will be described with reference to Fig. 6. First, the route control device 1 acquires the positions of the initial point and destination point of the vehicle 2 (step S1). For example, the route control device 1 can acquire the destination point and the current position of the vehicle 2 inputted to the car navigation system of the vehicle 2.

[0073] Next, the second acquisition unit 11 acquires the position of the communication area corresponding to the area where congestion occurs in the moving space (step S2). For example, the second acquisition unit 11 can acquire congestion information or position information of the area where traffic regulations occur from an external traffic information server, and identify the position of the corresponding communication area. The second acquisition unit 11 may also be configured to acquire the position of the communication area related to congestion or traffic regulations at a fixed period.

[0074] Next, the first acquisition unit 10 acquires the current position of the communication area A in which the vehicle 2 is present as the current position of the vehicle 2 (step S3). Specifically, the first acquisition unit 10 acquires, from the UDR 32 of the core network 3, the identification information of the communication area A or the base station BS associated with the position registration signal transmitted when the vehicle 2 crossed the communication area, and a time stamp. The first acquisition unit 10 acquires, as the current position of the vehicle 2, the position information of the communication area A associated with the identification information of the communication area A or the base station BS stored in the setting information storage unit 14.

[0075] Next, the learning unit 12 provides the current state of the vehicle 2 acquired in step S3, i.e., the position of the current communication area A in which the vehicle 2 is located, as an input to the neural network model, performs calculations on the neural network model, and outputs a first estimate Q1 of an action value function that represents the expected value of the cumulative value of future rewards to be obtained when the vehicle 2 takes each action, including a right turn, a left turn, and going straight, as the next course to be taken from the current communication area A (step S4).

[0076] Furthermore, the learning unit 12 provides the position of the communication area An that the vehicle 2 next arrives at as an input to the neural network model, performs calculations on the neural network model, and outputs a second estimated value Q2 of the action value function (step S6). The learning unit 12 calculates a target value from the second estimated value Q2 (step S7). Next, the learning unit 12 learns the weight parameters of the neural network model so that the first estimated value Q1 becomes the target value calculated from the second estimated value Q2 (step S8). Specifically, the learning unit 12 updates the weight parameters of the neural network model so as to minimize the loss function of the above formula (1).

[0077] The learning model storage unit 13 stores the learned weights obtained in step S8 (step S9).

[0078] Next, the learning process by the learning unit 12 in the case where a Fixed Target Q-Network using two neural networks, a main QN 121 and a target QN 123, is adopted will be described with reference to FIG.

[0079] The processes from step S1 to step S3 are the same as the steps of the learning process described in Fig. 6. After that, the learning unit 12 provides the position of the communication area A in which the vehicle 2 is present, acquired in step S3, as an input to the main QN 121, performs neural network calculations, outputs an action value function Q, and calculates a course a to be taken next (step S120).

[0080] Next, the learning unit 12 returns the behavior of the vehicle 2 on the path a obtained in step S120 to the environment 120, and obtains the next state s' of the vehicle 2, that is, the position of the communication area A to which the vehicle 2 has traveled, and a reward r (step S121). Note that the reward r given by the reward function can be configured to constantly reflect the degree of arrival at the position of the communication area A where congestion is occurring, which is acquired at regular intervals in step S2.

[0081] The learning unit 12 stores the experience (s, a, r, a') obtained in step S121 in the experience data 124 (step S122). Next, the learning unit 12 calculates a loss function L in the DQN loss calculation 122, and updates the weight of the main QN 121 with the gradient of the loss function L (step S123). The learning unit 12 repeats the processes from step S120 to step S123 a set number of times.

[0082] Thereafter, the learning unit 12 periodically copies the weights of the main QN 121 to the target QN 123 for synchronization (step S124). The synchronization of the target QN 123 is performed less frequently than the update frequency of the weights of the main QN 121. Next, the learning unit 12 extracts experience from the experience data 124, inputs past states to the target QN 123, and calculates the estimated value max a’ Q(s', a';θ - ) is output (step S126).

[0083] Next, the learning unit 12 calculates the estimated value max a’ Q(s', a';θ - ) based on the target value r+γmax a’ Q(s', a';θ - ) (step S127). Next, the learning unit 12 calculates a loss function L in the DQN loss calculation 122 using the target value calculated in step S127 (step S128). Next, the learning unit 12 learns the weights of the main QN 121 so as to minimize the loss given by the loss function L (step S129). After that, the learned weights are stored in the learning model storage unit 13 (step S9).

[0084] Next, the route control process by the route control device 1 will be described with reference to Fig. 8. First, the determination unit 15 loads the trained neural network model from the learning model storage unit 13 (step S40). In this embodiment, the determination unit 15 loads the trained DQN. Next, the first acquisition unit 10 acquires the position of the communication area A in which the vehicle 2 is present, which is the current position of the vehicle 2 (step S41).

[0085] Next, the determination unit 15 determines the next route that the vehicle 2 should take from the position of the current communication area based on the trained neural network model loaded in step S40, i.e., the plan of the route that the vehicle 2 should take sequentially from the position of each communication area A (step S42). Specifically, the determination unit 15 provides the trained neural network model with the current position of the communication area A acquired in step S41 as an input, and performs calculations on the trained neural network model to determine the next route that the vehicle 2 should take, either turning right or left, or going straight. The determination unit 15 selects the action with the highest probability value from the action value functions Q for each of the actions of turning right, turning left, and going straight output from the trained neural network model, and determines the selected action as the route.

[0086] Thereafter, the route control unit 16 instructs the vehicle 2 of the route determined in step S42 via the UPF 33 of the core network 3 to the vehicle 2 (step S43). When the vehicle 2 receives the route instruction, the ECU 22 included in the vehicle 2 outputs a control command to each control actuator according to the route instruction, and the vehicle 2 moves to the next communication area A.

[0087] Next, if the vehicle 2 has reached the destination point, the process ends (step S44: YES). On the other hand, if the vehicle 2 has not reached the destination point (step S44: NO), the process from step S41 to step S43 is repeated. For example, based on a location registration signal transmitted when the vehicle 2 crosses over a communication area A after moving in accordance with a route instruction, it is possible to determine whether the destination point has been reached or not based on whether the location of the associated communication area A matches the location of the communication area A set as the destination point.

[0088] As described above, the route control device 1 according to this embodiment acquires the position of the base station BS associated with the location registration signal transmitted when the vehicle 2 crosses the communication area A, i.e., the position of the communication area A, as the current position of the vehicle 2, and determines the next route to take based on the route strategy acquired by the trained neural network. Therefore, the route control of the vehicle 2 can be performed remotely with a simpler configuration.

[0089] Furthermore, according to the route control device 1 of this embodiment, the location registration signal is used to determine the next route to be taken based on the route strategy acquired by the trained neural network, so that even in situations where radio waves from positioning satellites cannot be received, the vehicle's own position can be estimated and the optimal route can be selected.

[0090] Furthermore, the route control device 1 according to this embodiment employs DQN as a learning model, making it possible to select a route that avoids points related to congestion and traffic regulations.

[0091] In the above embodiment, a route control system that complies with 5G has been exemplified, but the route control system may also be one that complies with LTE or 6G.

[0092] The above describes the embodiments of the route control device and route control method of the present invention, but the present invention is not limited to the described embodiments, and various modifications that can be envisioned by a person skilled in the art can be made within the scope of the invention described in the claims. [Explanation of symbols]

[0093] 1...route control device, 10...first acquisition unit, 11...second acquisition unit, 12...learning unit, 13...learning model memory unit, 14...setting information memory unit, 15...determination unit, 16...route control unit, 2...vehicle, 20...communication terminal, 21...SIM, 22...ECU, 3...core network, 30...AMF, 31...UDM, 32...UDR, 33...UPF, 101...bus, 102...processor, 103...main memory device, 32a, 33a, 104...communication interface, 105...auxiliary memory device, 106...input / output I / O, 120...environment, 121...main QN, 122...DQN loss calculation, 123...target QN, 124...empirical data, BS1 to BSn...base station, A1 to An...communication area, L, NW...network.

Claims

1. A route control device that controls a route of a vehicle from a position of a communication area at an initial point to a position of a communication area at a destination point in a movement space of the vehicle defined by a plurality of communication areas, a first acquisition unit configured to acquire, as a current position of the vehicle, a position of the communication area associated with a position registration signal transmitted from the vehicle when the vehicle crosses a communication area; a determination unit configured to determine a next route to be taken by the vehicle from the current communication area position acquired as the current position of the vehicle by the first acquisition unit, based on a course plan for the vehicle to take from the position of each communication area, learned using a learning model; a route control unit configured to instruct the vehicle on the next route to be taken by the vehicle, the route being determined by the determination unit, via a core network of a predetermined communication standard; A route control device comprising:

2. 2. The route control device according to claim 1, a learning unit configured to apply a reward function to an estimation result of calculating a course that the vehicle should take from the position of each communication area until the vehicle reaches the position of the communication area of ​​the destination point from the position of the communication area of ​​the initial point, and to update the course so as to maximize a reward for the vehicle to reach the position of the communication area of ​​the destination point, and to learn a policy for the course that the vehicle should take from the position of each communication area using the learning model; a storage unit configured to store the course strategies learned by the learning unit; Equipped with The determination unit reads out the course plan from the storage unit and determines the course that the vehicle should take next. A route control device characterized by the above.

3. 3. The route control device according to claim 2, A second acquisition unit configured to acquire a position of a communication area corresponding to an area where congestion occurs in the moving space; The reward function includes, as variables, a degree of arrival of the vehicle to a position of a communication area related to the destination point and a degree of arrival of the vehicle to a position of a communication area corresponding to the area where the congestion occurs. A route control device characterized by the above.

4. 4. The route control device according to claim 3, The learning model is a neural network model including an input layer, a hidden layer, and an output layer; the learning unit provides the position of the current communication area as an input to the neural network model, performs calculations on the neural network model, and outputs a first estimate of an action value function that represents an expected value of a cumulative value of the future reward to be obtained when the vehicle takes each of actions including a right turn, a left turn, and going straight as a next course to be taken from the position of the current communication area; The learning unit further provides a position of the next communication area reached by the vehicle as an input to the neural network model, performs a calculation on the neural network model, and outputs a second estimated value of the action-value function; the learning unit learns weight parameters of the neural network model so that the first estimated value becomes a target value calculated from the second estimated value; The storage unit stores the learned weight parameters. A route control device characterized by the above.

5. 5. The route control device according to claim 1, The first acquisition unit acquires a location of the current communication area of ​​the vehicle from an integrated data repository that is included in the core network and that manages subscriber information; The route control unit transmits instructions regarding the next route to the vehicle via a user plane function included in the core network. A route control device characterized by the above.

6. 1. A route control method for controlling a route of a vehicle from a position of a communication area of ​​an initial point to a position of a communication area of ​​a destination point in a movement space of the vehicle defined by a plurality of communication areas, comprising: a first acquisition step of acquiring, as a current position of the vehicle, a position of the communication area associated with a position registration signal transmitted from the vehicle when the vehicle crosses a communication area; a determination step of determining a next route to be taken by the vehicle from the current communication area position acquired as the current position of the vehicle in the first acquisition step, based on a course plan for the vehicle to take sequentially from the position of each communication area, learned using a learning model; a route control step of instructing the vehicle on the route to be taken next, which has been determined in the determination step, via a core network of a predetermined communication standard; A route control method comprising:

7. 7. The route control method according to claim 6, a learning step configured to apply a reward function to an estimation result obtained by calculating a course that the vehicle should take from the position of each communication area until the vehicle reaches the position of the communication area of ​​the destination point from the position of the communication area of ​​the initial point, and update the course so as to maximize a reward for the vehicle to reach the position of the communication area of ​​the destination point, and to learn a policy for the course that the vehicle should take from the position of each communication area using the learning model; a storage step of storing the course plan learned in the learning step in a storage unit; Equipped with The determining step reads out the course plan from the storage unit and determines the course that the vehicle should take next. A route control method comprising:

8. The route control method according to claim 7, a second acquisition step configured to acquire a position of a communication area corresponding to an area where congestion occurs in the moving space; The reward function includes, as variables, a degree of arrival of the vehicle to a position of a communication area related to the destination point and a degree of arrival of the vehicle to a position of a communication area corresponding to the area where the congestion occurs. A route control method comprising:

9. 9. The route control method according to claim 8, The learning model is a neural network model including an input layer, a hidden layer, and an output layer; The learning step includes providing the position of the current communication area as an input to the neural network model, performing calculations on the neural network model, and outputting a first estimate of an action-value function that represents an expected value of a cumulative value of the future reward that will be obtained when the vehicle takes each of actions including a right turn, a left turn, and going straight as a next course to be taken from the current communication area; The learning step further includes providing a position of a next communication area reached by the vehicle as an input to the neural network model, performing a calculation on the neural network model, and outputting a second estimated value of the action-value function; the learning step includes learning weight parameters of the neural network model so that the first estimated value becomes a target value calculated from the second estimated value; The storage step stores the learned weight parameters in the storage unit. A route control method comprising:

10. A route control method according to any one of claims 6 to 9, The first obtaining step obtains the current communication area location of the vehicle from an integrated data repository that manages subscriber information and is included in the core network; The route control step transmits an instruction regarding the next route to the vehicle via a user plane function included in the core network. A route control method comprising:

Images