Method and apparatus for reinforcement learning-based cargo transportation routing

Abstract

A method for performing reinforcement learning-based cargo transportation routing through a platform executed by a computing device, of embodiments, may comprise the steps of: constructing a dataset on the basis of vehicle owner information, shipper information and environment information; constructing a reinforcement learning-based route setting model from the constructed dataset; acquiring actual generated cargo information, actual environment information and actual vehicle owner information so as to provide same as an input of the reinforcement learning-based route setting model; deriving an initial cargo transportation route for each vehicle owner through the route setting model on the basis of the input; removing at least one piece of cargo information in the derived initial cargo transportation route for each vehicle owner and acquiring a reward and a penalty; and repeating the step of acquiring the reward and the penalty so as to update the route setting model, and deriving an optimal cargo transportation route for each vehicle owner.

Description

Reinforcement learning-based cargo transport routing method and device

[0001] The embodiments relate to a method and device for cargo transport routing based on reinforcement learning. More specifically, the embodiments relate to a method and device for establishing an optimal cargo transport route using reinforcement learning and performing cargo transport routing based on the established route.

[0002] In the past, the primary form of purchasing was door-to-door shopping, where customers would visit the retailer directly to purchase items. However, recent advancements in transportation and storage technology have led to an increase in online shopping, with customers viewing and ordering items online rather than purchasing them in person. Consequently, the volume of packages delivered to homes via courier service has increased exponentially, and the number of trucks is also increasing to accommodate this trend.

[0003] However, the destinations each truck must deliver to may vary, and considering the delivery times, a method may be needed to determine the optimal route for each truck to efficiently deliver the cargo. For example, the current delivery method involves establishing a warehouse for each region to gather cargo, transporting the cargo to each region, and then delivering it to individual homes within the region.

[0004] At this time, multiple cargoes with different routes can be delivered simultaneously on a vehicle. Therefore, it may be efficient for a vehicle to carry cargo that can be unloaded at the same drop-off point or at a drop-off point along the route. However, methods for efficiently setting up transport routes for cargo vehicles are currently lacking. Considering the above, there is a need to implement reinforcement learning-based transport route setting (or routing) for cargo vehicles. A method for this purpose is described below.

[0005]

[0006] [Prior Art Literature]

[0007] [Patent Document]

[0008] Korean Patent No. 10-2086801

[0009] This specification relates to a method and device for performing cargo transport routing based on reinforcement learning.

[0010] The present specification relates to a method and device for setting an initial cargo transport route, removing some points from the initial cargo transport route, and then repeating a route setting operation to derive an optimal cargo transport route.

[0011] The present disclosure relates to a method and apparatus for providing rewards or penalties based on optimization of cargo transportation routes.

[0012] The present specification relates to a method and device for optimizing cargo transportation routes on a platform where multiple shippers and multiple drivers are registered.

[0013] According to one embodiment of the present specification, a method for performing reinforcement learning-based cargo transport routing through a platform executed by a computing device may include the steps of: constructing a data set based on vehicle owner information, shipper information, and environmental information; constructing a reinforcement learning-based route setting model using the constructed data set; obtaining actual generated cargo information, actual environmental information, and actual vehicle owner information and providing the obtained information as inputs to the reinforcement learning-based route setting model; deriving an initial cargo transport route for each vehicle owner through the route setting model based on the inputs; removing at least one piece of cargo information within the derived initial cargo transport route for each vehicle owner and obtaining a reward and a penalty; and repeating the steps of obtaining the reward and penalty to update the route setting model and deriving an optimal cargo transport route for each vehicle owner.

[0014] Additionally, according to one embodiment of the present specification, the vehicle owner information may include vehicle owner information and vehicle owner driver information, the cargo information may include order information and cargo route information, and the environment information may include at least one of traffic information, map information, and hub location information.

[0015] In addition, according to one embodiment of the present specification, the cargo vehicle information may include information on the type of the cargo vehicle, the model year, the size of the cargo compartment, and the type of cargo contained in the cargo vehicle; the driver information may include information on the age, gender, career, and health of the driver; the order information may include information on the departure point, destination, and type of cargo; the cargo route information may include information on the movement route, total carbon emissions, and delivery completion time according to the order information for each driver; the traffic information may include information on road conditions and signaling systems by time; the map information may include a map for the movement route for each driver; and the hub location information may include information on possible locations for loading and unloading cargo.

[0016] In addition, according to one embodiment of the present specification, when a first cargo transportation initial route for a first vehicle is derived through a route setting model, the method may include a step of performing an operation of removing first order information within the first cargo transportation initial route and obtaining compensation information based on the removed first order information, a step of repeating an operation of removing order information different from the first order information within the first cargo transportation initial route and obtaining compensation information based on the removed order information, and a step of updating the first cargo transportation initial route to a first cargo transportation optimal route based on the repeating operation.

[0017] Additionally, according to one embodiment of the present specification, the method may include a step of allocating the removed first order information into the second cargo transport initial route of the second vehicle, a step of obtaining compensation information based on the allocation, and a step of updating the first cargo transport initial route to the first cargo transport optimal route by further considering the compensation information obtained based on the allocation.

[0018] Additionally, according to one embodiment of the present specification, the reward information may include at least one of reward information and penalty information determined based on delivery completion time and carbon emission information.

[0019] Additionally, according to one embodiment of the present specification, the removed first order information may be cargo information unloaded or loaded at a first point within the first cargo transport initial route.

[0020] Additionally, according to one embodiment of the present specification, the removed first order information may be determined based on the next driver information, and the removed first order information may be determined by reflecting the infectious disease outbreak risk information and the health status information of the next driver information as environmental information of the first point.

[0021] The present specification can provide a method for performing cargo transport routing based on reinforcement learning.

[0022] The present specification can provide a method for deriving an optimal cargo transport route by setting an initial cargo transport route, removing some points from the initial cargo transport route, and repeating a route setting operation.

[0023] This specification may provide rewards or penalties based on cargo transport route optimization.

[0024] The present specification may provide a method for optimizing cargo transportation routes on a platform in which multiple shippers and multiple drivers are registered.

[0025] It should be understood that the effects of this specification are not limited to the matters described above, but can be expanded to various contents that can be derived from the detailed description of the embodiments of the invention below.

[0026] FIG. 1 is a diagram illustrating an example of an operating environment of a system according to one embodiment of the present specification.

[0027] FIG. 2 is a block diagram for explaining the internal configuration of a computing device (200) in one embodiment of the present specification.

[0028] Figure 3 is a drawing showing a platform connecting a shipper and a vehicle owner in one embodiment of the present specification.

[0029] FIG. 4 is a diagram illustrating a method for performing cargo transport routing based on artificial intelligence in one embodiment of the present specification.

[0030] FIG. 5a is a diagram illustrating a method for setting a cargo transportation route in one embodiment of the present specification.

[0031] FIG. 5b is a diagram illustrating a method for transmitting set cargo transport route information to each vehicle owner in one embodiment of the present specification.

[0032] FIG. 6 is a diagram illustrating a method for performing reinforcement learning-based cargo transport routing in one embodiment of the present specification.

[0033] FIG. 7 is a diagram illustrating a method for performing reinforcement learning-based cargo transport routing in one embodiment of the present specification.

[0034] FIG. 8 is a diagram illustrating a method for controlling a cargo transport route in one embodiment of the present specification.

[0035] FIG. 9 is a flowchart illustrating a method for performing reinforcement learning-based cargo transport routing in one embodiment of the present specification.

[0036] In describing the embodiments of this specification, if a detailed description of a known configuration or function is judged to obscure the gist of the embodiments of this specification, a detailed description thereof will be omitted. In addition, parts of the drawings that are not related to the description of the embodiments of this specification have been omitted, and similar parts have been designated with similar drawing reference numerals.

[0037] In the embodiments of this specification, when a component is said to be "connected," "coupled," or "connected" to another component, this may include not only a direct connection, but also an indirect connection in which another component exists in between. Furthermore, when a component is said to "include" or "have" another component, unless otherwise specifically stated, this does not exclude the other component, but rather implies that the other component may be included.

[0038] In the embodiments of this specification, the terms first, second, etc. are used only for the purpose of distinguishing one component from another component, and do not limit the order or importance between components unless specifically stated otherwise. Therefore, within the scope of the embodiments of this specification, a first component in an embodiment may be referred to as a second component in another embodiment, and similarly, a second component in an embodiment may be referred to as a first component in another embodiment.

[0039] In the embodiments of this specification, distinct components are used to clearly illustrate their respective characteristics and do not necessarily imply separation. That is, multiple components may be integrated into a single hardware or software unit, or a single component may be distributed into multiple hardware or software units. Therefore, even if not specifically mentioned, such integrated or distributed embodiments are also included within the scope of the embodiments of this specification.

[0040] In this specification, the term "network" may encompass both wired and wireless networks. In this case, the term "network" may refer to a communications network that enables data exchange between devices, systems, and devices, and is not limited to a specific network.

[0041] Embodiments described herein may be entirely hardware, partially hardware and partially software, or entirely software. As used herein, "unit," "device," or "system" refers to a computer-related entity such as hardware, a combination of hardware and software, or software. For example, a unit, module, device, or system as used herein may be, but is not limited to, a running process, a processor, an object, an executable, a thread of execution, a program, and / or a computer. For example, both an application running on a computer and the computer itself may correspond to a unit, module, device, or system as used herein.

[0042] Additionally, in this specification, a device may be a mobile device such as a smartphone, tablet PC, wearable device, or HMD (Head Mounted Display), as well as a fixed device such as a PC or home appliance with display functions. Furthermore, as an example, a device may be an in-vehicle cluster or an IoT (Internet of Things) device. In other words, in this specification, a device may refer to any device capable of operating an application, and is not limited to a specific type. For convenience of explanation, the device on which an application operates is referred to as a device below.

[0043] In this specification, the network communication method is not limited, and connections between components may not be made using the same network method. The network may include not only communication methods utilizing communication networks (e.g., mobile communication networks, wired Internet, wireless Internet, broadcasting networks, satellite networks, etc.), but also short-range wireless communication between devices. For example, the network may include all communication methods that enable objects to network with each other, and is not limited to wired communication, wireless communication, 3G, 4G, 5G, or other methods. For example, wired and / or networks include Local Area Network (LAN), Metropolitan Area Network (MAN), Global System for Mobile Network (GSM), Enhanced Data GSM Environment (EDGE), High Speed ​​Downlink Packet Access (HSDPA), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Zigbee, Wi-Fi, VoIP (Voice over Internet Protocol), LTE Advanced, IEEE802.16m, WirelessMAN-Advanced, HSPA+, 3GPP Long Term Evolution (LTE), Mobile WiMAX (IEEE 802.16e), UMB (formerly EV-DO Rev. C), Flash-OFDM, iBurst and MBWA (IEEE 802.20) It may refer to a communication network using one or more communication methods selected from the group consisting of systems, HIPERMAN, Beam-Division Multiple Access (BDMA), Wi-MAX (World Interoperability for Microwave Access), and ultrasonic communication, but is not limited thereto.

[0044] The components described in various embodiments are not necessarily essential components, and some may be optional. Therefore, embodiments comprising a subset of the components described in the embodiments are also included within the scope of the embodiments of the present disclosure. Furthermore, embodiments including other components in addition to the components described in various embodiments are also included within the scope of the embodiments of the present disclosure.

[0045] Hereinafter, embodiments of the present specification will be described in detail with reference to the drawings.

[0046] FIG. 1 is a diagram illustrating an example of the operating environment of a system according to one embodiment of the present specification. Referring to FIG. 1, one or more user devices (110-1, 110-2) and one or more servers (120, 130, 140) are connected via a network (1). FIG. 1 is merely an example for explaining the invention, and the number of user devices or servers is not limited to that shown in FIG. 1.

[0047] One or more user devices (110-1, 110-2) may be fixed or mobile terminals implemented as a computer system. The one or more user devices (110-1, 110-2) may be, for example, a smart phone, a mobile phone, a navigation device, a computer, a laptop, a digital broadcasting terminal, a PDA (Personal Digital Assistants), a PMP (Portable Multimedia Player), a tablet PC, a game console, a wearable device, an IoT (Internet of Things) device, a VR (Virtual Reality) device, an AR (Augmented Reality) device, etc. For example, in the embodiments, the user device (110) may mean one of various physical computer systems that can communicate with other servers (120 to 140) through a network (1) using a wireless or wired communication method.

[0048] Each server may be implemented as a computer device or multiple computer devices that communicate with one or more user devices (110-1, 110-2) via a network (1) to provide commands, codes, files, contents, services, etc. For example, the server may be a system that provides each service to one or more user devices (110-1, 110-2) connected via the network (1). As a more specific example, the server may provide a service (e.g., provision of information, etc.) intended by an application to one or more user devices (110-1, 110-2) through an application as a computer program that is installed and run on one or more user devices (110-1, 110-2). As another example, the server may distribute a file for installing and running the above-described application to one or more user devices (110-1, 110-2) and receive user input information to provide a corresponding service.

[0049] FIG. 2 is a block diagram illustrating the internal configuration of a computing device (200) according to one embodiment of the present specification. This computing device (200) may be applied to one or more user devices (110-1, 110-2) or servers (120-140) described above with reference to FIG. 1, and each device and server may have the same or similar internal configuration by adding or excluding some components.

[0050] Referring to FIG. 2, a computing device (200) may include a memory (210), a processor (220), a communication module (230), and a transceiver (240). The memory (210) is a non-transitory computer-readable recording medium and may include a non-permanent mass storage device such as a random access memory (RAM), a read only memory (ROM), a disk drive, a solid state drive (SSD), a flash memory, etc. Here, the non-permanent mass storage device such as a ROM, an SSD, a flash memory, a disk drive, etc. may be included in the above-described device or server as a separate permanent storage device distinct from the memory (210). In addition, the memory (210) may store an operating system and at least one program code (for example, a browser installed and operated on a user device (110), or a code for an application installed on a user device (110) to provide a specific service). These software components may be loaded from a computer-readable recording medium separate from the memory (210). This separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, a memory card, etc.

[0051] In another embodiment, the software components may be loaded into the memory (210) via a communication module (230) rather than a computer-readable recording medium. For example, at least one program may be loaded into the memory (210) based on a computer program (e.g., the application described above) that is installed by files provided by developers or a file distribution system (e.g., the server described above) that distributes the installation files of the application via a network (1).

[0052] The processor (220) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to the processor (220) by the memory (210) or the communication module (230). For example, the processor (220) may be configured to execute instructions received according to program code stored in a storage device such as the memory (210).

[0053] The communication module (230) can provide a function for the user device (110) and the server (120 - 140) to communicate with each other via the network (1), and can provide a function for each of the device (110) and / or the server (120 - 140) to communicate with other electronic devices.

[0054] The transceiver (240) may be a means for interfacing with an external input / output device (not shown). For example, external input devices may include devices such as a keyboard, mouse, microphone, camera, etc., and external output devices may include devices such as a display, speaker, haptic feedback device, etc.

[0055] As another example, the transmitter / receiver (240) may be a means for interfacing with a device that integrates input and output functions, such as a touch screen.

[0056] In addition, in other embodiments, the computing device (200) may include more components than the components of FIG. 2 depending on the nature of the device to which it is applied. For example, when the computing device (200) is applied to a user device (110), it may be implemented to include at least some of the above-described input / output devices, or may further include other components such as a transceiver, a Global Positioning System (GPS) module, a camera, various sensors, a database, etc. As a more specific example, when the user device is a smartphone, it may be implemented to further include various components that are generally included in a smartphone, such as an acceleration sensor or a gyro sensor, a camera module, various physical buttons, buttons using a touch panel, input / output ports, and a vibrator for vibration.

[0057] The operations described herein below are described based on a computing device. For example, the computing device may be at least one of the servers and devices described above. In other words, the operations described below may be performed on a server or device and are not limited to a specific device. However, for convenience of explanation, the operations described below are based on a computing device.

[0058] Figure 3 is a drawing showing a platform connecting a shipper and a vehicle owner in one embodiment of the present specification.

[0059] Referring to FIG. 3, a platform (310) connecting a cargo owner (hereinafter, “shipper”) and a vehicle owner (hereinafter, “vehicle owner”) can be provided. For example, the platform (310) can be provided via a network (1) based on the aforementioned FIG. 1. Here, the platform (310) can be provided based on a computing device. The computing device providing the platform (310) can be a computing device implemented as a server (120-140) of the aforementioned FIG. 1. In addition, as an example, the shipper and the vehicle owner can also access the platform (310) based on a computing device. As a specific example, the computing devices of the shipper and the vehicle owner can access the platform (310) based on the network (1) and receive transportation services through the accessed platform (310). That is, the computing devices for the shipper and the vehicle owner can be terminals, and can be any of the devices disclosed in FIG. 1.

[0060] As a specific example, referring to FIG. 3, a shipper's computing device can register its cargo through the platform and request a transportation price. At this time, the shipper's computing device can provide the platform (310) with information regarding at least one of the following: cargo type information, destination (route) information, transportation time information, and other cargo-related information. For example, based on the cargo type information, the transportation price may be determined higher for items that are fragile or easily damaged and require careful handling. As another example, the farther the destination is from the shipper's location, the higher the transportation price may be determined. In yet another example, the transportation price may be determined higher for items with short transportation times, such as urgent delivery. In other words, the transportation price of a cargo can be determined based on various information. At this time, each piece of cargo-related information can be quantified.

[0061] For example, when providing transportation services through a platform (310), there are limitations to qualitatively assessing all information to determine transportation prices. Therefore, the platform (310) can quantify each piece of information based on cargo information. Furthermore, each piece of quantified information can be weighted. For example, information that significantly influences transportation prices can be given a higher weight. Conversely, information that has a smaller impact on transportation prices can be given a lower weight. Based on the aforementioned information, the platform can quantify and calculate transportation prices.

[0062] At this time, for example, the shipper's computing device can register the cargo on the platform (310) based on the above-described information. The platform (310) can check records of similar cargo based on the above-described information. At this time, the platform (310) can determine the similarity between the similar cargo and the registered cargo. If both the shipper and the cargo have the same destination but different destinations, the similarity may be determined to be high. For example, as described above, cargo-related information may be digitized, and the similarity with existing cargo may be determined based on the digitized cargo-related information. For example, cargo with similar values ​​considering the weight and digitized information may be priced similarly. At this time, for example, the platform (310) operating based on the computing device may determine the similarity of the cargo based on a similarity function. When the cargo is registered on the platform (310) based on the above-described method, an initial route may be set for each vehicle owner based on the cargo information. For example, the initial route setting for each vehicle owner may be an initial routing setting. Here, the platform (310) can perform initial route setting based on the response information provided by the driver and the location and type of the driver, and a number of delivery routes can be generated based on this.

[0063] As a specific example, the platform (310) may configure a route setting model for setting the initial route of cargo. The route setting model may be a model built based on artificial intelligence or machine learning, but may not be limited thereto. For example, the route setting model may be a model built by reflecting the aforementioned information about the cargo and characteristic information about each vehicle owner. For example, the platform (310) may build a data set based on vehicle owner information, shipper information, and environmental information, and acquire a route setting model based on the built data. For example, the route setting model may be a reinforcement learning-based route setting model, which will be described later. Thereafter, actual cargo information, actual environmental information, and actual vehicle owner information may be provided as inputs to the acquired route setting model, and an initial cargo transport route for each vehicle owner may be derived as output. Here, as an example, the vehicle owner information may include the vehicle owner's cargo vehicle information and the vehicle owner's driver information, the cargo information may include order information and cargo route information, and the environmental information may include at least one of traffic information, map information, and hub location information.

[0064] In addition, the freight vehicle information may include the type of freight vehicle, year of manufacture, size of cargo compartment, and type of cargo contained in the freight vehicle; the driver information may include the driver's age, gender, career, and health information; the order information may include the departure point, destination, and type of cargo information; the cargo route information may include the travel route according to order information for each driver, total carbon emissions, and delivery completion time information; the traffic information may include road conditions and signal system information by time; the map information may include a map of the travel route for each driver; and the hub location information may include information on possible locations for loading and unloading cargo.

[0065] For example, a route setting model can derive an initial route for each vehicle by considering a combination of multiple cargoes and multiple vehicle owners. The route setting model can operate based on artificial intelligence, and each of the multiple cargo-related information and the multiple vehicle owner-related information can be provided as inputs to the route setting model. Based on the aforementioned inputs, the route setting model can derive an initial route for each of the multiple vehicle owners as an output. For example, the initial route for each of the multiple vehicle owners can include all of the multiple cargoes (or order information) provided as inputs, and the initial route setting (or initial routing) can be performed as described above.

[0066] As another example, FIG. 4 is a diagram illustrating a method for performing cargo transport routing based on artificial intelligence according to one embodiment of the present disclosure. Referring to FIG. 4, the platform (310) can determine the cargo to be mixed for each vehicle owner based on artificial intelligence and set the initial route for cargo transport based on this. For example, the AI ​​model can determine the cargo to be mixed among multiple cargoes based on at least one of the type of cargo assigned to the vehicle owner, the destination, the required arrival time of the cargo (delivery limit time), the cargo transport distance, the traffic weight of the new route, additional time, priority, and vehicle owner information. The platform (310) provides the above-described information as input to the AI ​​model and can determine the cargo to be mixed for each vehicle owner based on this, and set the initial route for cargo transport based on this. For example, the AI ​​model of FIG. 4 may be identical to the route setting model described above. As another example, the AI ​​model and the route setting model of FIG. 4 may each exist and be interrelated. That is, the platform (310) can perform shipper and driver allocation based on the artificial intelligence model of FIG. 4, and the platform (310) can determine the cargo to be mixed by reflecting the registered cargo (or order information) and the driver's location information. Furthermore, the aforementioned information can be reflected in the route setting model to perform initial route setting.

[0067] For example, in FIG. 4, a specific driver can determine whether to mix cargo based on the cargo type. That is, for a specific driver, a specific cargo may be mixable, while another specific cargo may not be mixable. As a specific example, the platform (310) may assign fragile cargo to a specific driver. Here, the platform (310) may not assign cargo that weighs more than a preset value, taking into account the risk of cargo damage for the specific driver. That is, the platform (310) should assign the cargo to a specific driver when considering the minimum time and optimal route, but may not assign the cargo based on the cargo type.

[0068] As another example, the platform (310) may assign a cargo type requiring expedited delivery, such as food delivery, to a single driver. For example, even if another cargo offers the optimal route for that driver, the platform (310) may not assign that cargo, considering that food is assigned and requires expedited delivery. In other words, the platform (310) may assign cargo to a driver based on the cargo type or delivery method.

[0069] As another example, the platform (310) may consider the possibility of mixed loading for each cargo, and may perform transport route selection by giving priority to mixed loading zones where the transport efficiency of the corresponding vehicle is high, or may provide information on mixed loading zones. For example, if the vehicle currently has little cargo and the lowest mixed loading rate, and there is a lot of mixed loading possible cargo in a specific mixed loading zone, the platform (310) may provide a transport route for the vehicle by giving priority to the mixed loading zone. As another example, the platform (310) may give priority to the mixed loading zone based on the above-described method and allocate cargo to the vehicle, thereby increasing transport efficiency.

[0070] As another example, the platform (310) can allocate cargo to drivers and set initial routes based on incentives. Specifically, the platform (310) can provide incentive information to drivers who select routes via areas where mixed loads are likely to occur. The incentive information can include monetary information, insurance information, priority allocation information when mixed load requests occur, and other information. Specifically, the platform (310) can enhance utilization efficiency by providing compensation information when drivers consider additional cargo among various transport routes, consider detours, or increase transport time to transport additional cargo, thereby increasing transport efficiency. For example, the incentive information can be determined by considering at least one of the following: additional travel distance, additional time, and additional energy (electricity / fuel) consumption, which are generated based on areas where mixed loads are likely to occur. For example, the platform (310) can provide incentive information to drivers based on point information and other information, including a portion of the actual additional cost, and may not be limited to a specific form. Here, the platform (310) can perform route setting for each of multiple borrowers by reflecting the above-described incentive information.

[0071] Based on the above, the platform (310) can perform initial route setting for each of the multiple drivers based on the multiple assigned cargoes, thereby establishing a cargo transport route for each of the multiple drivers. Here, the cargo transport route for each of the multiple drivers can include all assigned cargoes (or order information).

[0072] As a specific example, FIG. 5A is a diagram illustrating a method for setting a cargo transportation route in one embodiment of the present specification. FIG. 5A is merely an example for convenience of explanation and may not be limited thereto. Referring to FIG. 5A, the platform (310) may set an initial route for each vehicle owner based on the above-described operation. In FIG. 5A, the platform (310) may set an initial route as a cargo transportation route for a specific vehicle owner (540). For example, the platform (310) may determine a starting location from point A (511) and an arrival location from point B (512). For example, the starting location point A (511) may be determined by reflecting the current location information of the specific vehicle owner (540), but may not be limited thereto. Thereafter, the platform (310) may set a cargo unloading area as a cargo delivery route from point A (511) to point B (512). For example, the cargo transportation route for a specific vehicle owner (540) may be initially set as a route from point A (511) through point 2 (522), point 1 (521), point 4 (524), point 5 (525), point 6 (526), ​​point 9 (529), point 8 (528), and point 7 (527) to point B (512) as the destination. However, this is only one example for the convenience of explanation and may not be limited thereto. Here, the platform (310) may provide the set cargo transportation initial route information (530) to the specific vehicle owner (540), and may be as shown in FIG. 5b. For example, the platform (310) may provide each point information for cargo transportation to the specific vehicle owner (540) as described above. Additionally, as an example, the platform (310) may provide cargo transportation initial route information (530) including cargo loading and unloading information, vehicle ID information, transportation time information, and other information at each point to a specific vehicle owner (540), but is not limited to a specific embodiment.

[0073] Figure 6 is a diagram illustrating a method for performing reinforcement learning-based cargo transport routing according to one embodiment of the present disclosure. Referring to Figure 6, the platform (310) can set an initial cargo transport route for each of multiple vehicle owners and provide corresponding information to each vehicle owner.

[0074] Here, the platform (310) can perform routing configuration by removing some of the points within the established initial cargo transport route. Specifically, the platform (310) can perform routing configuration by removing specific points within a specific initial cargo transport route. Here, the cargo for the removed specific points can be restored from the existing initial cargo transport route to perform a new routing configuration. For example, the platform (310) can provide the cargo for the removed specific points to another vehicle owner. Here, the platform (310) can provide a reward or penalty along with the cargo for the removed specific points, but is not limited thereto.

[0075] For example, the platform (310) can repeat the above-described actions, and the actions can be performed based on reinforcement learning. Reinforcement learning is a type of machine learning that operates in a dynamic environment rather than relying on a static data set and can be a method of performing learning through collected information. Reinforcement learning can be a model that initiates learning behavior on its own without intervention, operating based on appropriate incentives (or rewards) without requiring pre-training data collection, preprocessing, and labeling required in supervised or unsupervised learning. For example, reinforcement learning can determine the environment to which it will be applied and establish an interface between the agent and the environment within the environment. Then, based on the actions of the agent, reward information can be set to suit the environment. Afterwards, it can be a learning method that derives an optimal conclusion based on the actions of each agent within the environment.

[0076] For example, the platform (310) can derive an optimal cargo transport route by repeatedly performing tasks such as removing some paths from the initial cargo transport route and then restoring the existing route based on the aforementioned reinforcement learning. Here, the initial cargo transport route may be the environment described above, and the agent's action may be to remove some points from each initial cargo transport route. Here, cost- or time-based incentive information may be the reward information, and by repeating the aforementioned actions, the platform (310) can derive an optimal route.

[0077] As a specific example, the platform (310) can construct a data set consisting of order information, traffic information, map information, hub location information, truck information, truck driver information, truck route information, and other information, based on each cargo-related information and each truck owner-related information. For example, order information may include the origin, destination, cargo type, and other information. Furthermore, traffic information may include hourly road traffic conditions, signal system information, and other information. Furthermore, hub location information may include information on locations or points where cargo can be loaded or unloaded, as well as other information. Furthermore, truck information may include the truck type, model year, cargo space size, cargo type, and other information. Furthermore, truck driver information may include the driver's age, gender, career, health status, and other information. Furthermore, truck route information may include the movement route based on order information for each truck, total carbon emissions, delivery completion time, and other information, and is not limited to a specific form.

[0078] The platform (310) can build the aforementioned route setting model that outputs cargo route information based on the information contained in the aforementioned data set. Here, the platform (310) provides actual order information for each cargo, cargo delivery-related information such as traffic information, map information, hub location information, and truck information and truck driver information as owner-related information as inputs to the route setting model, and can derive an initial cargo transport route for each owner.

[0079] Here, the platform (310) can remove at least one piece of cargo information (or order information). For example, the at least one piece of order information removed by the platform (310) may be information set for each location or destination. Here, the platform (310) can obtain rewards or penalties based on the delivery completion time, carbon emissions, and other information from the cargo route information collected based on the removed cargo information (or order information). For example, the reward or penalty may be the reward information from the aforementioned reinforcement learning. Thereafter, the platform (310) can assign the removed order information (or corresponding location information) to another driver with adjacent order information. The platform (310) can repeat the above-described operations to derive an optimal cargo transport route.

[0080] As a specific example, the platform (310) may remove at least one piece of cargo information (or order information) from the initial freight transport route for the first vehicle owner. Here, if the route for the first vehicle owner is shortened or carbon emissions are reduced, such as through energy savings, based on the removal of the cargo information, the platform (310) may obtain a reward. Here, the platform (310) may assign the cargo information removed from the initial freight transport route for the first vehicle owner to another vehicle owner (a second vehicle owner) or maintain it as an unassigned cargo. For example, if the cargo is assigned to the second vehicle owner and delivery time is delayed or carbon emissions increase due to increased energy consumption, the platform (310) may obtain a penalty. Furthermore, if the removed cargo information becomes unassigned cargo information and processing of the corresponding cargo is delayed, the platform (310) may obtain a penalty. Here, the platform (310) may repeatedly perform the operations of removing at least one piece of cargo information for each of the multiple vehicles owner and assigning the removed cargo information to another vehicle owner, as described above. The platform (310) can derive an optimal route for each of multiple drivers by considering the rewards and penalties obtained while repeatedly performing the above-described operations.

[0081] As described above, when the platform (310) removes a specific point (or order information) from the route of a first delivery driver to a specific driver, the platform (310) may request the transfer of the removed point (or order information) to a second driver. If the platform (310) receives an approval message for the transfer request from the second driver, the platform (310) may transfer the point (or order information) to the second driver. Here, the platform (310) may provide compensation information to the second driver. For example, the compensation information may be a monetary incentive or the right to view and obtain order information of other delivery drivers, and is not limited to a specific form. As another example, the platform (310) may remove a specific point (or order information) based on information related to a specific driver. For example, the platform (310) may consider removing a specific point (or order information) by considering the health status (e.g., sick leave last week) and the risk of infectious disease outbreak of the specific driver. For example, if the number of COVID-19 patients in a specific area is increasing and a specific driver is in poor health, the platform (310) may assign a different driver. The platform (310) can derive an optimal route by repeating the actions (or simulations) for removing certain points (or order information) based on reinforcement learning, using rewards and penalties based on the aforementioned actions.

[0082] As another example, the platform (310) can determine order information to be removed based on the order cancellation probability. Here, the order cancellation probability can be determined based on the customer's previous order cancellation history, the type of product being ordered, and other historical information about the product being ordered. In other words, the platform (310) can check whether removing order information with a high order cancellation probability improves the efficiency of other deliveries and, based on this, change route information.

[0083] As a specific example, in FIG. 6, the initial freight transport route for a first vehicle owner may be set from a first point (610) to a second point (620). Here, the platform (310) may remove specific points (or order information, 631, 632, 633) based on the above-described method. For example, the platform (310) may provide a reward or penalty to the first vehicle owner based on the removed specific points. Furthermore, the platform (310) may assign the removed specific points (or order information, 631, 632, 633) to a second vehicle owner, and may provide a reward or penalty to the second vehicle owner based on this. For example, the platform (310) may repeat the operation of removing specific points (or order information) as described above based on reinforcement learning, thereby updating the optimal route. For example, in the first path in FIG. 6, existing specific points (or order information, 631, 632, 633) may be removed and a different type of path may be set, but is not limited thereto.

[0084] As a more specific example, referring to FIG. 7, the platform (310) may repeat the operation of removing some points based on the initial cargo transport route set in FIG. 5A. For example, the platform (310) may provide the set initial cargo transport route (A->2->1->4->5->6->9->8->7->B) to a reinforcement learning-based route setting model (710). Here, some points within the initial cargo transport route may be removed, and compensation information may be generated as a reward or penalty based on this. The path setting model (710) can generate reward information consisting of rewards or penalties based on a first path (A->2->1->5->6->9->8->7->B) from which some points have been removed, a second path (A->2->1->4->5->9->8->7->B) from which some other points have been removed, and a third path (A->2->1->4->5->6->9->8->B) from which some other points have been removed. Thereafter, an optimal path (A->2->4->9->B, 720) can be derived based on the operation of the path setting model (710).

[0085] For example, in FIG. 7, points "1", "4", "6", "8", and "7" may be removed from the optimal route of a specific vehicle, and the points may be assigned to other vehicles. As a more specific example, referring to FIG. 8, the first vehicle (810) as described above may be updated to the optimal route (A->2->4->9->B, 720), and the other removed points may be assigned to the second vehicle (820) and the third vehicle (830). For example, the platform (310) may provide reward information to the second vehicle (820) and the third vehicle (830) based on the removed specific points, and may enable optimal route setting to be performed based on the information.

[0086] As another example, the platform (310) can remove specific points from the initial cargo transport route as described above and set these points as undispatched cargo information. That is, the platform (310) can set an optimal route by removing some points from the initial cargo transport route based on reinforcement learning and set the information on some of the removed points as undispatched cargo information. Thereafter, the platform (310) can provide the undispatched cargo information to multiple registered drivers. For example, each driver can set a cargo transport route based on the undispatched cargo information and delivery type information. The platform (310) can provide reward information to drivers who have undispatched cargo assigned to some of the removed points, thereby improving delivery efficiency.

[0087] As another example, once the optimal route is derived based on the above, the platform (310) may provide route information to at least one of the registered vehicle owner and the registered shipper. Here, the platform (310) may provide visualized route information based on map information, allowing the registered vehicle owner and the registered shipper to obtain route-related information. As another example, the platform (310) may provide hub information at each location to the registered vehicle owner and the registered shipper, thereby enabling efficient route planning.

[0088] Figure 9 is a flowchart illustrating a reinforcement learning-based cargo transport routing method applicable to the present disclosure. Referring to FIG. 9, the platform (310) can build a data set based on vehicle owner information, shipper information, and environment information. (S910) Thereafter, the platform (310) can build a reinforcement learning-based route setting model using the built data set. (S920) Here, the platform (310) can obtain actual generated cargo information, actual environment information, and actual vehicle owner information, and provide them as inputs for the reinforcement learning-based route setting model. (S930) The platform (310) can derive an initial cargo transportation route for each vehicle owner through the route setting model based on the input. (S940) Here, the platform (310) can remove at least one cargo information within the derived initial cargo transportation route for each vehicle owner, and obtain a reward and a penalty. (S950) Thereafter, the platform (310) can repeat the operation of obtaining the reward and penalty (S960), through which the optimal cargo transportation route for each vehicle owner can be derived. Here, as an example, the owner information may include the owner's cargo vehicle information and the owner's driver information, the cargo information may include order information and cargo route information, and the environmental information may include at least one of traffic information, map information, and hub location information. In addition, the cargo vehicle information may include information on the type and year of the cargo vehicle, the size of the cargo compartment, and the type of cargo contained in the cargo vehicle, the owner's driver information may include information on the owner's age, gender, career, and health status, the order information may include information on the departure point, destination, and cargo type, the cargo route information may include information on the movement route, total carbon emissions, and delivery completion time according to the order information for each owner, the traffic information may include information on road conditions and signaling systems by time, the map information may include a map of the movement route for each owner, and the hub location information may include information on possible locations for loading and unloading cargo.

[0089] In addition, when the initial route for the first cargo transport for the first vehicle is derived through the route setting model, an operation of removing the first order information from the initial route for the first cargo transport and obtaining compensation information based on the removed first order information can be performed. In addition, the platform (310) can repeat the operation of removing order information that is different from the first order information from the initial route for the first cargo transport and obtaining compensation information based on the removed order information. Thereafter, the platform (310) can update the initial route for the first cargo transport to the optimal route for the first cargo transport based on the repeating operation. In addition, as an example, the removed first order information can be assigned to the initial route for the second cargo transport for the second vehicle. The platform (310) can obtain compensation information based on the assignment, and update the initial route for the first cargo transport to the optimal route for the first cargo transport by further considering the compensation information obtained based on the assignment. As an example, the compensation information can include at least one of reward information and penalty information determined based on delivery completion time and carbon emission information, as described above. Additionally, the removed first order information may be cargo information unloaded or loaded at the first point within the initial route of the first cargo transport. As another example, the removed first order information may be determined based further on the vehicle owner driver information. Here, the removed first order information may be determined by reflecting the environmental information at the first point, the risk of infectious disease outbreak, and the vehicle owner driver's health status information, but this may not be limited to this.

[0090] The embodiments described above can be implemented at least in part as computer programs and recorded on a computer-readable recording medium. A computer-readable recording medium on which a program for implementing the embodiments is recorded includes any type of recording device that stores data that can be read by a computer. Examples of computer-readable recording media include ROMs, RAMs, CD-ROMs, magnetic tapes, and optical data storage devices. In addition, the computer-readable recording medium can be distributed across network-connected computer systems, so that computer-readable codes can be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the present embodiments will be readily understood by those skilled in the art to which the present embodiments pertain.

[0091] Although the present specification has been described with reference to the embodiments illustrated in the drawings, these are merely exemplary, and those skilled in the art will understand that various modifications and variations of the embodiments are possible. However, such modifications should be considered within the technical protection scope of the present specification. Therefore, the true technical protection scope of the present specification should be determined to include other implementations, other embodiments, and equivalents to the claims, based on the technical spirit of the appended claims.

[0092] The present invention provides a method for deriving an optimal cargo transport route by setting an initial cargo transport route, removing some points from the initial cargo transport route, and repeating the route setting operation, thereby being applicable to industry.

Claims

1. A method for performing reinforcement learning-based cargo transport routing through a platform executed by a computing device, A step of constructing a data set based on vehicle owner information, shipper information, and environmental information; A step of constructing a reinforcement learning-based path setting model using the above-mentioned constructed data set; A step of obtaining actual generated cargo information, actual environmental information, and actual vehicle owner information and providing them as inputs to the reinforcement learning-based route setting model; A step of deriving an initial cargo transport route for each vehicle owner through the route setting model based on the above input; A step of removing at least one cargo information within the initial cargo transport route for each of the above-described owners and obtaining rewards and penalties; and A method for performing cargo routing, comprising: updating the route setting model by repeating the steps of obtaining the reward and the penalty, and deriving an optimal cargo transport route for each vehicle owner.

2. In paragraph 1, The above vehicle owner information includes the vehicle owner's cargo vehicle information and vehicle owner driver information. The above cargo information includes order information and cargo route information, A method for performing cargo routing, wherein the above environmental information includes at least one of traffic information, map information, and hub location information.

3. In paragraph 2, The above freight vehicle information includes the type of freight vehicle, year of manufacture, size of cargo compartment, and type of cargo contained in the freight vehicle. The above driver information includes the driver's age, gender, career, and health status information. The above order information includes information on origin, destination and cargo type, The above cargo route information includes the movement route, total carbon emissions, and delivery completion time information based on order information for each vehicle owner. The above traffic information includes hourly road conditions and signal system information, The above map information includes a map of the travel route for each vehicle, A method for performing cargo routing, wherein the above hub location information includes information on locations where cargo can be loaded or unloaded.

4. In paragraph 1, A step of performing an operation of removing first order information within the first cargo transport initial route and obtaining compensation information based on the removed first order information when the first cargo transport initial route for the first vehicle is derived through the above route setting model; A step of repeating an operation of removing order information different from the first order information within the first cargo transport initial route and obtaining compensation information based on the removed order information; and A method for performing cargo routing, comprising the step of updating the first cargo transport initial route to the first cargo transport optimal route based on the above-described repeated operation.

5. In paragraph 4, A step of allocating the removed first order information to the second cargo transport initial route of the second vehicle; A step of obtaining compensation information based on the above allocation; and A method for performing cargo routing, comprising the step of updating the first cargo transport initial route to the first cargo transport optimal route by further considering the compensation information obtained based on the above allocation.

6. In paragraph 5, A method for performing cargo routing, wherein the above compensation information includes at least one of reward information and penalty information determined based on delivery completion time and carbon emission information.

7. In paragraph 5, A method for performing cargo routing, wherein the first order information that has been removed is cargo information that is unloaded or loaded at a first point within the first cargo transport initial route.

8. In paragraph 7, The above first order information that has been removed is further determined based on the above next article information, A method for performing cargo routing, wherein the first order information is determined by reflecting the risk of infectious disease outbreak information and the health status information of the driver information based on the environmental information of the first point.

9. In paragraph 1, A step of performing an operation of removing first order information within the first cargo transport initial route and obtaining compensation information based on the removed first order information when the first cargo transport initial route for the first vehicle is derived through the above route setting model; A step of allocating the removed first order information to the second cargo transport initial route of the second vehicle; A step of obtaining compensation information based on the above allocation; A step of repeating an operation of removing order information different from the first order information within the first cargo transport initial route and assigning the removed order information to another vehicle's cargo transport initial route to obtain compensation information; and Including a step of updating the first cargo transport initial route to the first cargo transport optimal route based on the above repeated operation, The above vehicle owner information includes the vehicle owner's cargo vehicle information and vehicle owner driver information. The above cargo information includes order information and cargo route information, The above environmental information includes at least one of traffic information, map information, and hub location information, The above freight vehicle information includes the type of freight vehicle, year of manufacture, size of cargo compartment, and type of cargo contained in the freight vehicle. The above driver information includes the driver's age, gender, career, and health status information. The above order information includes information on origin, destination and cargo type, The above cargo route information includes the movement route, total carbon emissions, and delivery completion time information based on order information for each vehicle owner. The above traffic information includes hourly road conditions and signal system information, The above map information includes a map of the travel route for each vehicle, The above hub location information includes information on locations where cargo can be loaded and unloaded. A method for performing cargo routing, wherein the above compensation information includes at least one of reward information and penalty information determined based on delivery completion time and carbon emission information.

10. A computer program stored on a computer-readable medium that executes a method for performing reinforcement learning-based cargo transport routing according to any one of claims 1 to 9 in combination with hardware.

Images